base
The base collection is Spade's core block collection. It provides the primitives every pipeline is likely to need: tabular data transformations, map blocks for fan-out, reduce blocks for fan-in, and format conversion utilities.
The collection is implemented in Rust and ships as a single binary with one subcommand per block. Tabular blocks use Polars and accept either CSV or Parquet input — Parquet is the preferred internal format because it is columnar, typed, and self-describing.
Tabular blocks🔗
These blocks operate on tables (CSV or Parquet files) and produce Parquet output by default.
| Block | Kind | Description |
|---|---|---|
base.filter_rows | standard | Filter rows by a SQL WHERE-style predicate |
base.select_columns | standard | Keep or drop a subset of columns |
base.aggregate | standard | Compute aggregations over a table, optionally grouped |
base.group_by | standard | Group by columns and aggregate (convenience wrapper) |
base.csv_to_parquet | standard | Convert a CSV file to Parquet |
base.parquet_to_csv | standard | Convert a Parquet file to CSV |
Map blocks🔗
Map blocks enumerate items and emit an expansion manifest that fans downstream blocks out in parallel.
| Block | Kind | Description |
|---|---|---|
base.map_files | map | Fan out over every file in a collection |
base.map_list | map | Fan out over a literal list of scalar values |
base.map_range | map | Fan out over a numeric range |
Reduce blocks🔗
Reduce blocks collect mapped outputs back into a single result.
| Block | Kind | Description |
|---|---|---|
base.reduce_collection | reduce | Gather mapped outputs back into a collection |
base.reduce_stack | reduce | Concatenate tables row-wise (rbind / UNION ALL) |
base.reduce_join | reduce | Join tables on key columns (SQL JOIN) |
Installation🔗
spade install file:///path/to/blocks/basebase.filter_rows
Filter rows of a table by a SQL WHERE-style predicate.
Read →base.select_columns
Project a subset of columns from a table, keeping or dropping them.
Read →base.aggregate
Compute aggregations over a table, optionally grouped.
Read →base.group_by
Group a table by columns and compute aggregations per group.
Read →base.csv_to_parquet
Convert a CSV file to Parquet.
Read →base.parquet_to_csv
Convert a Parquet file to CSV.
Read →base.map_files
Fan out over every file in a collection for parallel processing.
Read →base.map_list
Fan out over a literal list of scalar values.
Read →base.map_range
Fan out over a numeric range.
Read →base.reduce_collection
Gather mapped outputs back into a single collection.
Read →base.reduce_stack
Concatenate a collection of tables row-wise.
Read →base.reduce_join
Join a collection of tables left-to-right on one or more key columns.
Read →