Pipeline Examples

This page contains complete, ready-to-use pipeline examples demonstrating common patterns, built entirely from real blocks in the base, data, gdal, and fiadb collections shipped with Spade (see the Block Catalog for the full list). Each example includes a description of the processing workflow, the full YAML pipeline file, and an explanation of the data flow.

Example 1: Simple two-block linear pipeline🔗

Description: Fetch a raster and reproject it to a different coordinate reference system. This is the simplest possible pipeline -- two blocks connected in sequence.

name: simple-reproject
version: "1.0"
description: Fetch a raster and reproject it to EPSG:4326

blocks:
  - id: "@source"
    name: data.read
    inputs: []
    args:
      uri: "s3://example-bucket/scenes/2025-06-01.tif"
      format: "GeoTIFF"

  - id: "@reproject"
    name: gdal.warp
    inputs:
      - "@source"
    args:
      target_crs: "EPSG:4326"
      resolution: 0
      resampling: "bilinear"
      output_format: "GTiff"

Data flow:

data.read --> gdal.warp
  1. data.read runs first. It has no inputs (source block), so it starts immediately. It fetches the raster from storage, producing a file output.
  2. gdal.warp runs after data.read completes. It receives the raster via a bare reference and reprojects it to EPSG:4326.

The bare reference works here because data.read produces one file output and gdal.warp expects one file input -- the type match is unambiguous.

Example 2: Parallel branches that merge🔗

Description: Fetch a reference raster and a vector layer independently and in parallel, then burn the vector geometries into the raster's grid.

name: parallel-rasterize
version: "1.0"
description: >
  Fetch a reference raster and a vector layer in parallel, then
  rasterize the vector onto the raster's grid.

blocks:
  # Step 1a: Fetch the reference raster (parallel branch 1)
  - id: "@dem"
    name: data.read
    inputs: []
    args:
      uri: "s3://example-bucket/dem.tif"
      format: "GeoTIFF"

  # Step 1b: Fetch the vector layer (parallel branch 2)
  - id: "@parcels"
    name: data.read
    inputs: []
    args:
      uri: "s3://example-bucket/parcels.geojson"
      format: "GeoJSON"

  # Step 2: Burn the parcels into the DEM's grid
  - id: "@burned"
    name: gdal.rasterize
    inputs:
      - block: "@parcels"
        output: file
        as: vectors
      - block: "@dem"
        output: file
        as: reference
    args:
      burn_value: 1
      attribute: ""
      all_touched: false

Data flow:

data.read (@dem)     --+
                        +--> gdal.rasterize
data.read (@parcels) --+
  1. @dem and @parcels are both source blocks with no dependencies, so Spade runs them in parallel.
  2. gdal.rasterize waits until both complete, then receives both outputs.

Both @dem and @parcels produce a file-typed output (data.read's only output is always file, regardless of format), and gdal.rasterize's two non-scalar inputs (vectors, reference) are also both type file. Spade's type matcher only looks at the coarse type field, not format -- so from its perspective these are two indistinguishable file-to-file connections. Bare references would trip a real ambiguity error here (the first one processed would find both vectors and reference as compatible candidates). This is why the pipeline above uses explicit references with as for both inputs, pinning @parcels to vectors and @dem to reference directly. See Input References for the full mechanics, including what happens if you provide block+output without as in a case like this.

Example 3: Map/reduce tile processing pipeline🔗

Description: Fetch a collection of already-tiled rasters and a shared study-area boundary, enumerate the tiles for parallel fan-out, reproject and clip each tile in parallel, then mosaic the results back into a single output image. This demonstrates the full map/reduce pattern.

name: tile-processing
version: "1.0"
description: >
  Fetch a collection of already-tiled rasters, reproject and clip
  each tile to a shared boundary, then mosaic the results.

blocks:
  # Fetch the pre-tiled raster collection (each file already exists
  # as its own separate object in storage)
  - id: "@tiles"
    name: data.read_collection
    inputs: []
    args:
      uri: "s3://example-bucket/scenes/2025-06/tiles/*.tif"
      format: "GeoTIFF"
      max_items: 200

  # Fetch the shared study-area boundary (broadcast input)
  - id: "@boundary"
    name: data.read
    inputs: []
    args:
      uri: "s3://example-bucket/boundaries/study-area.geojson"
      format: "GeoJSON"

  # Map: enumerate the tile collection
  - id: "@enumerate"
    name: gdal.map_raster_tiles
    inputs:
      - "@tiles"
    args: {}

  # Parallel: reproject each tile
  - id: "@reproject"
    name: gdal.warp
    inputs:
      - "@enumerate"
    args:
      target_crs: "EPSG:4326"
      resolution: 0
      resampling: "bilinear"
      output_format: "GTiff"

  # Parallel: clip each tile to the broadcast boundary
  - id: "@clip"
    name: gdal.clip_raster_by_vector
    inputs:
      - block: "@reproject"
        output: raster
        as: source
      - block: "@boundary"
        output: file
        as: boundary
    args:
      crop_to_cutline: true
      all_touched: false

  # Reduce: mosaic all clipped tiles
  - id: "@mosaic"
    name: gdal.reduce_mosaic
    inputs:
      - "@clip"
    args:
      resampling: "nearest"

Data flow:

data.read_collection --> gdal.map_raster_tiles --+--> gdal.warp (tile 0) --> gdal.clip_raster_by_vector (tile 0) --+
                                                  |                                                                |
data.read (@boundary) ---------broadcast----------+--> gdal.warp (tile 1) --> gdal.clip_raster_by_vector (tile 1) --+--> gdal.reduce_mosaic
                                                  |                                                                |
                                                  +--> gdal.warp (tile N) --> gdal.clip_raster_by_vector (tile N) --+
  1. data.read_collection and data.read run in parallel (both are source blocks with no dependencies).
  2. gdal.map_raster_tiles runs after the tile collection is fetched. It is a map block (kind: map in its manifest) that enumerates the tiles the collection already contains -- it does not split anything itself -- and produces an expansion manifest listing each one.
  3. gdal.warp enters map context because its input comes from the map block. Spade creates one invocation per tile, all running in parallel.
  4. gdal.clip_raster_by_vector also runs in map context. Each invocation receives two inputs:
    • Its own reprojected tile (mapped input from gdal.warp)
    • The study-area boundary (broadcast input from data.read, shared across all invocations)
  5. gdal.reduce_mosaic is a reduce block (kind: reduce in its manifest). It waits for all clip invocations to complete, receives a collection of all clipped tiles, and produces a single mosaic output.

Example 4: Explicit input references🔗

Description: Join two independently-produced tables on a shared key column, then derive a new column from the joined result. Because base.join's two inputs (left and right) are both type file, bare references would be ambiguous. This example uses explicit references to wire the correct outputs to the correct inputs.

name: explicit-references
version: "1.0"
description: >
  Join two tables and compute a derived column, demonstrating why
  explicit references with `as` are needed when both inputs share a type.

blocks:
  # Fetch the area-estimates table
  - id: "@estimates"
    name: data.read
    inputs: []
    args:
      uri: "s3://example-bucket/area-estimates.parquet"
      format: "Parquet"

  # Fetch the county covariates table
  - id: "@covariates"
    name: data.read
    inputs: []
    args:
      uri: "s3://example-bucket/county-covariates.parquet"
      format: "Parquet"

  # Join the two tables on the shared key column.
  # base.join declares two inputs of the same type ("left" and
  # "right", both `file`), so neither a bare reference nor a plain
  # block+output reference (without `as`) can safely disambiguate
  # them -- see the note below.
  - id: "@joined"
    name: base.join
    inputs:
      - block: "@estimates"
        output: file
        as: left
      - block: "@covariates"
        output: file
        as: right
    args:
      on: county_fips
      how: inner

  # Derive a per-capita column from the joined result
  - id: "@per-capita"
    name: base.mutate
    inputs:
      - "@joined"
    args:
      expressions: '[{"name":"per_capita","expr":"estimate / population"}]'

Data flow:

                          file output ---> left input
data.read (@estimates)                                   --> base.join --> base.mutate
                          file output ---> right input
data.read (@covariates)
  1. @estimates and @covariates each fetch a table, running in parallel.
  2. @joined needs both tables as input. Because both outputs and both inputs share the same type (file), type matching can't disambiguate them even after naming the output -- as is required to pin the wiring directly:
    • output: file from @estimates with as: left wires it straight to the left input.
    • output: file from @covariates with as: right wires it straight to the right input.
  3. @per-capita receives the joined table via a bare reference (unambiguous, since base.join produces one output) and computes a derived column.

If you attempted to use bare references for step 2:

# THIS FAILS VALIDATION
inputs:
  - "@estimates"
  - "@covariates"

spade check reports a real ambiguity error here, because the first bare reference it processes already has two type-compatible candidate inputs (left and right):

Pipeline validation failed with 1 error(s):
  - block <joined-id> (base.join): ambiguous type match: output <estimates-id>.file (type "file") matches multiple inputs: [left right]

If you instead used block+output without as for both references, spade check would not report an error at all -- it would silently wire whichever reference comes first in the list to left and the other to right, regardless of which table you meant to go where. See Input References for why this happens and why as is the only way to guarantee the correct wiring.