Map/Reduce Pipelines

Some processing tasks involve applying the same operation to many items independently -- for example, processing every raster tile that a previous step already produced, or transforming every file returned by a directory listing. Spade supports this pattern through map/reduce pipelines, where a map block fans out work to parallel invocations and a reduce block collects the results.

Overview of the pattern🔗

A map/reduce pipeline follows three stages:

  1. Map: A map block takes a collection input -- a set of items that already exist as separate files before the map block ever runs -- and produces an expansion: a manifest enumerating those items one by one. A map block does not itself split, tile, or divide a single large input; it enumerates items an upstream block already produced or fetched as a collection. Spade reads the expansion and creates one parallel invocation of each downstream block per item.

  2. Process: Downstream blocks connected to the map block run once per expanded item, in parallel. Each invocation receives one item from the expansion. These blocks are ordinary blocks -- they do not need any special map-aware logic.

  3. Reduce: A reduce block collects the outputs from all parallel invocations and combines them into a single result. It receives a collection input containing all the individual outputs.

              +---> process (item 0) ---+
              |                         |
source ---> map ---> process (item 1) ---+--> reduce ---> output
              |                         |
              +---> process (item 2) ---+

Spade's two built-in map blocks both follow this shape: base.map_files enumerates a collection of files (item_type: file) for parallel fan-out, and gdal.map_raster_tiles enumerates a collection of raster tiles that some upstream block already fetched or produced as separate files. Neither block divides anything itself -- they only fan out over items that are already separate before the pipeline reaches them.

Declaring a map block invocation🔗

A map block is a block whose manifest declares kind: map. Both of Spade's built-in map blocks declare a collection-typed input -- the set of already-separate items they enumerate -- and spade check only requires a map block to have an expansion-typed output; it does not require the input to be a collection (see Pipeline Validation). Still, matching the built-in shape is the recommended pattern: wire the map block's input to any upstream block whose output type is collection, such as a fetch/listing block.

blocks:
  - id: "@tiles"
    name: data.read_collection
    inputs: []
    args:
      uri: "s3://example-bucket/scenes/2025-06/tiles/*.tif"
      format: "GeoTIFF"
      max_items: 200

  - id: "@enumerate"
    name: gdal.map_raster_tiles
    inputs:
      - "@tiles"
    args: {}

In this example, data.read_collection fetches a collection of raster tiles that already exist as separate objects in storage -- it lists and downloads them, it does not create or split them. gdal.map_raster_tiles is the map block: it enumerates that collection and writes an expansion manifest listing each tile as a separate item. Spade reads this manifest and creates parallel invocations of any downstream blocks -- one per tile.

Connecting downstream blocks in map context🔗

Any block that lists a map block in its inputs automatically enters map context. Spade creates one invocation of the downstream block for each item in the expansion. Each invocation receives the corresponding item as its input.

From the downstream block's perspective, nothing is different -- it receives a single input and produces a single output, just like any non-mapped block. The parallelism is handled entirely by the Spade scheduler.

  - id: "@reproject"
    name: gdal.warp
    inputs:
      - "@enumerate"
    args:
      target_crs: "EPSG:4326"
      resolution: 0
      resampling: "bilinear"
      output_format: "GTiff"

If gdal.map_raster_tiles produced 12 tiles, Spade creates 12 parallel invocations of gdal.warp, each reprojecting one tile. The scheduler labels each parallel invocation by appending the tile's index in brackets to the block name in its progress output -- gdal.warp[0], gdal.warp[1], and so on.

You can chain multiple blocks in map context. If another block depends on gdal.warp, it also runs once per tile:

  - id: "@convert"
    name: gdal.translate
    inputs:
      - "@reproject"
    args:
      output_format: "COG"
      output_type: ""
      width: 0
      height: 0
      scale_min: 0
      scale_max: 0

This creates another 12 parallel invocations, each receiving its own reprojected tile and converting it to Cloud-Optimized GeoTIFF.

Broadcasting non-mapped inputs🔗

Sometimes a block in map context needs both a mapped input (one per item) and a shared input that is the same for every invocation. This is called broadcasting.

If a block in map context lists multiple inputs, Spade distinguishes between:

  • Mapped inputs: Inputs that come from an upstream block in the same map context. Each invocation gets a different item.
  • Broadcast inputs: Inputs that come from an upstream block outside the map context. Every invocation gets the same data.

Whether an input is mapped or broadcast is determined entirely by where its source block sits relative to the map context -- it has nothing to do with whether the reference is bare or explicit. In fact, gdal.clip_raster_by_vector (below) needs explicit references for both of its inputs, because source and boundary are both declared as type file: with no format-independent way to tell them apart, a bare reference here would trip Spade's real ambiguity check (see Input References). Use as to pin each one, exactly as you would outside a map context.

blocks:
  # Source: fetch the shared study-area boundary (not mapped)
  - id: "@boundary"
    name: data.read
    inputs: []
    args:
      uri: "s3://example-bucket/boundaries/study-area.geojson"
      format: "GeoJSON"

  # ... "@tiles", "@enumerate", "@reproject", "@convert" as above ...

  # Process each tile: clip to the shared boundary
  - id: "@clip"
    name: gdal.clip_raster_by_vector
    inputs:
      - block: "@convert"
        output: raster
        as: source
      - block: "@boundary"
        output: file
        as: boundary
    args:
      crop_to_cutline: true
      all_touched: false

In this example, gdal.clip_raster_by_vector runs once per tile. Each invocation receives its own reprojected/converted tile (mapped, from @convert, which sits inside the same map context) plus the same study-area boundary (broadcast, from @boundary, which sits outside every map context in this pipeline).

Spade determines automatically which inputs are mapped and which are broadcast based on whether the upstream block is inside or outside the map context -- you do not configure it separately from the reference itself.

Reduce blocks collecting results🔗

A reduce block is a block whose manifest declares kind: reduce. It collects the outputs from all parallel invocations in a map context and produces a single combined result.

In the pipeline, the reduce block lists the last mapped block in its inputs. Spade gathers all the parallel outputs into a collection and passes them to the reduce block as a single input.

  - id: "@mosaic"
    name: gdal.reduce_mosaic
    inputs:
      - "@clip"
    args:
      resampling: "nearest"

The gdal.reduce_mosaic block is a reduce block. It receives a collection of all clipped tiles and combines them into a single output raster. The reduce block runs exactly once, after all parallel invocations of the upstream block have completed.

Inside the reduce block's handler, the input is a collection type (e.g., RasterFileCollection in Python, which provides a list of file paths) rather than a single file.

Complete example: tile processing pipeline🔗

Below is a complete end-to-end pipeline that demonstrates the full map/reduce pattern. The pipeline:

  1. Fetches a pre-tiled raster collection and a shared study-area boundary
  2. Enumerates the tiles for parallel fan-out (map)
  3. Reprojects each tile (parallel, mapped)
  4. Converts each tile to Cloud-Optimized GeoTIFF (parallel, mapped)
  5. Clips each tile to the shared boundary (parallel, mapped + broadcast)
  6. Mosaics the clipped tiles back together (reduce)
name: tile-processing
version: "1.0"
description: >
  Fetch a collection of already-tiled rasters, reproject and convert
  each tile, clip every tile to a shared study-area boundary, and
  mosaic the results back into one raster.

blocks:

  # ---- Sources (no dependencies) ----

  # Fetch the pre-tiled raster collection -- each file already exists
  # as its own separate object in storage.
  - id: "@tiles"
    name: data.read_collection
    inputs: []
    args:
      uri: "s3://example-bucket/scenes/2025-06/tiles/*.tif"
      format: "GeoTIFF"
      max_items: 200

  # Fetch the shared study-area boundary used to clip every tile
  - id: "@boundary"
    name: data.read
    inputs: []
    args:
      uri: "s3://example-bucket/boundaries/study-area.geojson"
      format: "GeoJSON"

  # ---- Map: enumerate the tile collection ----

  - id: "@enumerate"
    name: gdal.map_raster_tiles
    inputs:
      - "@tiles"
    args: {}

  # ---- Parallel processing (one invocation per tile) ----

  # Reproject each tile
  - id: "@reproject"
    name: gdal.warp
    inputs:
      - "@enumerate"
    args:
      target_crs: "EPSG:4326"
      resolution: 0
      resampling: "bilinear"
      output_format: "GTiff"

  # Convert each tile to Cloud-Optimized GeoTIFF
  - id: "@convert"
    name: gdal.translate
    inputs:
      - "@reproject"
    args:
      output_format: "COG"
      output_type: ""
      width: 0
      height: 0
      scale_min: 0
      scale_max: 0

  # Clip each tile to the broadcast boundary
  - id: "@clip"
    name: gdal.clip_raster_by_vector
    inputs:
      - block: "@convert"
        output: raster
        as: source
      - block: "@boundary"
        output: file
        as: boundary
    args:
      crop_to_cutline: true
      all_touched: false

  # ---- Reduce: mosaic all clipped tiles ----

  - id: "@mosaic"
    name: gdal.reduce_mosaic
    inputs:
      - "@clip"
    args:
      resampling: "nearest"

The execution flow is:

  1. data.read_collection and data.read run in parallel (no dependencies on each other).
  2. gdal.map_raster_tiles runs after the tile collection is fetched. It produces an expansion of N items -- one per tile already present in the collection.
  3. gdal.warp runs N times in parallel, once per tile.
  4. gdal.translate runs N times in parallel, once per tile, each depending on its own gdal.warp invocation.
  5. gdal.clip_raster_by_vector runs N times in parallel. Each invocation receives its own converted tile (mapped) and the shared boundary (broadcast from @boundary, which sits outside the map context).
  6. gdal.reduce_mosaic runs once after all clip invocations complete. It receives a collection of all N clipped tiles and produces a single output mosaic.

Nested map/reduce🔗

A map block may itself sit inside another map block's context, giving you multi-level fan-out -- for example, enumerate scenes, then enumerate tiles within each scene, process each tile, mosaic per scene, then combine all scenes. Nesting requires no special YAML: it falls out of which blocks depend on which. Downstream invocation IDs gain one index component per enclosing map level (@clip.1.4 is tile 4 of scene 1), and inner reduce blocks run once per outer item rather than once for the whole pipeline. Nesting is capped at 4 levels deep, since invocation counts multiply at each level.

See Nested map/reduce for the full mechanics (ragged fan-out, broadcasting by context depth, well-nestedness) and a worked YAML example.

Constraints and limitations🔗

  • Reduce blocks must have kind: reduce in their manifest. An ordinary block cannot receive a collection input from a map context -- Spade will report a type error during validation.
  • Broadcast inputs must come from outside the map context they're feeding, or from an enclosing context. A block cannot broadcast an input from a sibling invocation inside its own context.
  • Every map context must be closed by a matching reduce, and contexts must be well-nested -- you cannot combine the outputs of two sibling unclosed contexts. See Pipeline Validation for the exact rules spade check enforces.
  • All parallel invocations in a context must complete before that context's reduce block runs. There is no partial reduction or streaming behavior.