Pipeline Validation

Running spade check <pipeline.yaml> validates your pipeline file against a set of rules before execution. This catches structural errors, missing blocks, broken references, and type mismatches early -- before any blocks actually run.

A valid pipeline produces output like:

Pipeline "satellite-reproject" is valid.

(If this is the first time a short-code pipeline is checked, spade check prints an extra Wrote <pipeline-stem>.lock.yaml line first, since that's also when the lockfile gets created.)

An invalid pipeline produces the batched error list instead:

Pipeline validation failed with 2 error(s):
  - <first error>
  - <second error>

Every line in that list is exactly error.Error() for one entry in the slice core.ValidatePipeline returns -- there is no separate human-friendly formatting layer. The examples below use the real strings the validator produces (see core/pipeline.go), wrapped exactly the way cli/cmd/check.go prints them.

Validation rules🔗

Spade checks the following seven core rules, in order, plus two additional rule sets that only apply if your pipeline uses short codes or map/reduce (see Additional validation for short codes and map/reduce below). Each core rule is described below with an explanation and an example of a pipeline that violates it.

Rule 1: Unique invocation IDs🔗

Every block invocation in the pipeline must have a unique id. No two blocks may share the same invocation ID.

Why this matters: Invocation IDs are used to reference blocks in inputs lists. Duplicate IDs would make references ambiguous, and Spade would not know which invocation to use as the data source.

Example of a violation:

name: duplicate-id-example
version: "1.0"

blocks:
  - id: "@fetch"  # <-- same ID
    name: data.census_acs
    inputs: []
    args:
      year: 2022
      dataset: "acs5"
      table: "B01003"
      geography: "state:*"
      variables: ""

  - id: "@fetch"  # <-- same ID
    name: fiadb.parameters
    inputs: []
    args:
      name: "snum"

Real error (duplicate block invocation id: %s in core/pipeline.go):

Pipeline validation failed with 1 error(s):
  - duplicate block invocation id: 019cf4bc-1111-7000-0000-000000000000

Note that the message is intentionally terse -- it names the offending UUID only, not the two block names that collided.

Rule 2: All referenced IDs exist🔗

Every invocation ID referenced in an inputs list must correspond to an actual block invocation in the pipeline. This applies to both bare references and the block key in explicit references.

Why this matters: A reference to a non-existent ID means the block is expecting data from a step that does not exist. This is usually caused by a typo or a deleted block.

Example of a violation:

name: broken-ref-example
version: "1.0"

blocks:
  - id: "@source"
    name: data.read
    inputs: []
    args:
      uri: "s3://example-bucket/raw.parquet"
      format: "Parquet"

  - id: "@filtered"
    name: base.filter_rows
    inputs:
      - "@deleted-block"  # <-- does not exist
    args:
      expression: "state = 'ME'"

Real error: this rule and the acyclic-graph check (Rule 4) both independently notice the same broken reference, so a single typo produces two entries in the batched list -- one from ValidatePipeline's own loop (block %s references unknown invocation id %s), and a second from BuildDependencyGraph, which ValidatePipeline calls as part of Rule 4 and which does its own existence check while building edges (block %s references unknown dependency %s):

Pipeline validation failed with 2 error(s):
  - block 019cf4bc-2222-7000-0000-000000000000 references unknown invocation id 019cf4bc-9999-7000-0000-000000000000
  - block 019cf4bc-2222-7000-0000-000000000000 references unknown dependency 019cf4bc-9999-7000-0000-000000000000

Because BuildDependencyGraph failing is treated as fatal (there's no valid graph to check further rules against), ValidatePipeline returns immediately after this -- rules 5, 6, 7, and the map/reduce checks never run in the same invocation when a reference is broken this way.

Rule 3: Block names refer to installed blocks🔗

Every name field in the blocks list must refer to a block that is installed in the local Spade environment. The name uses collection.block format, and both the collection and the specific block must be present.

Why this matters: If Spade cannot find the block definition, it cannot determine the block's inputs, outputs, or how to execute it.

Example of a violation:

name: missing-block-example
version: "1.0"

blocks:
  - id: "@source"
    name: data.read
    inputs: []
    args:
      uri: "s3://example-bucket/raw.tif"
      format: "GeoTIFF"

  - id: "@transform"
    name: gdal.fancy-transform  # <-- not a real block
    inputs:
      - "@source"
    args: {}

Real behavior is split across two layers here, and it's worth knowing both:

core.ValidatePipeline itself has a dedicated rule for this (block %s references unknown block type %q) that would appear in the batched list like any other rule -- if it ever ran. In practice, spade check <pipeline.yaml> almost never reaches it: before calling ValidatePipeline at all, cli/cmd/check.go looks up every block name in the local registry to load its manifest, and returns immediately on the first failure:

Error: block type "gdal.fancy-transform" not found in registry: record not found

This is a plain top-level error (via cobra's default Error: ... printing), not one entry in a Pipeline validation failed with N error(s) list -- and because check.go returns as soon as this happens, no other rule gets a chance to run in the same invocation, even if the rest of the pipeline also has problems.

If you call core.ValidatePipeline directly with a manifest map that's missing an entry (for example, from a different tool built on the core library), you'd see the batched form instead:

Pipeline validation failed with 1 error(s):
  - block 019cf4bc-2222-7000-0000-000000000000 references unknown block type "gdal.fancy-transform"

Rule 4: Dependency graph is acyclic🔗

The dependency graph formed by inputs references must be a directed acyclic graph (DAG). In other words, there must be no circular dependencies where Block A depends on Block B, which depends on Block C, which depends on Block A.

Why this matters: Spade executes blocks by running dependencies first. A cycle means no block in the cycle can run before the others, creating a deadlock. Processing pipelines are inherently feedforward -- data flows from sources to sinks.

Example of a violation:

name: cycle-example
version: "1.0"

blocks:
  - id: "@process-a"
    name: gdal.warp
    inputs:
      - "@process-c"  # depends on Block 3
    args:
      target_crs: "EPSG:4326"
      resolution: 0
      resampling: "bilinear"
      output_format: "GTiff"

  - id: "@process-b"
    name: gdal.sieve
    inputs:
      - "@process-a"  # depends on Block 1
    args:
      threshold: 10
      connectedness: 8

  - id: "@process-c"
    name: gdal.fill_nodata
    inputs:
      - "@process-b"  # depends on Block 2
    args:
      max_distance: 100
      smoothing_iterations: 0

Real error (bare cycle detected in dependency graph, with no interpolation at all) -- and it actually appears twice:

Pipeline validation failed with 2 error(s):
  - cycle detected in dependency graph
  - cycle detected in dependency graph

This duplication is real, not a copy-paste mistake in this doc: ValidatePipeline runs graph.TopologicalSort() directly as Rule 4 and appends the error, but does not stop there -- it continues on to the remaining rules (unlike a totally unknown dependency ID, which does stop everything early in Rule 2 above, since there's no valid graph left to check further rules against). Validation then reaches the map/reduce structural checks at the end of ValidatePipeline, which call BuildContextTree unconditionally, even when the pipeline has no map or reduce blocks at all -- and the very first thing BuildContextTree does is call TopologicalSort() again on the same graph, which fails the same way and appends the identical message a second time. If your pipeline's cycle happens to also break rules 5, 6, or 7, you'll see those errors sandwiched between the two identical cycle lines.

Rule 5: Input/output type compatibility🔗

When two blocks are connected (one listed in the other's inputs), the upstream block's output type must be compatible with the downstream block's input type. For bare references, at least one unambiguous type match must exist. For explicit references that use as, the named input's type must be compatible with the named output's type.

Type compatibility is checked at the coarse type field only (file, collection, json, string, number, boolean, expansion, directory) -- the more specific format annotation you see in manifests (GeoTIFF, CSV, Parquet, GeoJSON, ...) is documentation only and is never compared by the validator. A block declaring a file/CSV input is, as far as spade check is concerned, fully compatible with any upstream file/GeoTIFF output. If a block reads the wrong format, that fails at runtime inside the block's handler, not during validation.

Why this matters: Type checking prevents runtime failures where a block receives data of a structurally incompatible kind (a JSON summary where a file collection was expected, for instance). It does not, however, protect you from wiring together files of the wrong format -- that's on you (or the block's own input validation) to catch.

Example of a violation:

name: type-mismatch-example
version: "1.0"

blocks:
  - id: "@snum-dictionary"
    name: fiadb.parameters
    inputs: []
    args:
      name: "snum"

    # fiadb.parameters produces output "dictionary", type: json

  - id: "@upload"
    name: data.write_collection
    inputs:
      - block: "@snum-dictionary"
        output: dictionary
        as: files
    args:
      uri: "s3://example-bucket/uploads/"
      overwrite: false

    # data.write_collection's "files" input is type: collection

Real error (explicit reference to %q: type %q not compatible with output type %q, wrapped as block %s (%s): %w):

Pipeline validation failed with 1 error(s):
  - block 019cf4bc-2222-7000-0000-000000000000 (data.write_collection): explicit reference to "files": type "collection" not compatible with output type "json"

Rule 6: Named outputs match declarations🔗

When an explicit reference uses the output key, the named output must actually exist in the upstream block's manifest. If the upstream block does not declare an output with that name, validation fails.

Why this matters: An explicit reference to a non-existent output is an error -- there would be no data to wire. This is often caused by a typo in the output name.

Example of a violation:

name: bad-output-name-example
version: "1.0"

blocks:
  - id: "@source"
    name: data.read
    inputs: []
    args:
      uri: "s3://example-bucket/raw.parquet"
      format: "Parquet"

  - id: "@filtered"
    name: base.filter_rows
    inputs:
      - "@source"
    args:
      expression: "state = 'ME'"

    # base.filter_rows declares exactly one output: "result"

  - id: "@next"
    name: base.select_columns
    inputs:
      - block: "@filtered"
        output: filtered  # <-- "filtered" is not a declared output
    args:
      columns: "county_fips"
      mode: keep

Real error (block %s has no output named %q, wrapped as block %s (%s): %w):

Pipeline validation failed with 1 error(s):
  - block 019cf4bc-2222-7000-0000-000000000000 (base.select_columns): block 019cf4bc-1111-7000-0000-000000000000 has no output named "filtered"

Note the message names the upstream block's ID a second time inside the wrapped text -- it isn't repackaged into a friendlier "available outputs: ..." style message.

Rule 7: Required args are present🔗

Every scalar input (string, number, or boolean) declared in a block's manifest must have a corresponding entry in the args map of the pipeline invocation. This is unconditional: core's InputDeclaration type has no default-value or required/optional field at all, so a description that mentions a "default" (for example, gdal.warp's resampling: "Default nearest") is documentation only -- it is not read or enforced by validation or by the block runtime. If the key is missing from args, validation fails regardless of what the manifest's description text says.

Why this matters: If a required parameter is missing, the block handler will fail at runtime when it tries to read the parameter from params.yaml.

Example of a violation:

name: missing-args-example
version: "1.0"

blocks:
  - id: "@source"
    name: data.read
    inputs: []
    args:
      uri: "s3://example-bucket/raw.parquet"
      format: "Parquet"

  - id: "@filtered"
    name: base.filter_rows
    inputs:
      - "@source"
    args: {}
    # Missing required arg: expression

  - id: "@reproject"
    name: gdal.warp
    inputs:
      - "@filtered"
    args:
      resolution: 0
      resampling: "bilinear"
      output_format: "GTiff"
      # Missing required arg: target_crs

Real error (block %s missing required arg %q):

Pipeline validation failed with 2 error(s):
  - block 019cf4bc-1111-7000-0000-000000000000 missing required arg "expression"
  - block 019cf4bc-2222-7000-0000-000000000000 missing required arg "target_crs"

Additional validation for short codes and map/reduce🔗

These rules only apply to pipelines that use the relevant feature -- they run in addition to the seven core rules above.

Short codes🔗

If a pipeline uses @-prefixed short codes instead of UUIDs, spade check additionally verifies:

1. Grammar -- every short code matches @[A-Za-z_][A-Za-z0-9_]*.
2. Resolution -- every short code referenced in inputs is defined as the id of some block in the pipeline.
3. Uniqueness -- no two blocks share the same short code (a repeat resolves to the same UUID, which then trips Rule 1 above).
4. Lockfile validity -- every binding in the sibling .lock.yaml is a valid UUIDv7, and every bound short code that's still referenced in the source exists.

See Short Codes and Hand-Authoring for the full detail and error message examples.

Map/reduce🔗

If a pipeline contains kind: map or kind: reduce blocks, spade check additionally verifies:

1. Map blocks output expansion -- a kind: map block must declare at least one output of type expansion. Real error: map block %s (%s) must have an expansion output.
2. Reduce blocks accept collection -- a kind: reduce block must declare at least one input of type collection. Real error: reduce block %s (%s) must have a collection input.
3. Every map context is closed by a reduce -- a fan-out that never reaches a matching kind: reduce block is rejected. Real error: map block %s (%s) has no reduce block closing its context.
4. Contexts are well-nested -- a block may not combine the outputs of two sibling map contexts unless at least one has already been closed by its reduce.
5. Nesting depth does not exceed 4 levels -- since invocation counts multiply at each nesting level, this bounds worst-case fan-out.

See Map/Reduce Pipelines and Nested map/reduce for the full mechanics.

Summary of validation rules🔗

Tips for fixing validation errors🔗

• Duplicate IDs: Generate a new UUIDv7 for one of the conflicting blocks. If you used the same short code (e.g. "@foo") on two blocks, rename one -- short codes resolve to the same UUID, so duplicates trigger this rule.
• Broken references: Check for typos in the invocation ID or short code. Copy-paste the ID or short code directly from the target block.
• Missing blocks: Run spade install <repository> to install the required collection. Remember that this error surfaces on its own, before the rest of validation runs -- fix it and re-run spade check to see any remaining problems.
• Cycles: Restructure your pipeline so data flows in one direction. If you have a feedback loop in your algorithm, consider implementing it inside a single block rather than across multiple blocks.
• Type mismatches: Check the block manifests (spade check with no arguments in a collection directory) to confirm the input and output type fields. Remember format isn't enforced -- if two blocks agree on type but not format, validation will pass and the mismatch will only surface as a runtime failure inside the handler.
• Bad output names: Run spade check in the upstream block's collection to see its declared outputs, or inspect its blocks/<name>.yaml manifest directly.
• Missing args: Check the block manifest for every scalar (string/number/boolean) input and make sure each one has a key in your args map -- a "default" mentioned in the description is not enforced anywhere.
• Corrupt lockfile: If spade check reports invalid lockfile: ..., delete the sibling <pipeline-stem>.lock.yaml to regenerate bindings from scratch. See Short Codes and Hand-Authoring for the full set of lockfile rules.