spade submit

The spade submit command runs a pipeline in the cloud, converting its local file references on the way.

A pipeline you wrote on your own machine reads its inputs with data.read pointed at a path — /data/boundary.tif, file:///home/you/dem.tif. A cloud worker has no such filesystem, so that pipeline cannot run there as written. spade submit matches each of those files against your data catalog, uploads whatever is missing, rewrites the blocks into the equivalent platform.upload_* blocks, and submits the result.

Usage🔗

spade submit <pipeline.yaml>
spade submit <pipeline.yaml> --dry-run
spade submit <pipeline.yaml> --save --yes
spade submit <pipeline.yaml> --type @load_dem=raster

Requires a spade login session.

Flags🔗

What gets converted🔗

Only URIs that name your filesystem — a file:// URL or a bare absolute path.

Remote URIs are left exactly as written. data.read declares network: true and runs perfectly well on a worker, so an s3:// or https:// reference already works in both places and needs no conversion. The gap is narrower than "local blocks versus cloud blocks" suggests.

# converted — names a path on your machine
- name: data.read
  args: { uri: /data/boundary.tif }

# left alone — a worker can reach this
- name: data.read
  args: { uri: s3://bucket/boundary.tif }

Matching against your catalog🔗

Each local file is hashed (SHA-256) and looked up in your catalog. A hit means those exact bytes are already stored, so nothing is transferred — the block simply points at the existing asset.

Two cases cannot be settled by hash:

• Assets uploaded before content hashing existed carry no hash, so they can never match one.
• Folder assets are not hashed yet at all.

In both, spade submit shows you the candidates that share the file's name (and size) and asks which, if any, is the same asset. The default is always to upload a fresh copy. Under --yes, which cannot ask, it takes that default and tells you it did:

tiles: 1 catalog asset(s) match by name, but --yes never binds an unverified match; uploading a fresh copy

This is deliberate. Storing the same bytes twice costs storage and is easy to undo. Binding a pipeline to the wrong asset because two files happen to share a name is not: the run succeeds, and quietly computes the wrong answer.

Collections and globs🔗

A data.read_collection reading a directory becomes a folder asset containing that directory. Reading a glob becomes a folder asset containing exactly the files the glob matched:

- name: data.read_collection
  args: { uri: /data/tiles/*.tif }     # only the .tif files are uploaded

Uploading the whole directory instead would silently widen the collection — and collection order is positional, so the extra members would not merely be ignored downstream, they would shift every index.

Glob-derived assets are named after both the directory and the pattern (tiles/*.tif), so two different globs over one directory stay distinguishable the next time you submit.

Your pipeline file is not modified🔗

A submit is not an edit. The file on disk is left exactly as you wrote it.

If you want the converted version, --dry-run prints it and --save stores it in your cloud library:

spade submit pipeline.yaml --dry-run

name: demo
blocks:
  - id: 019faad0-3b21-762b-884b-3ab2bfb0759b
    name: platform.upload_raster
    inputs: []
    args:
      asset_id: <assigned-on-upload>
      object_key: <assigned-on-upload>

The placeholders are literal: a dry run transfers nothing, so assets that would be created have no id yet.

The conversion is one-way🔗

There is no cloud-to-local rewrite, and none is needed: upload blocks run locally too. The converted pipeline still works on your machine, so converting costs you nothing — which is exactly why the cloud-shaped pipeline is the portable form.

A pipeline written against the upload blocks' pre-0.3.0 collection name (cloud) is upgraded to platform.* in passing, so submitting an old pipeline modernizes it.

Tracking the run🔗

After submitting, spade submit polls and prints each status change until the run finishes:

Submitted run 019faad0-... Tracking progress (Ctrl-C to stop):
  • queued
  • running
  • succeeded

Run succeeded: https://spade.example/results/019faad0-...

A failed run exits non-zero. Ctrl-C stops watching, not the run.