User Uploads

Pipelines often need your own data — a custom raster, a boundary vector, a table of plots — not just public datasets. Spade brings these in through uploads: you upload a file once, and pipelines reference it like any other data, share it without re-uploading, and consume it through an ordinary block.

The model🔗

An upload is a first-class, persistent asset, not a per-pipeline attachment:

  • Bytes live in Spade's object storage.
  • Metadata (name, size, type, owner, storage key) lives in a catalog.

A pipeline references an asset by a stable object key, never by a URL. The key is owner-namespaced, which is also the authorization anchor: a pipeline may read an asset only when the asset belongs to the pipeline's owner.

Uploads are pulled into a pipeline by a first-party upload block rather than a special pipeline primitive — so the "everything is a node" model stays uniform. See the platform collection for the blocks themselves.

Uploading a file🔗

Web UI. Drag an Upload node onto the canvas. It lets you upload a new file or pick an existing asset from your catalog, and shows a type dropdown defaulted from the filename (raster / vector / table / file) that you can override. Connect the node's output to the block that should consume it. The node serializes to the matching platform.upload_<type> block with the asset's id and object key — the variant is hidden from you.

CLI. spade data upload boundary.tif uploads the file and prints an asset id. Point it at a directory to upload a whole folder as one asset; large folders transfer several files at a time and resume where they left off if interrupted. See spade data.

Because assets are addressed by a stable key, uploading the same file once and referencing it from several pipelines reuses the same object — no re-transfer, no duplication. Uploads are also content-hashed: if you upload a file you already have, the CLI recognizes the bytes and skips the transfer entirely rather than storing a second copy.

Access at run time (pre-signed URLs)🔗

No standing storage credential ever lives inside a block. When a cloud run reaches an upload block, the worker mints a short-lived, single-object pre-signed GET URL for the asset and injects it into the block, which fetches the bytes and writes them to its output. The URL expires quickly and is never logged; a redelivered job mints a fresh one.

Running locally🔗

Upload blocks run locally too. There is no object storage on your machine, so spade run resolves each asset to a file on disk and hands the block a file:// URL instead of a pre-signed one — the block fetches both through the same code path, so the pipeline itself is unchanged.

Resolution tries three things, most specific first:

  1. --asset <asset-id>=<path> — point the block at a local copy explicitly. Repeatable.
  2. The local asset cache — a copy under ~/.spade/assets/<asset-id>/, from an earlier run or download.
  3. A fetch from your catalog, which then fills the cache. When the catalog knows the asset's hash, the download is verified against it.

The first two need no network and no login, so once an asset has been fetched even once, that pipeline runs offline. spade check will tell you which assets are cached and which a run would have to fetch, without downloading anything.

spade run pipeline.yaml                                    # fetch or reuse the cache
spade run pipeline.yaml --asset 019cf4bc-aaaa=./boundary.tif   # use a local copy

Going the other way🔗

A pipeline you wrote by hand probably reads its inputs with data.read pointed at a path on your machine — which a cloud worker has no way to open. spade submit converts those references for you: it matches each local file against your catalog by content hash, uploads whatever is missing, rewrites the blocks to the matching upload block, and runs the result in the cloud.

Since upload blocks also run locally, the converted pipeline still works on your machine — you don't give anything up by converting. Your pipeline file is left untouched; --dry-run prints the converted version if you want to keep it.

Out of scope🔗

Cross-user sharing beyond the owner-read model, retention/cleanup of unreferenced assets, and format conversion on ingest are not covered here — the upload block passes bytes through, and the declared type is your assertion about the file.