Output Retention

Every block in a cloud run writes its outputs to object storage, and by default they stay there. For small pipelines that is what you want. For large ones the intermediates can be many times the size of the results, and you pay to store all of it.

Retention gives you three controls:

• a policy on the pipeline that drops intermediates when the run succeeds and expires everything else after a while,
• a per-block override for an intermediate you want to keep,
• a Delete outputs button on a finished run.

Retention applies to runs in the cloud. spade run reads the policy, checks it, and ignores it: a local run's working directory is on your own disk, and spade clean is how you clear it.

The policy🔗

Add a retention field at the top of the pipeline:

name: upper-miss-river
version: "1"
retention:
  intermediates: drop
  expires: 30d
blocks:
  - id: "@tiles"
    name: data.read_collection
    inputs: []
    args:
      uri: "s3://example-bucket/tiles/*.laz"

The two are independent. Use intermediates: drop alone to keep only the results, forever. Use expires alone to keep everything for a while. Use both to keep only the results, for a while.

What counts as an intermediate🔗

An output is an intermediate when some other block in the same pipeline uses it as an input. An output nothing consumes is a result, and intermediates: drop never touches it.

How the downstream block refers to its input matters:

• An explicit reference (block: and output:) consumes only the output it names.
• A bare reference consumes every output of the block it points at.

So if a block produces a raster and a report, and the next block takes the raster through a bare reference, both the raster and the report are dropped. If you want the report, either reference the raster explicitly or mark the producing block retain: true.

Keeping one intermediate🔗

  - id: "@reprojected"
    name: gdal.warp
    retain: true
    inputs:
      - "@source"
    args:
      target_crs: "EPSG:4326"

retain: true keeps all of that block's outputs when intermediates: drop would otherwise delete them. It does not exempt them from expires.

Failed and cancelled runs🔗

A run that fails or is cancelled has no results, so intermediates: drop does not apply to it: everything it produced before stopping is kept, which is what you need to work out what went wrong. Those outputs then expire after expires if you set one, and otherwise after the platform's default for failed runs (seven days unless your administrator changed it). A failed run never keeps its outputs forever.

Deleting a run's outputs by hand🔗

Open a finished run and choose Delete outputs. This removes every output file of the run from storage. It does not remove the run: its record, its logs, and the list of files it produced all stay, with each file marked as deleted.

A few things to know:

• Only the run's owner can do it, and only once the run has finished. Cancel a running run first.
• It cannot be undone.
• Deletion is not instant. Outputs are removed within a few minutes, and the run page shows the deletion as in progress until then.
• If you shared the run, the people you shared it with can no longer download its outputs. They see that the outputs were deleted.

What you are charged for🔗

A run's page shows how much storage its outputs occupy, and the runs list shows your total across all your runs. Deleted and expired outputs are not counted. Files you uploaded through the data catalog are not counted here either; they are your assets rather than a run's outputs, and retention never deletes them.

When a size has not been reported yet (older runs, mostly), the figure is shown as "at least" the known amount and fills in over time.

What is never deleted🔗

• Logs. Retention removes output files only.
• Uploaded assets. A pipeline that passes an uploaded file straight through lists that file among its outputs, but it is still your asset and stays exactly as long as you keep it in the catalog.
• The run itself. Its record, status and file listing remain after its outputs are gone.

A recipe for large jobs🔗

For a pipeline that reads a lot, produces large intermediates, and writes its results somewhere else:

retention:
  intermediates: drop
  expires: 7d

Intermediates disappear as soon as the run succeeds. The results stay for a week, long enough to check them, and then go too. If the run fails, everything it produced is there for a week while you debug.