data
The data collection imports data into Spade pipelines. It provides two kinds of blocks:
- Generic storage blocks built on Apache OpenDAL that read, write, list, and stat objects on any supported storage backend (S3, GCS, Azure Blob, HTTP(S), SFTP, WebDAV, Google Drive, OneDrive, Dropbox, and the local filesystem). The backend is inferred from the URI scheme.
- Domain-specific blocks that know how to download particular public datasets (US Census, USDA, USGS, Natural Earth, OpenStreetMap, NLCD, NHD, PRISM, SSURGO, FIA).
All blocks in this collection declare network: true because they fetch data from the internet.
Generic storage blocks🔗
| Block | Kind | Description |
|---|---|---|
data.read | standard | Fetch a single object from any supported backend |
data.read_collection | standard | Fetch multiple objects matching a prefix or glob |
data.write | standard | Write a single local file to a remote backend |
data.write_collection | standard | Write a collection of local files to a remote prefix |
data.list | standard | List objects under a URI prefix |
data.stat | standard | Fetch metadata for a single object |
US Census🔗
| Block | Kind | Description |
|---|---|---|
data.census_acs | standard | Query the US Census ACS Data API |
data.census_tiger | standard | Download Census TIGER/Line shapefiles |
Public geospatial datasets🔗
| Block | Kind | Description |
|---|---|---|
data.naturalearth_raster | standard | Download Natural Earth raster datasets |
data.naturalearth_vector | standard | Download Natural Earth vector datasets |
data.nhd | standard | Download USGS National Hydrography Dataset by HUC |
data.nlcd | standard | Download National Land Cover Database rasters |
data.osm_extract_pbf | standard | Download OpenStreetMap extracts in .osm.pbf format |
data.osm_extract_shp | standard | Download OpenStreetMap extracts as shapefiles |
data.prism | standard | Download PRISM climate rasters |
data.ssurgo | standard | Download USDA SSURGO soils data |
data.usgs_3dep | standard | Fetch USGS 3DEP elevation tiles overlapping an AOI |
data.fia | standard | Download USFS Forest Inventory and Analysis (FIA) data |
Installation🔗
spade install file:///path/to/blocks/datadata.read
Fetch a single object from any supported storage backend.
Read →data.read_collection
Fetch multiple objects matching a URI prefix or glob.
Read →data.write
Write a single local file to any supported storage backend.
Read →data.write_collection
Write every file in a local collection to a remote prefix.
Read →data.list
List objects under a URI prefix.
Read →data.stat
Fetch object metadata for a single URI.
Read →data.census_acs
Query the US Census ACS Data API and write the result as CSV.
Read →data.census_tiger
Download US Census TIGER/Line shapefiles.
Read →data.naturalearth_raster
Download Natural Earth raster datasets.
Read →data.naturalearth_vector
Download Natural Earth vector datasets.
Read →data.nhd
Download USGS National Hydrography Dataset data by watershed.
Read →data.nlcd
Download National Land Cover Database rasters.
Read →data.osm_extract_pbf
Download OpenStreetMap extracts in .osm.pbf format.
Read →data.osm_extract_shp
Download OpenStreetMap extracts as Geofabrik shapefiles.
Read →data.prism
Download PRISM climate rasters for a date range.
Read →data.ssurgo
Download USDA SSURGO soils data.
Read →data.usgs_3dep
Fetch USGS 3DEP elevation tiles overlapping an AOI.
Read →data.fia
Download USFS Forest Inventory and Analysis (FIA) data.
Read →