models

The models collection scores fitted models and fits linear models on tables too large for memory. It is written in Rust and reads and writes one batch at a time, so its memory does not depend on the number of rows: 20 million rows score in about 66 MB and fit in about 50 MB.

A fitted model travels between blocks as a SpadeModel file: a small archive holding the model's description -- its inputs, how they become design columns, its coefficients -- and, for tree models, an ONNX graph. Nothing in it runs code when loaded.

• models.predict scores a table with any SpadeModel: from stats.lm / stats.glm, from the sae blocks, from the ml blocks, or from this collection. Predictions are on the response scale, with optional prediction or confidence intervals for linear and glm models; classifiers write a label and one probability per class. The table is checked against the model before any row is scored.
• models.linear and models.glm fit least squares and generalised linear models by streaming. They expand formulas and code categoricals exactly as R does and are tested to fit the models stats.lm and stats.glm fit, so a pipeline can move from the in-memory R blocks to these without its model changing. They do not offer cross-validation or formula nesting (/, %in%).

ONNX Runtime. Models from the ml collection carry an ONNX graph, and models.predict scores it with ONNX Runtime, loaded at run time from the path in ORT_DYLIB_PATH (libonnxruntime.so, version 1.30). The worker image provides it. For local runs, set the variable -- for example to the library bundled with the ml collection's Python environment -- or download the release from the ONNX Runtime project. Linear, glm and Fay-Herriot models need no runtime.

All blocks in this collection declare network: false.

Blocks🔗

Installation🔗

spade install file:///path/to/blocks/models