Introduction

infrastore is a Rust library for managing time-series data in power-systems and energy simulations. It separates persistence into two concerns: numerical arrays are stored in HDF5, while the metadata that associates each array with an owning component lives in SQLite. Identical arrays are stored once and shared through content addressing.

The library ships native Rust, Python (via PyO3), and Julia (via a C ABI) interfaces, plus the infrastore command-line tool and a read-only gRPC server with a Rust client.

flowchart TB
    subgraph clients["Language Interfaces"]
        RUST["Rust<br/>(native crate)"]
        PY["Python<br/>(PyO3 wheel)"]
        JL["Julia<br/>(C ABI)"]
        CLI["infrastore<br/>(CLI)"]
    end

    subgraph core["infrastore-core"]
        STORE["Store"]
        STORE --> NC[("HDF5<br/>arrays")]
        STORE --> SQL[("SQLite<br/>metadata")]
    end

    subgraph remote["Remote Access"]
        SRV["gRPC server<br/>(read-only)"]
        RC["Rust client"]
    end

    RUST --> STORE
    PY --> STORE
    JL --> STORE
    CLI --> STORE
    SRV --> STORE
    RC -->|"gRPC"| SRV

    style RUST fill:#4a9eff,color:#fff
    style PY fill:#17a2b8,color:#fff
    style JL fill:#9558b2,color:#fff
    style CLI fill:#fd7e14,color:#fff
    style STORE fill:#28a745,color:#fff
    style NC fill:#28a745,color:#fff
    style SQL fill:#28a745,color:#fff
    style SRV fill:#ffc107,color:#000
    style RC fill:#ffc107,color:#000

Key Features

• One array, stored once — Arrays are addressed by a SHA-256 content hash, so identical series shared across components are written to disk a single time (content addressing)
• HDF5 for arrays, SQLite for metadata — Numerical data lands in a compact, chunked HDF5 file; queryable associations live in a catalog SQLite database (storage model)
• Feature-tagged associations — Each association carries an arbitrary map of typed features (int / float / bool / str) so multiple variants of a series can coexist under one owner
• Typed, N-dimensional arrays — Store f64, f32, i64, i32, u64, or bool values, with an optional per-step element shape (e.g. the coefficient tuple of a cost curve)
• Three language bindings — Use it from Rust, Python, or Julia with the same on-disk format
• An infrastore command-line tool — Load time series from CSV, and list, read, export, plot, diff, and inspect a store straight from a terminal, with table / json / jsonl / csv output (CLI guide)
• Read-only gRPC service — Serve a store over the network for remote readers, with optional API-key authentication
• Designed for power-systems data — The data model maps onto InfrastructureSystems.jl and infrasys owners, categories, and time-series concepts

Who Should Read This

Most people reading this are building a package on top of infrastore — infrasys or InfrastructureSystems.jl, or something like them — rather than calling it directly. That is the first row, and it is the one page to read if you read only one.

Next Steps

• Setting up? Start with Installation.
• Want the 60-second tour? Read the Quick Start for Python, Julia, or the CLI. Rust users can go straight to the Rust Developer Guide.
• Not sure which type your data is? See Choosing a Type.
• Want to understand how it works? Read the Architecture.
• Need exact bytes on disk? See the On-Disk File Format.

Embedding in a Parent Package

This guide is for developers of a package that uses infrastore as the time-series layer behind its own component model — the shipped examples are infrasys (Python) and InfrastructureSystems.jl (Julia). It collects the contracts such a package has to honor and the patterns both consumers already use, in one place. The per-language guides (Python, Julia, Rust) show each call; this page says which calls to reach for and why. The reasoning behind the trade-offs is in Design Choices.

What the Store Owns, and What You Do

infrastore is deliberately narrow. It stores arrays, associates each one with an owner, and records a few relationships between owners. Everything that makes those owners mean something lives in the parent package.

The store never parses a component, never walks a type tree, and never rescales a value. Every filter it offers takes concrete strings and exact scalars; a parent package that exposes anything richer builds it on top.

Map Your Model Onto the Catalog

A series is addressed by an owner plus a key (Data Model § Identity). Deciding how your objects project onto those fields is the first integration decision, and the one hardest to change later because it is written into every user's artifact.

owner_id is a stable i64. The store keys on it and never sees your object identities. If your components are identified by UUIDs (both shipped consumers are), you allocate an integer per component, persist that mapping alongside the store, and keep it stable across save/load — the store cannot reconstruct it. Component ids and supplemental-attribute ids are independent streams, so the same integer may name one of each; the owner_category (Component / SupplementalAttribute) is part of the owner identity and every owner-scoped call takes it.

owner_type is the concrete type name and nothing more. It is descriptive (not part of the uniqueness constraint) and the store has no view of subtyping, so a query for "every Generator" must expand the abstract type into its concrete subtypes yourself and pass each one — this is what get_all_subtype_names does on the InfrastructureSystems.jl side. An empty type list is a deliberate "none of these" and matches nothing.

name identifies, component_field describes. name is part of the key; component_field names the field on the owning component whose time-varying form these values are ("max_active_power"). They often coincide by convention, but a component may carry several series for one field — an actual and a forecast, several weather years — and only name plus features distinguishes them. Set component_field whenever your model knows it: it is the one descriptor that is also a filter, and a parent package that wants "every series that varies rating" gets it for free.

features are typed scalars (int / float / bool / str) and part of the key. A handful of names are reserved because consumers spread feature maps into keyword queries; the store refuses them on write.

units, quantity_kind, unit_system describe the values and affect neither identity nor storage. unit_system is a label the store never acts on — component_base means "per-unit against a base the owning component holds in your object graph", and converting back is your job. Unset means unspecified, which is not the same as natural_units; do not read one as the other. See Optional Descriptors.

time_reference records how the timestamps were spelled — an instant in UTC, an instant at a fixed offset, an instant in a named IANA zone, or a wall clock naming no instant. Each binding infers it from the input type, so nothing takes a new argument; a native Rust caller declares it, having no naive datetime type to infer from. It changes nothing about the stored instants, the grid, or either content hash — but the store does hold a series to its claim: a query bound must be spelled the way the series is, and a zoneless series cannot share one reader axis or one ranged bulk read with instant-bearing ones. See Time references.

application_data is yours. It is an opaque payload stored and returned verbatim, so a parent package can carry whatever it needs per association (a serialized type tag, a provenance record) without the store knowing. Because the store never validates it, version it yourself if its shape may change.

Exact Keys, Subset Filters

A filter matches features two ways, and a parent package that offers its own resolution must know which it asked for:

• features matches as a subset: a series matches when it carries every requested pair, whatever else it carries. This is the useful default for a listing — "every series tagged scenario=high".
• exact_features pins the whole set by its content hash. An existence check posed against a complete identity wants this one, or a sibling carrying an extra feature would answer yes about a series that does not exist. It is an equality on an indexed column, so it is also the cheaper of the two.

Both are on the same ListFilter, which every listing, existence probe, filtered removal and reader build takes.

InfrastructureSystems.jl resolves user queries by subset, so it cannot delegate that resolution to a keyed lookup; it lists with the filter and then decides what more than one match means. A parent package that inherits the same semantics should do the same, and treat ambiguity as its own error to raise — the store will happily return two rows.

Two more identity facts that surface in parent-package code:

• A DeterministicSingleTimeSeries is derived from a stored SingleTimeSeries with transform_single_time_series and reads back as a Deterministic. The tag stays visible in keys, metadata, and counts, and a Deterministic filter matches both. Your reads should not special-case it; your catalog displays may.
• Descriptors are outside the key, so two adds that differ only in units or application_data are a duplicate. Changing a descriptor means remove and re-add.

The Store Lifecycle Inside a System Object

Both consumers follow the same shape, and the API was shaped around it. The arrays stay on disk rather than in RAM — a system with hundreds of thousands of series does not fit — so the working store is on-disk from the start, in a scratch directory that lives as long as the system object. There is an in-memory backend (in_memory=True), but it holds every array in RAM and is meant for tests and small stores; see the note below.

Build: scratch directory, in-memory catalog

# Python
store = Store.create(scratch / "time_series.h5", catalog="memory")

# Julia
store = Store(; path=joinpath(scratch, "time_series.h5"), catalog=:memory)

catalog="memory" keeps the SQLite half in RAM and skips the per-commit journaling an attached catalog pays. That is the right trade for a store beside volatile in-process state: a crash loses the system under construction regardless, so durability of the scratch catalog buys nothing. Arrays still stream to the HDF5 file, so memory use does not grow with the data. The scratch directory holds no .sqlite until the first save — a half-artifact by design, and one the store refuses to open as attached later, because arrays without a catalog naming them are not a store. See Where the Catalog Lives.

Use in_memory=True (no file at all) for unit tests and for stores you know are small. It is not a substitute for the scratch-directory store in a real system.

Save: one call, atomic pair

store.persist_to(dest)          # Python

persist!(store, dest)           # Julia

persist_to writes both halves to uniquely named temporaries, fsyncs, and renames them into place, stamping the pair with a fresh generation so a save interrupted between the two renames is detected on the next open (MismatchedArtifact) rather than read as a valid store. This replaced the copy-both-files-by-hand dance both consumers once carried, including the close/reopen needed for HDF5's Windows file lock. Two things to carry into your own save path:

• A failed save may have destroyed the destination. The renames replace whatever was there. Retry from the still-live store rather than assuming the old artifact survived.
• Saving an attached store onto its own path is a no-op, while the in-memory-catalog case is the real work — the arrays are already at path and the save is what writes the catalog beside them. persist_catalog does only that half when the arrays are already where they belong, and is a checkpoint rather than a mode switch: the catalog stays in RAM afterwards.

Load for editing: always a copy

store = Store.open_copy(src, scratch / "time_series.h5", catalog="memory")   # Python

store = open_copy(src, joinpath(scratch, "time_series.h5"); catalog=:memory)  # Julia

open defaults to read-write in every binding, and a read-write open on a user's artifact is the one way this library will damage a file they care about: HDF5 has no journal and no repair tool, so an interrupted in-place write is unrecoverable. open_copy copies both halves and opens the copy; the original is only replaced by the final atomic rename of the next persist_to. Both consumers did this by hand before the call existed, and a test in infrasys asserts the loaded directory differs from the source — keep an assertion like that, because the copy is load-bearing.

For a read-only load (a viewer, a reporting script), open(path, read_only=True) is the right call: nothing is copied and any mutation raises ReadOnlyStoreError.

Create: refuse to clobber

Store.create on a path that already holds either half raises StoreExists. The failure mode it prevents is a re-run build script producing an empty array file paired with last week's catalog — a store that opens cleanly and has nothing behind any row. Pass overwrite=True (overwrite=true in Julia) only on a path your package owns and means to discard. See Protecting a Saved Artifact.

Close, and move the pair together

Close the store explicitly when the system object is done (both bindings offer a context-manager / do-block form). The .h5 and .h5.sqlite files are one artifact: move, copy, and delete them together, and never ship one without the other — the paired generation stamp makes a lone half a MismatchedArtifact on open.

One writer, local disk

A Store handle is not thread-safe. The Python class is unsendable — touching it from a thread other than the one that created it raises — and the Julia one is unsynchronized, so concurrent calls from two tasks are undefined behavior; confine a store (and any reader built from it) to one thread or task, or guard every call with your own lock. On disk, assume a single writer, and keep a live store off network filesystems — HDF5's file lock is best-effort and silently absent on Lustre, GPFS, and NFS, and SQLite's WAL is unsafe there too. Build locally, then copy the finished artifact to shared storage. See One writer, and not on a network filesystem.

Writing: Batch, and Make Multi-Step Changes Atomic

Two mechanisms compose, and neither substitutes for the other:

• Bulk add (add_time_series_bulk / AddBatch + add_time_series_bulk!) commits a whole batch in one catalog transaction and takes the block-sized HDF5 write path. Series sharing a (dtype, element_shape, length, resolution) pack into one dataset whose chunks span every column, which is what makes the simulation read below fast — so land same-shaped series in the same batch. A loop of single adds outside a transaction is an order of magnitude slower and fills chunks one column at a time; inside one it is buffered and written as a block instead.
• Transactions (with store.transaction(): / transaction(store) do … end) make several operations succeed or fail together. Inside one, a removal is reversible; outside one it is not, because the array bytes are reclaimed immediately. A transaction holds the SQLite write lock until it ends, and does not batch anything by itself.

Values are immutable: there is no API to edit a value, slice, or column in place, in any binding. "Update this series" in a parent package is add the new array, remove the old one — inside a transaction if the user must never observe the gap. Content addressing makes the add cheap when the data did not actually change. See Design Choices.

Reading in a Simulation Loop

The layout is optimized for "every component at one timestamp", and the readers are the API for that access: build a StaticReader or ForecastReader once, then step it. A ForecastReader reads each distinct backing array once per step and fans it out to every component referencing it, so a forecast shared by a hundred components costs one decompression; entry_slot lets your own per-component work dedup the same way. The inverse access — one component's full history — is the slow direction, and read_by_ids over many ids is the right call when you need it, not a loop of read_by_id. See the per-language sections: Python, Julia, Rust. The concepts are in Readers.

Time

• Every stored instant is a whole number of milliseconds; the write path raises InvalidParameter rather than truncating, because the C ABI exchanges instants as Unix milliseconds while Python's datetime is microsecond. Quantize now() before storing it. Query bounds are unconstrained.
• Python requires timezone-aware datetimes everywhere and returns UTC; a naive value raises.
• Julia reads a bare DateTime as a wall clock, recording ZonelessReference(); with using TimeZones a ZonedDateTime is accepted anywhere a timestamp goes and converted to the instant it names. Reads still return a bare DateTime holding the instant, because InfrastructureSystems.jl destructures them.

A parent package that has a notion of local time therefore converts at its own boundary and stores instants. See Timestamp precision.

Associations Beyond Time Series

The catalog records two relationship tables that have nothing to do with time series: supplemental-attribute attachments (component ↔ attribute, with counts and grouped summaries) and parent/child edges (directed component ↔ component). Both hold only the relationship — bare ids and type names — so a parent package keeps the objects in its own graph and uses the store as the index. replace_owner / reassign renumbers a component in every catalog at once. Bulk inserts are all-or-nothing, removals return a count (matching nothing is 0, not an error), and the *_types filters take concrete type names, exactly like owner_type above. See Associations Between Entities.

Versions and Errors

Pin a minor range. The on-disk format is governed by DATA_FORMAT_VERSION; opening an artifact written by an incompatible version raises IncompatibleFormat rather than misreading it. The bindings track the workspace version, so a parent package depends on a compatible range (infrastore>=0.11,<0.12 in pyproject.toml; a [compat] entry in Project.toml) and bumps it deliberately. Do not cite core source line numbers in your own docs — they move.

Map the error taxonomy, do not flatten it. Every binding exposes the core's TimeSeriesError variants as distinct types (NotFound, DuplicateTimeSeries, DuplicateAssociation, InvalidParameter, ReadOnlyStore, StoreExists, MismatchedArtifact, IncompatibleFormat, Integrity, …). The ones a parent package typically translates into its own exceptions are NotFound and DuplicateTimeSeries (user-facing), and StoreExists / MismatchedArtifact / IncompatibleFormat (artifact-level, usually re-raised with the path). See Python exceptions and Julia errors.

Testing an Integration

• Use in_memory=True stores in unit tests; they exercise the same core.
• verify_integrity re-hashes every array and time axis the catalog references; is_empty is the cheap "nothing here" probe.
• The infrastore CLI reads any artifact your package writes. infrastore list and infrastore info are the fastest way to see what a consumer actually stored, and infrastore diff --against compares two artifacts by hash without reading arrays — a usable CI gate for "the rewrite produced the same store". See Use the infrastore CLI.
• To test unreleased core changes from a consumer checkout, install the binding into the consumer's environment (maturin develop --manifest-path crates/infrastore-py/Cargo.toml with the consumer's venv active; Pkg.develop(path=...) plus INFRASTORE_LIB for Julia).

Checklist

• Integer owner ids are allocated by the package, persisted with the system, and stable across save/load.
• Abstract-type queries expand to concrete owner_type names before reaching the store.
• Feature resolution knows whether it wants exact-key or subset semantics, and ambiguity is the package's error.
• The working store is on disk in a scratch directory with an in-memory catalog; in_memory stores are for tests.
• Save goes through persist_to; load-for-edit goes through open_copy; read-only loads pass read_only.
• Multi-series writes use bulk add; multi-step changes that must be atomic use a transaction.
• Timestamps are instants at millisecond precision, converted at the package boundary.
• The dependency pins a compatible minor range of infrastore.

Installation

Most users install a published package and need no build tools at all:

The Python wheels and the Julia binary artifact are prebuilt and self-contained. The Rust crates build HDF5 and zlib from vendored sources and link them statically, so they need cmake and a C compiler but no system HDF5. The same vendored, statically linked stack backs every channel, so the HDF5 version behind the on-disk format is pinned by infrastore rather than by the target environment.

The infrastore CLI

Download a prebuilt binary

Each tagged release attaches archives to the Releases page. The executables are linked statically against HDF5 and zlib, so there is nothing else to install.

Linux ships two archives because they serve different consumers. The executables are built against musl and linked statically, so they run on any distribution regardless of its glibc version — including an HPC login node much older than the build machine. The C library is built against glibc instead: it gets loaded into a running Julia or Python process, and a musl shared library there would put two C libraries in one address space.

VERSION=v0.14.0    # pick a release from the Releases page
BASE=https://github.com/NatLabRockies/infrastore/releases/download/$VERSION
curl -fsSLO $BASE/infrastore-aarch64-apple-darwin.tar.gz
tar xzf infrastore-aarch64-apple-darwin.tar.gz
./infrastore --version

Move infrastore onto your PATH to finish. Every archive carries a .sha256 sidecar if you want to verify the download first:

curl -fsSLO $BASE/infrastore-aarch64-apple-darwin.tar.gz.sha256
shasum -a 256 -c infrastore-aarch64-apple-darwin.tar.gz.sha256   # sha256sum -c on Linux

macOS. The binaries are not notarized, so Gatekeeper blocks the first run of a downloaded executable. Clear the quarantine flag with xattr -d com.apple.quarantine ./infrastore.

Install from crates.io

cargo install infrastore-cli      # installs the `infrastore` binary

This compiles HDF5 from vendored sources, so it needs cmake and a C compiler (see Build Prerequisites) and takes a few minutes on the first build.

Parquet support (export -f parquet, add --parquet) is on by default, which is what pulls in the Arrow dependency tree. To leave it out -- a smaller binary and a shorter build, at the cost of those two flags:

cargo install infrastore-cli --no-default-features --features vendored

That binary still accepts the flags and names the feature to rebuild with, rather than reporting parquet as an unknown format. The Arrow tree reaches only the CLI: infrastore-core, the Python wheel, and the FFI cdylib never link it.

Julia

InfraStore.jl is registered in the Julia General registry:

using Pkg
Pkg.add("InfraStore")

The package does not link a system HDF5 or HDF5_jll. Its Artifacts.toml points at the libinfrastore_ffi tarball attached to the matching GitHub Release, so Pkg.add downloads a prebuilt, statically linked library for the platform — Linux x86_64 and aarch64 (glibc), macOS x86_64 and Apple Silicon, and Windows x86_64 — and nothing else needs installing. To run against a locally built library instead (a working tree, or a platform outside that list), set INFRASTORE_LIB; it takes precedence over the artifact. The Julia guide has both recipes, and Releasing explains why the binary is self-hosted rather than a JLL.

The rest of this page covers building the workspace from a checkout.

Build Prerequisites

A Rust toolchain, cmake, a C compiler, and protobuf for the gRPC codegen. The Python and Julia bindings additionally need a Python interpreter (3.11+) or Julia (1.11+).

brew install cmake protobuf maturin              # macOS
sudo apt-get install cmake protobuf-compiler     # Linux (Debian / Ubuntu)

The first build compiles HDF5 from source — a few minutes — and then caches the result.

Do not set HDF5_DIR. The vendored build forwards it to cmake as HDF5_ROOT while still requesting static libraries, which fails against a shared-only install. To build against system libraries instead, turn vendoring off with --no-default-features and install them (brew install hdf5 / apt-get install libhdf5-dev).

Because hdf5-metno-sys declares links = "hdf5", Cargo's feature unification makes the vendored-versus-system choice all-or-nothing across the whole dependency graph — an individual crate cannot opt out on its own.

Rust Toolchain

The workspace targets edition 2024 and declares an MSRV of Rust 1.94 — that is the oldest toolchain it is guaranteed to build on. (rust-version in the root Cargo.toml is the authority; the repo pins no rust-toolchain file, and CI builds on stable.)

rustup update stable

Build the Workspace

git clone https://github.com/NatLabRockies/infrastore
cd infrastore
cargo build --workspace --all-features
cargo test --workspace --all-features
cargo clippy --workspace --all-targets --all-features -- -D warnings

The workspace Cargo config (.cargo/config.toml) sets macOS linker flags so cargo build --workspace can link the PyO3 cdylib without maturin. On Linux those flags are inert.

Build the Native Library

InfraStore.jl and any other C consumer load the C ABI cdylib. Building it is the one target most people need out of a checkout:

cargo build -p infrastore-ffi --release
# -> target/release/libinfrastore_ffi.{dylib,so,dll}
# Regenerates the C header at crates/infrastore-ffi/include/infrastore.h

Point consumers at it with INFRASTORE_LIB, which takes precedence over the artifact Pkg installed:

export INFRASTORE_LIB=$PWD/target/release/libinfrastore_ffi.dylib  # .so on Linux

Add it to your shell profile to make it permanent. The Python wheel is built separately with maturin — see the Python guide.

Crates in the Workspace

Next Steps

• Build a store and round-trip a series in the Python, Julia, or CLI Quick Start.
• Read the developer guide for your language: Python · Julia · Rust · CLI.
• Stand up the gRPC server.

Quick Start (Python)

This walkthrough creates an in-memory store, adds a SingleTimeSeries, and reads it back — the shortest path to a working round-trip. It assumes the infrastore wheel is installed in the active environment; if import infrastore fails, see Integrate with Python.

A Minimal Round-Trip

from datetime import datetime, timedelta, timezone

import numpy as np
from infrastore import OwnerCategory, SingleTimeSeries, Store

# `in_memory=True` means no filesystem I/O. Pass `path=` instead to write a
# HDF5 file plus its SQLite catalog.
store = Store.create(in_memory=True)

# The name and the units live on the series object, not on `add_time_series`.
ts = SingleTimeSeries(
    datetime(2024, 1, 1, tzinfo=timezone.utc),  # initial timestamp (timezone-aware)
    timedelta(hours=1),                         # resolution
    np.arange(24, dtype=np.float64) + 100,      # 24 hourly values
    "load",                                     # name
    units="MW",                                 # optional, like every descriptor
)

# The owner is identified by an integer id, an owner type, and a category.
# Features are optional.
series_id = store.add_time_series(
    owner_id=42,
    owner_type="Generator",
    owner_category=OwnerCategory.Component,
    time_series=ts,
    features={"model_year": 2030},
)

got = store.read_by_id(series_id)
print(f"read {got.length} values @ {got.resolution} from {got.initial_timestamp}")
# read 24 values @ PT1H from 2024-01-01 00:00:00+00:00
assert np.array_equal(np.asarray(got.data), np.asarray(ts.data))

A read hands back the values and the timeline side by side. To fuse them into a dataframe, convert the series to a pyarrow.Table with to_arrow() and hand it straight to Polars — the two-column table is exactly a dataframe's shape, so the conversion is zero-copy and needs no glue:

import polars as pl

df = pl.from_arrow(got.to_arrow())
print(df.head(3))
# shape: (3, 2)
# ┌─────────────────────────┬───────┐
# │ timestamp               ┆ value │
# │ ---                     ┆ ---   │
# │ datetime[ms, UTC]       ┆ f64   │
# ╞═════════════════════════╪═══════╡
# │ 2024-01-01 00:00:00 UTC ┆ 100.0 │
# │ 2024-01-01 01:00:00 UTC ┆ 101.0 │
# │ 2024-01-01 02:00:00 UTC ┆ 102.0 │
# └─────────────────────────┴───────┘

print(df.group_by_dynamic("timestamp", every="6h").agg(pl.col("value").mean()))
# 102.5, 108.5, 114.5, 120.5 — one row per six-hour block

The timestamp column arrives typed in the series' own spelling (datetime[ms, UTC] here; an IANA zone or no zone at all for a series written that way), so Polars' time-aware operations work without you relabelling anything. The descriptive attributes ride along in got.to_arrow().schema.metadata — name, units, resolution, and the rest — which Polars does not carry onto the dataframe, so read them off the table when you need them.

to_arrow() needs pyarrow, which the wheel does not install by default: pip install 'infrastore[arrow]'.

What Just Happened

1. Store.create(in_memory=True) built a store backed by an in-memory array backend and an in-memory SQLite metadata database.
2. add_time_series hashed the array, wrote it to the backend (deduplicating on the hash), and recorded a catalog association filed under (owner_id, owner_category, type, name, resolution, interval, features). It returned that row's id — the handle to record in your own object model, and what every read and removal takes from here on.
3. read_by_id(series_id) looked up the row by primary key, read the array back by its content hash, and reconstructed a SingleTimeSeries.

The array is any NumPy array whose dtype is float64, float32, int64, int32, int16, int8, uint64, uint32, uint16, uint8, or bool — whatever you pass round-trips unchanged. Shapes beyond (length,) attach a per-step element shape, such as the coefficient tuple of a cost curve.

Slice and List

Pass a (start, end) tuple of datetimes to read a window instead of the whole series (end is exclusive):

(window,) = store.read_by_ids_range(
    [series_id],
    (
        datetime(2024, 1, 1, 6, tzinfo=timezone.utc),
        datetime(2024, 1, 1, 12, tzinfo=timezone.utc),
    ),
)
print(window.length)   # 6

list_metadata is how you find a series you did not just write: it returns one dict per catalog row, filtered by any combination of arguments, and each row carries the id to read it by:

for m in store.list_metadata(owner_id=42):
    print(m["name"], m["resolution"], m["units"], m["features"])
# load PT1H MW {'model_year': 2030}

Writing to Disk

Swap the constructor to persist:

store = Store.create(path="system.h5")
# ... add_time_series ...
store.flush()   # sync buffered HDF5 writes to disk

This produces two files that travel together:

• system.h5 — the HDF5 file holding the arrays.
• system.h5.sqlite — the catalog holding the metadata associations.

Reopen them later with Store.open("system.h5", read_only=True).

Next Steps

• Work through the Python Developer Guide for forecasts, bulk reads, associations, and error handling.
• Understand the Data Model: owners, keys, and features.
• Browse the full Python API reference.

Quick Start (Julia)

This walkthrough creates an in-memory store, adds a SingleTimeSeries, and reads it back — the shortest path to a working round-trip. It assumes InfraStore.jl can find the native library; if the first store call errors, see Integrate with Julia.

A Minimal Round-Trip

using Dates, InfraStore

# `in_memory=true` means no filesystem I/O. Pass `path=` with `in_memory=false`
# to write an HDF5 file plus its SQLite catalog.
store = Store(in_memory=true)

# The name and the units live on the series struct, not on `add_time_series!`.
ts = SingleTimeSeries(
    DateTime(2024, 1, 1),    # initial timestamp
    Hour(1),                 # resolution
    collect(100.0:123.0),    # 24 hourly values
    "load";                  # name
    units = "MW",            # optional, like every other descriptor
)

# The owner is identified by an integer id, an owner type, and a category.
# Features are optional.
id = add_time_series!(
    store,
    42,             # owner_id
    "Generator",    # owner_type
    Component,      # owner_category
    ts;
    features = Dict("model_year" => 2030),
)

got = read_by_id(store, id)
println("read $(length(got)) values @ $(got.resolution) from $(got.initial_timestamp)")
# read 24 values @ 3600000 milliseconds from 2024-01-01T00:00:00
@assert got.data == ts.data

What Just Happened

1. Store(in_memory=true) built a store backed by an in-memory array backend and an in-memory SQLite metadata database.
2. add_time_series! hashed the array, wrote it to the backend (deduplicating on the hash), and recorded a catalog association filed under (owner_id, owner_category, type, name, resolution, interval, features). It returned that row's id — the handle to record in your own object model, and what every read and removal takes from here on.
3. read_by_id(store, id) looked up the row by primary key, read the array back by its content hash, and reconstructed a SingleTimeSeries. Note that resolution comes back as a Millisecond.

features is serialized to JSON, so its values must be JSON scalars (Int, Float64, Bool, String). The data is any AbstractArray of Float64, Float32, Int64, Int32, UInt64, or Bool; dimensions beyond the first attach a per-step element shape, such as the coefficient tuple of a cost curve.

Finding a Series You Did Not Just Write

The store splits identify from act. list_metadata is the identify half — it answers which series exist and hands back the id that addresses each — and every read and removal takes that id. A caller that records ids in its own object model does the first half once and skips it from then on:

row = only(list_metadata(store; owner_id = 42, name = "load", resolution = Hour(1)))
got = read_by_id(store, row.id)

for m in list_metadata(store; owner_id = 42)   # Vector{TimeSeriesMetadata}
    println(m.name, " ", m.resolution, " ", m.units)
end
# load 3600000 milliseconds MW

list_metadata matches features as a subset; pass exact_features when you mean the whole set. There is deliberately no separate attribute-to-id resolver — a caller that wants exactly one row poses the filter and checks that it got one, which is what only does above.

Writing to Disk

Swap the constructor to persist. The do-block form closes the store on exit, including on throw:

Store(in_memory=false, path="system.h5") do store
    add_time_series!(store, 42, "Generator", Component, ts)
    flush!(store)   # sync buffered HDF5 writes to disk
end

This produces two files that travel together:

• system.h5 — the HDF5 file holding the arrays.
• system.h5.sqlite — the catalog holding the metadata associations.

Reopen them later with open_store("system.h5"; read_only=true), which has a do-block form too:

open_store("system.h5"; read_only=true) do store
    rows = list_metadata(store; owner_id = 42, owner_category = Component)
    series = read_by_id(store, rows[1].id)
end

Next Steps

• Work through the Julia Developer Guide for forecasts, readers, associations, and error handling.
• Understand the Data Model: owners, ids, and features.
• Browse the full Julia API reference.

Quick Start (CLI)

The infrastore binary reads and writes an on-disk store directly — no server, no binding, no Python or Julia environment. This is the shortest path from a CSV to a store you can inspect.

Get the binary from the Releases page (the executables are statically linked, so there is nothing else to install) or with cargo install infrastore-cli. See Installation for the per-platform archives.

A Minimal Round-Trip

Two files: the values, and a descriptor saying what they mean.

# load.csv
value
100.0
101.5
103.0
104.2

{
  "owner_id": 42,
  "owner_type": "Generator",
  "owner_category": "Component",
  "name": "load",
  "type": "SingleTimeSeries",
  "element_type": "f64",
  "units": "MW",
  "csv": "load.csv",
  "initial_timestamp": "2024-01-01T00:00:00Z",
  "resolution": "PT1H"
}

Save that as load.json, then add and read it back:

infrastore --store demo.h5 add --descriptor load.json
infrastore --store demo.h5 list
infrastore --store demo.h5 get --owner-id 42 --name load

╭────┬───────┬────────────┬───────────┬──────────────────┬──────┬──────────┬──────────────┬────────────┬──────────┬────────┬───────┬──────────────╮
│ ID │ Owner │ Owner Type │ Category  │ Type             │ Name │ Features │ Element Type │ Resolution │ Interval │ Length │ Units │ Hash         │
├────┼───────┼────────────┼───────────┼──────────────────┼──────┼──────────┼──────────────┼────────────┼──────────┼────────┼───────┼──────────────┤
│ 1  │ 42    │ Generator  │ Component │ SingleTimeSeries │ load │ -        │ f64          │ PT1H       │ -        │ 4      │ MW    │ 09ec58683de3 │
╰────┴───────┴────────────┴───────────┴──────────────────┴──────┴──────────┴──────────────┴────────────┴──────────┴────────┴───────┴──────────────╯

╭───────────────────────────┬───────╮
│ timestamp                 │ value │
├───────────────────────────┼───────┤
│ 2024-01-01T00:00:00+00:00 │ 100   │
│ 2024-01-01T01:00:00+00:00 │ 101.5 │
│ 2024-01-01T02:00:00+00:00 │ 103   │
│ 2024-01-01T03:00:00+00:00 │ 104.2 │
╰───────────────────────────┴───────╯

What Just Happened

• demo.h5 and demo.h5.sqlite were both created. They are one artifact: the arrays are in the HDF5 file, the catalog row in the SQLite one. Move, copy, and delete them together.
• The values came from the CSV; everything else came from the descriptor. A flat grid of numbers fits a CSV; an owner, a resolution, and a feature map do not.
• The header row is required. add reads it to tell a hand-written value-only file from one infrastore export wrote, so a file whose first row is data is rejected rather than silently losing that row.
• The store assigned id 1. That id is how every later read and removal addresses the series — see Association IDs.
• Timestamps must name an instant. 2024-01-01T00:00:00Z does; a bare 2024-01-01 00:00:00 does not, and is rejected. Pass --assume-timezone UTC to say what a zoneless file meant.

Print a starting descriptor for any of the five writable types with infrastore template:

infrastore template NonSequentialTimeSeries > outages.json

Look Around

infrastore --store demo.h5 names                    # distinct series names
infrastore --store demo.h5 get --name load --plot   # a terminal sparkline
infrastore --store demo.h5 store-info               # format version, compression, catalog state
infrastore --store demo.h5 -f json list             # every read command honors -f

Next Steps

• The whole workflow — wide CSVs, forecasts, charts, associations, diff and merge — is in the CLI Developer Guide.
• Every flag and the descriptor schema: CLI Reference.
• Doing this from a program instead: Python · Julia.

Explanation

This section explains how infrastore is put together and why. It is understanding-oriented: read it to build a mental model, not to accomplish a specific task. For the calls that do the work see the Developer Guides; for exhaustive listings see the Reference.

• Architecture — The crates, the two-file storage split, and how the language bindings sit on top of a single core.
• Design Choices — What infrastore optimizes for and why, written for developers of parent packages like IS.jl and infrasys.
• Time-Series Types — The seven types, which one your data wants, and the vocabulary they share: periods, timestamp precision, typed arrays.
• Data Model — Owners, features, identity, association ids, and the associations between catalog entities.
• Time References — How a series' timestamps are spelled, what that does and does not change, and why a named zone is safe.
• Readers — The columnar bulk-read surface: why it exists, what one timeline per reader means, and when to reach for something else.
• Storage Model — Why arrays go to HDF5 and metadata goes to SQLite, and how the two stay consistent.
• Content Addressing — How arrays are hashed, deduplicated, and verified.
• Language Bindings — How the Python, Julia, and gRPC interfaces wrap the Rust core.

Architecture

infrastore is a Rust workspace with one core library and a ring of interface crates around it. Every interface — native Rust, Python, Julia, the infrastore CLI, and the gRPC server — ultimately drives the same Store type in infrastore-core, and every interface reads and writes the same on-disk format.

Crate Layout

flowchart TB
    subgraph ifaces["Interface crates"]
        PY["infrastore-py<br/>PyO3 wheel"]
        FFI["infrastore-ffi<br/>C ABI cdylib"]
        SRV["infrastore-server<br/>gRPC server + Rust client"]
        CLI["infrastore-cli<br/>infrastore binary"]
    end

    PYMOD["infrastore<br/>(Python module)"]
    JL["InfraStore.jl<br/>(Julia package)"]
    PROTO["infrastore-proto<br/>protobuf + tonic"]
    PARQ["infrastore-parquet<br/>Parquet export/import"]

    subgraph core["infrastore-core"]
        STORE["Store"]
        META["MetadataStore<br/>(SQLite)"]
        BACK["StorageBackend<br/>(trait)"]
        NC["Hdf5Backend"]
        MEM["MemoryBackend"]
        STORE --> META
        STORE --> BACK
        BACK --> NC
        BACK --> MEM
    end

    PY --> STORE
    FFI --> STORE
    SRV --> STORE
    CLI --> STORE
    PYMOD -->|"import"| PY
    JL -->|"ccall"| FFI
    SRV --> PROTO
    CLI --> PARQ
    PARQ --> STORE

    style STORE fill:#28a745,color:#fff
    style META fill:#28a745,color:#fff
    style BACK fill:#1e7e34,color:#fff
    style NC fill:#1e7e34,color:#fff
    style MEM fill:#1e7e34,color:#fff
    style PY fill:#17a2b8,color:#fff
    style PYMOD fill:#17a2b8,color:#fff
    style FFI fill:#9558b2,color:#fff
    style JL fill:#9558b2,color:#fff
    style SRV fill:#ffc107,color:#000
    style PROTO fill:#ffc107,color:#000
    style CLI fill:#fd7e14,color:#fff
    style PARQ fill:#fd7e14,color:#fff

The Core: Store

Store is a thin orchestration layer that composes two collaborators:

• A StorageBackend that holds the numerical arrays, addressed by content hash.
• A MetadataStore (SQLite) that holds the associations between owners and arrays.

flowchart LR
    CALL["add_time_series(...)"] --> HASH["array_hash()"]
    HASH --> PUT["backend.put_array(hash, data)"]
    HASH --> INS["MetadataStore::insert(association)"]
    PUT --> NC[("HDF5")]
    INS --> SQL[("SQLite")]

    style CALL fill:#4a9eff,color:#fff
    style HASH fill:#6f42c1,color:#fff
    style PUT fill:#28a745,color:#fff
    style INS fill:#28a745,color:#fff
    style NC fill:#1e7e34,color:#fff
    style SQL fill:#1e7e34,color:#fff

The backend is chosen behind the StorageBackend trait. There are two implementations:

• MemoryBackend — arrays in a hash map; selected when in_memory = true. No filesystem I/O.
• Hdf5Backend — arrays in an HDF5 file; selected when a path is given.

Because the seam is a trait, the metadata layer, the hashing, and every binding are identical no matter where the arrays live. Tests run against the memory backend; production uses HDF5.

Why Two Files

Numerical arrays and their descriptive metadata have different access patterns. Arrays are large, append-mostly, and read by content; metadata is small, frequently queried, and benefits from indexes and transactions. infrastore puts each where it is strongest:

• Arrays → HDF5. Chunked, compressed, columnar storage that the whole HDF5 ecosystem reads.
• Metadata → SQLite. A queryable, transactional catalog at <path>.h5.sqlite.

The Storage Model page covers the trade-offs and the consistency protocol that keeps the two files in agreement.

Read Paths: Local and Remote

Writes always require local filesystem access — they go straight through a Store. Reads can happen two ways:

flowchart LR
    subgraph local["Local process"]
        APP["Your code"] --> STORE["Store"]
    end
    subgraph network["Over the network"]
        APP2["Reader"] --> RC["RemoteClient"]
        RC -->|"gRPC"| GS["gRPC server"]
        GS --> STORE2["Store (read-only)"]
    end

    style STORE fill:#28a745,color:#fff
    style STORE2 fill:#28a745,color:#fff
    style GS fill:#ffc107,color:#000
    style RC fill:#ffc107,color:#000

The gRPC server exposes a read-only subset of the API (list, get, keys, resolutions, counts, existence checks, integrity). It never writes. See Language Bindings and the gRPC Server guide.

Concurrency

Within a process, Hdf5Backend guards its HDF5 handle with a Mutex, so the storage backend itself is Send + Sync. The Store as a whole, however, is Send but not Sync: its MetadataStore wraps a single rusqlite::Connection, which is internally a RefCell and therefore cannot be shared between threads. In practice this means a Store can be moved to another thread, but sharing one across threads requires external synchronization — the gRPC server holds its store as an Arc<Mutex<Store>>, and the PyO3 binding marks the class unsendable so a Python Store stays on the thread that created it.

MetadataStore uses transactions for atomic multi-row writes. The library does not coordinate multiple processes writing the same file concurrently — a single writer owns the files at a time.

Within one process the rule is enforced: opening an artifact that another Store in the process already holds — read-only or not — fails with StoreInUse, and so do create_replacing, open_without_catalog, persist_to, and persist_arrays_to aimed at a held path. Each handle indexes the HDF5 file's packed columns once at open, and libhdf5 shares one file object between two opens of a file, so a second handle would read and write the wrong columns. Drop the handle before opening another.

Design Choices

infrastore is a foundation library. End users rarely call it directly — they reach it through a parent package such as InfrastructureSystems.jl (IS.jl) or infrasys, which embed infrastore to persist the time-series data behind their component models. This page records the decisions that shape the API and the on-disk format, and the reasoning behind them, so that developers of those parent packages understand what infrastore optimizes for — and, just as importantly, what it deliberately does not. The practical counterpart — which calls a parent package should make, and in what order — is Embedding in a Parent Package.

Data Orientation: Optimize for Reading Every Component at One Timestamp

The decision. In the HDF5 file, SingleTimeSeries arrays that share a (dtype, element_shape, length, resolution) are packed as columns of one dataset: columns are series, rows are timesteps, and the HDF5 chunking spans the whole width — (rows, cols, *element_shape), one row unless the dataset is narrow enough that a single timestamp row would make an uneconomically small chunk (file format). A chunk therefore holds one timestamp, or a few consecutive ones, across every column. We optimize for reading all components' values at a given timestamp, and accept that reading one component's entire array is comparatively slow.

Why. The workload that matters is simulation. A production-cost or power-flow model steps through time and, at each step, needs the value of every generator, load, and branch for that one timestamp — a slice across series, not down one. With this layout that slice is one chunk read per dataset, and a sweep of them costs the same whatever the chunk's row count, since it visits every chunk anyway; the ForecastReader / StaticReader columnar surface is built directly on it. The inverse access — pulling one component's full history — has to touch every chunk band and is slow by design. That trade is deliberate: the simulation read path is the hot one, and it is the one parent packages hand to their users.

What this means for parent-package developers.

• Lay out bulk writes so that series sharing a shape land in the same dataset — that is what fills whole chunks in one pass and keeps the timestamp-slice read fast. See add_time_series_bulk and bulk_add.
• Do not build a user-facing feature whose common path is "read this one component's entire array" and expect it to be cheap. It works, but it is the slow direction. If a downstream workload genuinely needs that orientation, that is a signal to raise with infrastore, not to work around with many single-series reads.
• The orientation is a property of the packed HDF5 layout only. NonSequentialTimeSeries and the dense forecast types are stored as standalone per-array variables and do not participate in it.

Values are immutable. There is no API — in any binding, by design — to edit a single value, slice, row, or column of an array already in the store. A stored array is added or deleted as a whole; changing data means writing a new array (content-addressed, so unchanged neighbors are not rewritten) and deleting the old one. This falls out of the two priorities above. A chunk holds one timestamp across many series, so editing one value would force a read-modify-write of a whole chunk band — the slow direction turned into the write path. And because arrays are content-addressed, an array's identity is its bytes: mutating it in place would invalidate the hash that every association row and every dedup reference depends on. Parent packages that expose "update this value" to their users must implement it as replace-the-array, not edit-in-place.

Forecast Storage: Chunked So One Window Is Cheap

Dense forecasts (Deterministic, Probabilistic, Scenarios) do not use the packed, cross-component layout above — each forecast is stored as its own standalone array, [H, count, *E] for a Deterministic, where H is the horizon length and count the number of forecast windows. But reading one window at a time is a first-class access pattern (a simulation stepping the forecast timeline, one issue time per step), so the array is chunked in bounded blocks along the count axis rather than as one whole-array chunk. A window read then decompresses one block instead of the entire year of windows.

Two consequences a parent-package developer should know:

• A window sweep is cheap; naive per-window reads are not, unless you go through the reader. Because a chunk is the decompression unit, reading a single window still pulls its whole block. The ForecastReader sizes its in-memory cache to that same block width, so stepping the window timeline decompresses each block exactly once. Reach for ForecastReader (or read the whole array once and index it) rather than issuing an independent whole-array read per window — the latter re-decompresses overlapping data.
• Cross-component savings come from dedup, not packing. Where static series share storage by packing many components into one dataset, forecasts share it by content addressing: identical forecast arrays are stored once, and ForecastReader reads each unique array a single time and fans it out to every component that references it.

The block width is a write-time storage choice only — it reads transparently regardless of the width a store was written with, so it does not change the on-disk format version.

Split Arrays From Metadata

Numerical arrays live in HDF5 and metadata associations live in a companion SQLite catalog, because the two have opposite size, access, and mutation profiles and each format is strongest at one of them. The full rationale, the consistency ordering that keeps the two files in step, and the compaction behavior are covered in the Storage Model.

Content-Address and Deduplicate Arrays

Arrays are keyed by the SHA-256 of their contents, so two associations with identical data share one stored array and writes are idempotent on hash. This is what lets many components reference the same profile without duplicating storage, and it is why deletes are reference-counted. See Content Addressing.

Keep the Multi-Language Surface Consistent

infrastore-core is the single source of truth; the Rust, Python (PyO3), Julia (C ABI), CLI, and gRPC interfaces are thin wrappers over the same Store. A capability is not considered done until it behaves the same across the bindings that support it, and unsupported operations return an explicit error rather than silently changing semantics. This keeps a parent package free to move between bindings — for example, Julia via the C ABI and Python via the wheel — without the data model shifting underneath it. See Language Bindings.

Make Transactions Span Operations, Without Enlisting HDF5

Every mutating entry point is atomic on its own, and a bulk add commits a whole batch in one catalog transaction. Neither helps when several operations have to succeed or fail together — add a series and remove the one it replaces, or write a batch and derive a forecast from it. Store::begin_transaction opens a unit of work spanning any number of operations, and only its outermost commit makes anything durable.

The obstacle is that a store is two artifacts and only one of them has transactions. SQLite rolls back its own statements; HDF5 has nothing to enlist. Rather than trying to give HDF5 a transaction, the array store is made append-only for the transaction's duration, which content addressing makes cheap:

• Writes are recorded as they happen and removed on rollback. An array is recorded only if it was physically written — a write of content that already exists is a no-op on hash, so there is nothing to undo.
• Frees are deferred to the outermost commit. While the transaction is open, an array whose last association was removed must keep its bytes, because a rollback restores the rows that point at it. At commit the reference count is rechecked against the state the commit is about to make permanent, so a hash removed and re-added inside the same transaction is never freed.

That deferral is what makes removals reversible inside a transaction, which they are not outside one. It is the capability a caller cannot build for itself: a client-side undo log can re-insert a catalog row, but it cannot bring back array bytes the store already reclaimed.

Two consequences fall out of the mechanism rather than being designed in. Reads inside a transaction see its uncommitted writes, because they go through the same connection — so a binding needs no staging overlay to give a caller read-your-own-writes. And nesting is free: each level is a SQLite savepoint, so an inner failure unwinds only its own work and leaves the enclosing transaction usable.

Append-only also means "not yet written". A packed single add outside a transaction fills one slot of a thousand-column growth pool, and because that pool is chunked one timestamp row across every column, filling one column rewrites every chunk in it. Inside a transaction that write is owed to nobody until the outermost commit, so it is buffered per pool instead and the buffered arrays are written together with the same block writer a bulk add uses — at the commit, when a read needs one of them to have a physical position, or when the buffer hits one of the bounds below. A loop of single adds inside one transaction therefore produces the file one bulk add of the same items produces: same dataset names, same widths, same chunking. Nothing about the format changes; these are layouts the bulk path already wrote.

Two edges keep that from being a worse trade than the one it replaces. A block of one is not a block: a span holding a single array for a pool fills a growth-pool slot, because a dataset sized to one column is chunked (1, 1) and gives a scalar f64 series an eight-byte chunk per timestep, whose per-chunk overhead dwarfs the data — the same reason add_time_series_bulk sends a batch of one down the single-add path. That matters because "several operations atomic together" is most often a removal and its replacement, not an ingest. The same rule settles whether an irregular series packs with a cohort or stands alone: that is a bet on the axis being shared, and inside a span only the block knows the answer, so a cohort added one series at a time pools exactly as the bulk add of it does. And the buffer is bounded, per pool at the width the block writer spills a batch at and across every pool at a fixed byte ceiling; crossing either writes a block out early, which costs an extra dataset and nothing else. The per-pool bound is the chunk budget rather than the growth pool's thousand columns because a bulk add inside a transaction is buffered too, and a narrower cap gave it ten times the datasets — and a columnar read ten times the chunks — of the same add outside one.

One failure the mechanism cannot see coming is SQLite ending the transaction itself: a statement that fails on a full disk or an I/O error can roll the whole transaction back, savepoints included, and the store's bookkeeping does not learn of it. So the store reads it off the connection after the fact — nesting levels still recorded while the connection is back in autocommit — and from then on refuses every write and the commit until rollback_transaction discards the dead transaction, every level at once. The alternative, letting the next write open an implicit transaction of its own and commit on its own, would make part of the span durable behind the caller's back.

The costs are real and bound where this is worth using. A transaction holds the SQLite write lock until it finishes, so a concurrent writer on the same artifact blocks and then fails on its busy timeout. The buffer holds its not-yet-written arrays in memory — the same memory the equivalent bulk add allocates, bounded per pool by the width it spills at. And a transaction still does not replace batching for everything: feature-set dedup across a batch comes from bulk_add, which is also the direct spelling when the whole cohort is already in hand as a list. The two compose — batch each operation, and use a transaction when several of them must be atomic together.

Upgrade a Store In Place Rather Than Bricking It

DATA_FORMAT_VERSION used to be checked by strict equality on open, which made every bump a wall: "re-create the store." It was bumped six times between 0.12 and 0.19 and each one meant exactly that. The reason it had to be a wall is that the catalog DDL is CREATE TABLE IF NOT EXISTS — idempotent, so a new table or index lands on an existing store for free, but not version-agnostic: it will not alter a table that already exists, so a new column or a changed CHECK never reaches an old catalog at all.

This release replaces the wall with a three-tier compatibility model plus a migration ladder.

Two revisions, answering different questions

• DATA_FORMAT_VERSION describes the artifact as a whole and is stamped on the HDF5 root. It moves for anything that changes the meaning of bytes already on disk: the array layout, the dtype encoding, the timestamp encoding, a hash domain.
• CATALOG_SCHEMA_REVISION describes the SQLite catalog alone and lives in its schema_version table. Any catalog change the idempotent DDL cannot make to an existing table — a new column, a changed CHECK, a rebuilt table, a backfill — needs a CATALOG_SCHEMA_REVISION bump plus an append-only entry in MIGRATIONS.

MIN_UPGRADABLE_VERSION says how far back the ladder reaches. A stamp between it and DATA_FORMAT_VERSION is Upgradable; anything older, anything newer, and anything unparseable is Incompatible and is refused exactly as before. When a bump genuinely does strand older stores, MIN_UPGRADABLE_VERSION is raised to match it; when the ladder can absorb it, it is left alone.

A writable open upgrades; a read-only open reports

Opening a store for writing runs every migration above the catalog's recorded revision, in order, each in its own transaction. Opening it read-only cannot change anything, so it reports CatalogMigrationRequired — an error that names the remedy (open it once for writing) instead of a raw no such column from inside some later query. This is what a read-only consumer such as the gRPC server now sees against a store that has not yet been upgraded.

A catalog written by a newer build is CatalogTooNew and is refused in both directions. There is no downgrade path, and this build's DDL and ladder both describe an older shape.

The ordering that makes it safe

Three steps, and each one is load-bearing:

1. Open the HDF5 half and evaluate its version stamp. This must come first so a store too old to migrate reports IncompatibleFormat rather than a confusing SQLite error, and so a bad path does not leave a freshly created empty .sqlite behind. An upgradable stamp is noted, not yet rewritten.
2. Open the catalog, which is where the ladder runs.
3. Only if (2) succeeded, and only for a writable open, re-stamp the HDF5 half.

The two stamps cannot be written atomically together, so the order decides which half is ahead when something fails in between. Catalog-first leaves a migrated catalog under an older array-file stamp: an older build then opens the store and simply never writes a row of a type it does not know, which is harmless. The reverse would leave a store claiming the new format over an un-migrated catalog — the exact failure the ladder exists to eliminate.

Migration is not a save: the paired generation stamps that pair the two halves are carried across untouched.

Append-only, never edited

A landed MIGRATIONS entry is frozen. Stores in the wild have already run it, so editing it changes nothing for them and silently diverges the shape a fresh store gets from the shape an upgraded one gets. Add a new entry instead. For the same reason each migration carries a frozen snapshot of the table shape it produces rather than deriving it from the live DDL, which will keep moving.

Revision 1 is defined as whatever a pre-ladder build stamped — which is exactly what every existing store already says, since the schema_version table was seeded with a literal 1 and never read back. That is why the ladder needs no detection heuristic. It is deliberately not a claim about one particular table shape, though: nothing stamped a revision while the catalog was still moving, so 1 spans several, and a migration must tolerate any of them. The ladder starts there rather than trying to resurrect the 0.12–0.16 formats, which changed the meaning of bytes on disk and are still rejected outright.

The first rung

Revision 2 widens the time_series_associations time_series_type CHECK from BETWEEN 0 AND 5 to >= 0. TimeSeriesType::from_code is the real gate on that domain and runs on every write and every read, so the numeric bound bought nothing SQLite had to enforce — while turning the eventual appending of a seventh type into a table rebuild. Moving the domain onto the enum leaves the CHECK as a non-negativity test, which still refuses a corrupted value.

SQLite has no ALTER TABLE … DROP CONSTRAINT, so applying it is the table rebuild, and the rebuild is what makes the two subtle parts of the ladder concrete: the time_series_readable view has to be dropped first (SQLite refuses to drop a table a view still names), and the AUTOINCREMENT high-water mark has to be carried across by hand, because DROP TABLE takes the old table's sqlite_sequence row with it and the copy would restart the counter at max(id) — handing back an id a deleted row already used, which is the one thing AUTOINCREMENT is there to prevent.

Time-Series Types

infrastore stores seven time-series types. Three are static — a value per instant, on a grid, on explicit timestamps, or held forward from the last breakpoint — and four are forecasts, where each entry is a window of values issued at one time. This page covers what each one means, when to reach for it, and the vocabulary they share: periods, timestamp precision, and typed arrays.

For how a series is filed and addressed once stored — owners, features, identity, the association id — see the Data Model. For how the timestamps are spelled, see Time References.

Choosing a Type

The static types differ in one thing: what value the series yields at an instant you did not store. That question, not the shape of your input data, is what picks one.

Two follow-ups that come up every time:

• The value shape is a separate axis. A series of cost curves is not a new type — it is one of the types above with an element_type naming the curve. See the element axis.
• A grid you cannot fill densely is not automatically irregular. A SingleTimeSeries costs one value per step and packs into a shared dataset; the irregular types carry an explicit timestamp vector. Prefer the grid when there is one.

The Seven Types

All seven are present in the TimeSeriesType enum and in the metadata schema, and all seven can be read from every interface: the Rust core, the C ABI, Python, Julia, the infrastore CLI, and the gRPC server. The write paths differ, because the read-only gRPC server accepts none of them and one type is never written directly at all:

Every write path in the table is available in the Rust core, the C ABI, Python, Julia, and the CLI. No interface adds a DeterministicSingleTimeSeries directly: it only ever comes into existence by transforming a stored SingleTimeSeries.

DeterministicSingleTimeSeries is a storage-level view, and reads always return a Deterministic. This is by design in every binding: TimeSeriesData has no DeterministicSingleTimeSeries variant, and a read synthesizes the windowed Deterministic from the underlying static array without copying it. The DeterministicSingleTimeSeries tag stays visible in catalog surfaces — keys, metadata rows, counts, summaries — so callers can see which of their forecasts are synthetic, and can address, copy, or remove the association.

It is never something you must ask for. A request for Deterministic matches both storage forms — in reads, key resolution, catalog filters, and reader builds alike — so which one a store holds stays an internal detail. Requesting DeterministicSingleTimeSeries narrows to the derived form, which is how a caller audits what it has. This mirrors InfrastructureSystems.jl, where a Deterministic request lowers to both concrete type names.

See Forecasts below for how the four windowed types are laid out.

SingleTimeSeries

A SingleTimeSeries is an initial_timestamp, a resolution (a period), and an array of values:

value
  ^
  |        *
  |     *     *
  |  *           *
  +--+--+--+--+--+--> time
   t0 t0+r  ...   t0+(n-1)r

The timestamps are implied — sample i is at initial_timestamp + i * resolution — so only the values are stored.

NonSequentialTimeSeries

A NonSequentialTimeSeries pairs each value with an explicit UTC timestamp. Timestamps must be strictly increasing and their count must match the data length.

The timestamp vector is stored in the HDF5 file, content-addressed and shared: series sampled at the same instants — an outage schedule, a set of event times, a market timeline — hold one copy between them rather than one each. That shared vector is also the series' cohort: the values of every series on it are column-packed into one timestamp-major HDF5 dataset, exactly as SingleTimeSeries at one resolution are, so a StaticReader can sweep them a timestamp at a time. A series alone on its time axis keeps a standalone array instead — packing only pays once a cohort is several columns wide. See the storage model for the layout.

PersistentTimeSeries

A PersistentTimeSeries is a sparse step function: a strictly increasing vector of breakpoints plus one value each, where the value at an arbitrary instant is the one belonging to the greatest breakpoint at or before it.

value
  ^
  |            +------------------->
  |     +------+
  |  +--+
  |  ?
  +--+------+--+---------+---------->  time
     b0     b1 b2

Formally the values define a right-continuous step function:

• constant on [b_k, b_{k+1}) — a read between breakpoints returns the previous breakpoint's value;
• extending to +∞ past the last breakpoint — a read after the end returns the last value;
• undefined before the first breakpoint — a read there is an error, never a clamp. A value there was never declared, and inventing one would be a guess.

Because the function is total on [b_0, +∞), asking a series in hand for its value at an instant is spelled plainly: value_at in every binding (PersistentTimeSeries::value_at::<f64>(t) in Rust, with row_at for a shaped per-step element). It is not an approximation of a value — it is the value. Only the row that value came from sits earlier, which is what index_at and breakpoint_at report.

Structurally it is identical to a NonSequentialTimeSeries — same fields, same validation, same storage. The two even share arrays: a persistent series and an irregular one on the same breakpoints, dtype, element shape, and values occupy one content-addressed array in one nsts_… dataset, because PackGroup is keyed by the time axis and never by the series type. The difference is entirely in read semantics:

That is why it is a separate type rather than a read flag. "An irregular timeline has no value between its timestamps" is a guarantee NonSequentialTimeSeries's docs and error messages lean on; making it conditional would take it away from everyone.

The motivating data is a monthly fuel or gas price curve: a dozen breakpoints spanning a year, read at simulation timestamps that almost never coincide with one. Read as a NonSequentialTimeSeries that would error at nearly every step.

A time range slices on the step function's own terms. The returned series begins at the breakpoint in force at start, not the first breakpoint at or after it, so the result always defines a value at the start of the caller's window. A start before the first breakpoint is an error. The one exception is a window with no instants in it at all: a zero-width range (end == start) selects nothing, here as for every other type, and that includes a zero-width range before the first breakpoint, which is empty rather than an error.

Scalar-collapse policy belongs to the application, not the store. A consumer that needs to know whether a curve should be expanded to a full series or evaluated once at a midpoint carries that in application_data, the opaque package-owned payload the store never interprets. infrastore records breakpoints and values, and nothing else; there are no catalog columns for expansion policy and there will not be.

It is an infrastore-local extension, not a Sienna type, and does not travel in an OpenAPI document. The vendored sienna_schemas/TimeSeries/TimeSeriesAssociation.json is a oneOf over a closed set of six canonical types owned by the data layer, and there is no upstream schema for a seventh — so the wire form has no way to spell one. The export therefore omits persistent rows: an unfiltered export of a mixed store emits its six-type rows and drops these, and an export whose filter names the type is refused rather than answered with an empty array. The import refuses the type independently, since a document from elsewhere can still name it: every incoming row is checked against the schema its own time_series_type selects, and none selects this one.

Ask the catalog what an export leaves behind — list_metadata filtered to PersistentTimeSeries — and carry those series in the artifact itself, which holds them in full.

Reading a step function in a columnar sweep

A StaticReader over PersistentTimeSeries columns is the one place the "one timeline per reader" rule bends, and only because a step function makes it safe to: every column has a value at every instant from its own first breakpoint onward, so the columns need not share a breakpoint vector. This is deliberate — the motivating data is per-fuel monthly price curves whose breakpoints do not line up.

Such a reader interns the distinct vectors and gives each column the one it resolves against. Its public axis is the sorted union of every column's breakpoints — every instant at which some column changes value — so a sweep over reader.timestamps() sees every distinct combination of column values. There is still no presence mask: a carried-forward value always resolves once the read instant is at or after a column's first breakpoint, and an instant before some column's first breakpoint is a hard error naming that column rather than a hole in the result.

index_at on such a reader reports a position on the union axis and is not a storage row index for any column; the read path resolves each column on its own vector instead.

Forecasts

The four forecast types store their values as a content-addressed TypedArray in its native shape (the dense types as standalone HDF5 variables; a DeterministicSingleTimeSeries reuses its backing SingleTimeSeries array), while the windowing parameters live in metadata. A forecast association records horizon (the span each window covers), interval (the spacing between successive window start times), count (the number of windows), and — for Probabilistic — a percentiles vector.

The store does not interpret the layout — the caller owns the array shape (the Rust core takes a native-shape TypedArray inside a Deterministic / Probabilistic / Scenarios object; the C ABI takes a row-major byte buffer with explicit dims, and the Julia wrapper accepts a native array and serializes it row-major), and a DeterministicSingleTimeSeries deduplicates against the static series it forecasts. A DeterministicSingleTimeSeries is not added directly — it is derived from every stored SingleTimeSeries by transform_single_time_series (Rust core, C ABI, Python, Julia), sharing the underlying array. Because a DeterministicSingleTimeSeries is a synthetic view of a SingleTimeSeries, it is mutually exclusive with a real Deterministic for the same family (owner, name, resolution, features, regardless of interval): adding a Deterministic when a DeterministicSingleTimeSeries view exists — or deriving one when a Deterministic exists — raises InvalidParameter. Forecast values read back through the same path as everything else: read_by_id returns the forecast object matching the row's stored type, in every binding and over gRPC — a read names only an id, so there is no requested type to disagree with what is stored. The low-level metadata + array path remains available for raw access. See the Rust API and C ABI.

Shared Vocabulary

Periods

resolution, and the forecast horizon/interval, are calendar-aware periods, not plain fixed spans. A period is one of two kinds:

• fixed — a fixed nanosecond span (Hour, Minute, Day, Week), backed by a duration;
• calendar — a count of calendar months (Month = 1, Quarter = 3, Year = 12), where initial_timestamp + i * resolution is computed by calendar arithmetic (so a monthly grid lands on the same day-of-month each step rather than every N milliseconds).

A fixed period is never equal to a calendar one, even when their spans coincide for a given month. Periods are encoded as ISO-8601 duration strings (PT1H, P1M, P1Y) on disk and across every binding (the Python/gRPC surfaces accept a timedelta/duration for fixed periods and an ISO-8601 string for either kind, and return the ISO-8601 string).

Timestamp precision

Every instant the store records — a SingleTimeSeries or forecast initial_timestamp, every entry of a NonSequentialTimeSeries timestamp vector, and every breakpoint of a PersistentTimeSeries — is a whole number of milliseconds, the same floor a fixed period has. One millisecond is the finest resolution a period can express, and it is likewise the finest instant a series can be written at.

The rule is enforced on write, in the core, for all six addable types: a finer instant is rejected with an InvalidParameter error rather than truncated. This is what makes a timestamp mean the same thing in every consumer. The bindings do not share one precision — the C ABI and Julia exchange instants as i64 Unix milliseconds, Python's datetime is microsecond, and gRPC and the Rust core carry a full RFC 3339 string — so a finer instant would be silently truncated at some boundaries and not others, putting the same series on different instants depending on who read it. For a NonSequentialTimeSeries whose timestamps are less than a millisecond apart it is worse: two distinct timestamps collapse into one, and the vector stops being strictly increasing on the way back out. The same reasoning applies breakpoint for breakpoint to a PersistentTimeSeries.

A leap second is refused by the same rule, for the same reason, though it is not a matter of precision. Chrono spells one as a sub-second component at or above one second (23:59:60), which is a whole number of milliseconds and would otherwise pass — but a Unix millisecond count cannot express a leap second at all, so writing one would store the following second. That is not merely lossy: a leap second and the second after it are distinct instants that would become one stored instant, so two genuinely different NonSequentialTimeSeries time axes would share a content hash and be interned as one, and a single vector holding 23:59:60 followed by 00:00:00 would go in strictly increasing and come back out with a duplicate. Use the second either side of it.

A series needing a finer grid should scale its unit and record it in units, exactly as it must for a sub-millisecond resolution: a 500 µs series is a 500-unit series.

Two things are deliberately not constrained. A query bound — a time_range end, a reader's when — may be arbitrarily fine; it is not stored, and the read paths already say what an off-grid bound does (see reading a time range). And reads stay permissive: an artifact written before this rule may hold finer instants and still reads back exactly as written, which is why the rule does not change DATA_FORMAT_VERSION.

The element axis is not the time axis

Two independent things decide what a series is, and requests to add a type usually conflate them:

So a cost curve that varies over time is not another type: it is one of the types above with element_type = piecewise_linear — the dates on the time axis, the curve points as the values, and the non-curve fields that are constant across the curve (a volume window, a curve-kind tag) in application_data. A read decodes the element type back into curves rather than handing back a packing.

What does not belong in a value is a JSON blob: anything the store cannot describe cannot be deduplicated, hashed, or read columnar, and it puts the consumer back in the business of parsing its own storage.

Typed, N-dimensional arrays

Every series' values are a TypedArray: an element dtype (f64, f32, the integer widths, or bool) and a shape [length, k1, k2, …]. The first axis is time; the trailing axes are a fixed per-step element shape, so a step can hold a scalar (empty element shape) or a small tuple — for example the 3 coefficients of a quadratic cost curve (element shape [3]). The association's element_type says what those elements mean and how a ragged value (a piecewise curve, say) is packed into a fixed-width row — see Element types. The optional application_data payload travels alongside for a binding's own use; the store never interprets it.

Data Model

The data model mirrors the time-series concepts originally developed in InfrastructureSystems.jl: a component (or supplemental attribute) owns one or more named time series, and each time series may exist in several variants distinguished by features.

This page covers the catalog side of that — who owns a series, what distinguishes two series that share a name, how a row is filed, and how it is addressed once stored. Two neighbors cover the rest: Time-Series Types for what the six types mean and which to reach for, and Time References for how a series' timestamps are spelled.

Owners

Every time series belongs to an owner, identified by three fields:

owner_id is a signed 64-bit integer identifier. The owner identity is the pair (owner_id, owner_category): both participate in the association's uniqueness constraint, while owner_type is descriptive. Component and supplemental-attribute integer-id streams are independent, so the same owner_id can name a component and a supplemental attribute at once — the category disambiguates them, keeping the two owners' series distinct. Owner-scoped operations therefore take the category alongside the id (see Identity).

Features

Two series can share an owner and a name yet differ — for example a load profile for model year 2030 versus 2050. Features disambiguate them. A feature map is a set of typed key/value pairs:

features = {"model_year": 2030, "scenario": "high", "calibrated": True}

Feature values are one of four kinds: int, float, bool, or str. Internally the map is sorted by key (a BTreeMap), which gives a stable order for hashing and for the uniqueness constraint.

Reserved feature names

A feature name may not collide with a field of a time series or of the identity a row is filed under. Consumers routinely spread a feature map into a keyword-argument query — for example list_metadata(...; name = "load", model_year = 2030) — and a feature called name or resolution would shadow the real field there and silently change what the query means. Adding a time series with one of these names raises InvalidParameter:

application_data, component_field, count, data, data_hash, dtype, element_shape, element_type, ext,
features, horizon, id, initial_timestamp, interval, length, name, owner_category, owner_id,
owner_type, percentiles, quantity_kind, resolution, scenario_count, time_reference,
time_series_type, timestamps, unit_system, units

dtype and ext no longer name metadata fields — element_type and application_data replaced them — but both stay reserved. dtype is still how every binding spells a TypedArray's physical type, and a consumer still passing the retired ext= should fail loudly rather than have it silently accepted as an ordinary feature.

The match is exact and case-sensitive, like every other identifier in the catalog: resolution is rejected, while Resolution and resolution_hours are ordinary feature names. The rule applies to writes only, so a store written before it existed stays readable and its series can still be listed and removed.

Identity

Every association is filed under a tuple that must be unique:

identity = (owner_id, owner_category, time_series_type, name, resolution, interval, features)

This is what the catalog de-duplicates on — it is not how a caller addresses a series. That is the association id below. Two series with the same identity cannot coexist — attempting to add a duplicate raises DuplicateTimeSeries. Change any element of the tuple (a different name, a different model_year feature, a different resolution, a different forecast interval, or a different owner_category) and you have a distinct series. interval is NULL for the static types (which never carry one); for forecasts it lets two series of one variable at the same resolution but different intervals (e.g. a day-ahead and a real-time forecast) coexist as distinct series. Because owner_category is part of the key, a component and a supplemental attribute that share a numeric owner_id keep entirely separate sets of series.

flowchart LR
    OWNER["owner_id=42, category=Component, type=Generator"]
    OWNER --> K1["name=load<br/>year=2030"]
    OWNER --> K2["name=load<br/>year=2050"]
    OWNER --> K3["name=max_active_power"]
    K1 --> A1[("array A")]
    K2 --> A2[("array B")]
    K3 --> A1

    style OWNER fill:#4a9eff,color:#fff
    style K1 fill:#17a2b8,color:#fff
    style K2 fill:#17a2b8,color:#fff
    style K3 fill:#17a2b8,color:#fff
    style A1 fill:#28a745,color:#fff
    style A2 fill:#28a745,color:#fff

Note that two different series (K1 and K3 above) can point at the same underlying array. An identity is a metadata concept; the array is shared by content addressing.

Association IDs

Every catalog row has an id: a plain integer, assigned by the store, that names that row. It is the way to address a series — every read, removal and copy takes one.

The identity above describes a series: owner, name, resolution, features. An id names the row the store filed it under. That difference is the point: a consumer that wants to record "this generator's cost curve is that series" inside its own object model would otherwise have to embed the whole identity tuple, and keep it in step with every change to it. An id is one integer, and nothing moves it: a name is fixed once written, so the two can never drift apart.

id = add_time_series!(store, 42, "ThermalStandard", Component, cost_curve)
generator.operation_cost.variable = id   # one integer, stored in the model

The surface splits in two along that line. Identify — list_metadata and its by-id companions — answers which series exist and hands back the id for each. Act — every read, removal and copy — takes that id. A caller that knows a series only by its attributes does the first half once and keeps the id; there is deliberately no combined resolver, because the two halves have different costs and a caller that repeats a lookup it could have cached should be able to see that it is.

Three properties make an id safe to persist:

• It is never reissued. Deleting a row strands its id permanently. A reference to a deleted series stops resolving — it can never come back meaning a different series, which is the failure a recycled row number would cause silently, with no foreign key anywhere to catch it.
• It survives the operations that change a series' description. A rename or a reassignment to a new owner keeps the id, as do compact and a save-and-reopen. Those are UPDATEs and file copies, not new rows.
• It is not part of identity. Two series differing only in id are the same series to the uniqueness rule and to both content hashes. It describes the row, not the data.

The store assigns it; no add accepts one. Not add_time_series, not a bulk add, and not either association catalog's attach / link. This is what makes "never reissued" a guarantee rather than a convention: AUTOINCREMENT only ratchets its counter upward, so an assigned id is never handed out twice, while a caller free to name one could re-file a retired id and make a stale reference in some consumer's model quietly resolve to a different series. The association row types carry an id field because a listing populates one, but it is an output — an add ignores it, so a row read from one store and attached to another is filed under a fresh id there.

What an id does not do is travel between stores. It is the row's number in one catalog, so a merge assigns fresh ids in the destination, and two stores holding identical content will disagree about them. The one place ids do cross a boundary — and the one writer that files rows under ids it was given — is the OpenAPI document round trip, where preserving them is the whole point: an import that assigned fresh ids would leave every reference the document carries pointing at the wrong series. That wire form spells the field association_id — in a document traveling beside components and supplemental attributes, an unqualified id would not say which id it is — and the schema requires it on every time-series row. Because the import is the only door, the guarantee holds there too: a supplied id must sit above the destination catalog's counter (DuplicateAssociationId otherwise, so a document's ids fit a fresh store but not one that has issued ids of its own), and a document supplies one for every row or for none. Neither association catalog's wire form carries an id at all, so both always assign.

The same round trip carries two fields the schema gained late (crates/infrastore-core/sienna_schemas/SOURCE.md records the upstream commit vendored here): array_shape, the stored array's full native shape ([length, *element_shape] in the catalog's terms, where the schema's element_shape is only the per-step trailing shape), and time_reference. Both exist so an imported row is identical to the exported one — the forecast layouts above are conventions the caller owns, so the native shape cannot be rebuilt from horizon and count. Neither is required: a producer predating them writes rows without them, a reader that does not know them ignores them, and an import that finds them absent falls back to the schema's own fields.

Reading an artifact back from arrays plus a document

Those two fields exist so that a document can stand in for the catalog entirely. A consumer that already ships the association rows in JSON of its own — PowerSystems writes a system.json beside a time_series.h5 — carries the .sqlite half for nothing: the arrays are the part that cannot be reconstructed, and the catalog can be replayed. Store::open_without_catalog is the way in. It opens the array half of an artifact whose catalog is absent and mints an empty one carrying the array file's own generation stamp, so the rebuilt pair opens normally ever after; an ordinary open cannot, because a stamped array file beside an unstamped catalog is exactly the half-finished save the paired-stamp check exists to catch.

All six of the types the wire contract defines make the trip, including NonSequentialTimeSeries — which needs one field the others do not. (A PersistentTimeSeries does not: it is an infrastore-local extension outside the vendored oneOf, so the export omits its rows and the import refuses them. A store holding them still exports the rest, but a restore from arrays plus a document alone will not carry them.) Its timestamp vector lives in the store rather than the document, content-addressed so a cohort sharing one axis stores it once, and the values cannot imply which axis a row is on: arrays are content-addressed too, so two irregular series with byte-identical values on different axes share one stored array and only the catalog's timestamps_hash distinguishes them. An import that guessed would hand back another series' timestamps. So the wire form locates the axis, as timestamps_uri — the same kind of field uri is for the values, and a locator rather than the vector itself precisely because the axis is shared, where inlining it would repeat the whole vector on every row of the cohort. The axis ships in the array file alongside the arrays, so the import resolves the locator against the store; a row missing it, or naming an axis the store does not hold, is refused.

Both imports validate every incoming row against those vendored schemas before decoding it, so a document that drifted from the contract is refused in the schema's own terms — the row, the field, and what was expected — rather than by whatever the Rust struct happens to notice first. A time-series row is checked against the per-type schema its own time_series_type selects, which is what the wrapper's discriminator prescribes and what keeps the error specific: a failed oneOf can only report that nothing matched.

Each of the three catalog tables keeps its own independent counter, so an id is only meaningful alongside the table it came from.

Writes report the id they used, and reads take one:

In the Rust core an id is the newtype TimeSeriesId, so an owner_id cannot be passed where a series id belongs; the dynamic bindings exchange a plain integer.

association_exists fetches no row, so a consumer can check every reference in its model on load rather than discovering a dangling one mid-simulation.

read_by_ids and remove_by_ids both refuse a set containing an id that names no row — the read returns nothing, the removal removes nothing. That is deliberate: a caller working from references it recorded earlier has a model that disagrees with the store, and since an id is never reissued the disagreement will not resolve itself. Sift the set with association_exists first when some references are expected to have gone.

read_by_id is the single-id read, and it also takes the slice: a start_time plus a len of timesteps or a count of windows. Both halves happen in one call because the primary-key lookup already returns the row the window resolves against — a consumer holding an id spends nothing to learn a series' resolution or count before asking for the second day of it. A window is checked, where read_by_ids_range clips: a start off the series' own grid, or an extent running past its end, is an error rather than the smaller answer a range would return. A range says "whatever lies between these bounds" — which is what an export wants, knowing the bounds and not the step count — while a window says "these exact steps", and a caller that asked for 24 and silently received 3 has a bug the store can see and it cannot. What a range clips to is type-specific: a regular series' value covers its step, so a start inside a step selects that step and the sliced initial_timestamp can precede start; an irregular series' value is an instant, so only timestamps at or after start are selected; and a forecast window is a whole array with nothing partial to return, so a forecast's start must be a window boundary at or before the last window (an error otherwise) and only its end clips. A start earlier than the first window is the one exception, and it clips: there is no partial window before the first one, only no window at all, and refusing it would fail every export window wider than the data. See Reading a time range.

A calendar period is not closed under slicing

A SingleTimeSeries and a dense forecast are each stored as an anchor, a period, and a count, so a sliced read has to describe its answer the same way — anchored at the slice's own first point. For a fixed period that is exact. For a Period::Months it is not, because the end-of-month clamp is not associative: a grid stepping monthly from Jan-31 is Jan-31, Feb-29, Mar-31, but re-anchored at its own Feb-29 it becomes Feb-29, Mar-29, Apr-29 — the stored values under dates the store does not hold. No anchor fixes it, because the sub-grid keeps the original anchor's day of month and no instant in the slice carries it.

So such a slice is refused (InvalidParameter) rather than answered with a grid that is not the one stored — the values would be right, the dates wrong, and nothing would signal it. The refusal is narrow: it needs a calendar period, an anchor on a day some month is too short for, and a slice starting at a clamped point and running past it. Read the series whole and slice the materialized timestamps, or store the instants explicitly with NonSequentialTimeSeries. transform_single_time_series is held to the same rule at write time, since each window of the view it derives is a run of the source's own steps described that same way.

The owner guard

read_by_id and remove_by_ids each take an optional expected owner — read_by_id_for_owner / remove_by_ids_for_owner in the Rust core, an owner=(id, category) keyword in Julia, keyword-only owner_id / owner_category in Python, has_owner beside the two across the C ABI. The row is held to that owner, and one belonging to anyone else is OwnerMismatch rather than a read or a delete.

It exists because the two halves cannot be checked separately. An id is the whole address and it survives replace_owner, so a consumer whose model says "this component's series" — and which therefore wants to confirm the owner before acting — has a window between the confirming call and the acting one. A reassignment landing in that window makes the removal retire the new owner's series, which is exactly what checking the owner was meant to prevent. Passing the owner into the call closes the window: the check and the act are one transaction. On the read side there is no window either way, but the guard is still the cheaper spelling — the owner comes off the same row the values are materialized from, so it costs nothing, where a separate check is a second round trip.

Optional Descriptors

Each association can also carry:

• units — a free-form, end-user-facing label such as "MW". No dimensional analysis is performed.
• quantity_kind — what kind of physical quantity the values measure, e.g. "ActivePower", "Energy", "Length". Free-form; the recommended vocabulary is a QUDT QuantityKind local name. It sits above units rather than duplicating it, for two reasons. A units library's dimensional analysis cannot separate active from reactive power — both are [M L^2 T^-3] — but a quantity kind can. And when unit_system is component_base the values are per-unit and therefore dimensionless, so this is the only surviving record of what they measure and which base converts them back. The column is deliberately unconstrained: the composite economic quantities an energy modeler needs ($/MWh, MMBtu/MWh) are exactly where QUDT's coverage thins out.
• unit_system — which basis the values are expressed in: natural_units (the units named by units) or component_base (per-unit against the owning component's own base). This is the per-unit declaration power-systems modelers know as the unit system; PowerSystems.jl spells the same idea UnitSystem, with NATURAL_UNITS and DEVICE_BASE. It is a label, not a conversion: the store holds no base value and rescales nothing, so converting component_base values back to natural units is the consumer's job, using the base that lives on the owning component in its own object graph. Unset means unspecified, which is deliberately not the same as natural_units — every association written before this field existed is unset, and reading those as natural units would assert a basis nobody declared.
• component_field — the field on the owning component whose value these values are the time-varying form of, e.g. "max_active_power" or "rating". Free-form and never interpreted: it names a field in the consumer's own object model, which the store has no view of. It records what the values are for, where name only says which series they are — the two coincide by convention in many models but are not the same thing, since one component may carry several series for one field (a forecast and an actual, a set of weather years) and name is part of a series' identity where this is not. Named for the common case; when the owner is a supplemental attribute it names a field on that attribute.
• time_reference — how this series' timestamps were spelled: utc, zoneless, a fixed offset (-07:00), or an IANA zone name (America/Denver). See Time References, which this one deserves a page of its own for.
• application_data — an opaque, package-owned extension payload stored verbatim (typically JSON) that a binding writes and reads for its own purposes. The store never parses or interprets it, and end users are not expected to set it. Element typing does not live here: that is element_type below, a first-class column the store owns and validates. This is also where application policy about a series belongs — a consumer's rule for collapsing a PersistentTimeSeries to a scalar, say. Such a rule is not storage, and it gets no catalog column.
• element_type — what the array's elements mean, in the store's own language-neutral vocabulary: a dtype spelling (f64, i64, …) for plain numbers, else tuple(N,dtype) or one of the function-data kinds (linear_function, quadratic_function, piecewise_linear, piecewise_step). It supersedes a separate physical dtype: the dtype of the stored bytes is derived from it. Unlike units and application_data it is not inert — the write path validates the array's dtype and per-step shape against it. See Element types.

units, quantity_kind, unit_system, time_reference, component_field, and application_data are recorded in metadata and returned on read, but they do not affect identity or storage: they are absent from the key and from both content hashes, so two series differing only in a descriptor are a duplicate.

component_field is the one descriptor that is also a filter (ListFilter::component_field, and its equivalent in every binding): "every series that varies this field", alone or scoped to one owner. Being descriptive rather than identifying, it narrows a listing but never addresses a single row on its own — one component may carry several series for one field, distinguished by name or features. It matches exactly and case-sensitively, and a series that declares no component_field matches no value, so the filter cannot select the rows that left it unset.

Associations Between Entities

Beyond owning time series, catalog entities can be related to each other. The catalog records two such relationships, in two separate tables, because they are not the same kind of thing: attaching an attribute to a component and wiring one component to another have different identities and different query patterns.

Supplemental attributes attached to components

Identity is the (component_id, attribute_id) pair. The type names are denormalized labels, not part of identity: re-attaching the same pair under different type names is a duplicate and is rejected. One attribute may be attached to many components, and one component may carry many attributes; only the exact pair is constrained.

Parent/child edges between components

Both endpoints are always components, so unlike time-series owners there is no category to disambiguate. Identity is the ordered (parent_id, child_id) pair — the reversed pair is a different edge. There is no relationship-kind column, so a given pair may be related at most once.

Properties shared by both

Two consequences of the deliberate absence of foreign keys and cascades:

• Associations and time series are independent in both directions. Removing a component's time series does not remove its attribute attachments or its edges, and removing either does not touch any series. A consumer that wants both effects makes both calls.
• The store never observes a deletion it did not perform. Components and attributes live in the consumer's object graph, so a cascade could never fire; consumers call the matching remove_* with the appropriate filter instead.

Filtering takes lists of concrete type names, rendered into SQL IN (…). Expanding an abstract type into its subtypes stays in the calling language, where the type hierarchy lives.

Terminology: rows of the time_series_associations table — the owner-to-time-series records described above — are also called "associations" throughout this documentation and the code. They are unrelated to the entity-to-entity tables described in this section.

Both are available in the Rust core, the C ABI, Julia, Python, and the infrastore CLI (attach / detach / link / unlink); neither is exposed over the read-only gRPC server. The supplemental-attribute surface is the wider of the two (it carries counts and a grouped summary) because each of its operations is driven by an existing consumer; the parent/child surface is deliberately narrower for now.

Store attributes

Everything above describes a row: a series' application_data, a component's supplemental attributes, an edge between two components. Store attributes describe the artifact itself.

A store attribute is a key/value pair recorded once per store — who built it, from what source system, under which of the consumer's own schema versions. The store never interprets a value, in exactly the same spirit as application_data: keys and values are TEXT, nothing here participates in any identity, hash, or query, and a caller wanting structure stores JSON.

store.set_store_attribute("creator", "sienna-build")?;
store.set_store_attribute("source_system", "WECC 2032 ADS")?;
assert_eq!(store.get_store_attribute("creator")?.as_deref(), Some("sienna-build"));

Note the term and the prefix. In this project a bare "attribute" means a supplemental attribute (above) and a bare "metadata" means a TimeSeriesMetadata row, so every identifier for this feature carries store_: set_store_attribute, list_store_attributes, the CLI's store-attr, the store_attributes table. Neither misreading is available.

Four rules are worth stating outright:

• A set replaces. key is the primary key, so an artifact records one creator rather than a history of them.
• An absent key is None, not an error. A consumer asking whether the artifact carries a key is asking a question, the same reading get_metadata_by_id takes. A key set to the empty string is present; every binding keeps the two distinguishable.
• infrastore. is reserved, refused on removal as well as on write, so the store can stamp facts of its own later without colliding with a consumer's keys — and so a reserved key cannot be worked around by deleting it.
• Attributes are content. is_empty reports false for a store holding nothing but provenance: those rows are the consumer's own text, recoverable from nowhere else, and a consumer that skips writing an "empty" store would drop them with no error.

They live in the catalog, so they travel with persist_to and persist_catalog, survive compact (which rewrites only the array half), and come across with open_copy. open_without_catalog mints an empty set, since there is nowhere else for them to have been. They are not carried by the OpenAPI export or import: the vendored schema has no place for them.

Available in the Rust core, the C ABI, Julia, Python, and the CLI (store-attr, plus store_attributes in store-info); the read-only gRPC server carries the read half (ListStoreAttributes, GetStoreAttribute).

The CLI's two cross-store commands take a position rather than ignoring the table. merge is additive with the destination winning: a merge brings data into an artifact that already has an identity, so a key the destination lacks is copied, a key both sides agree on is a no-op, and a disagreement is reported and left as the destination has it. diff gives them a section of their own — there is no series identity to pair them on — and counts a difference toward its nonzero exit.

Time References

The store records instants. A time_reference records what those instants were written as, so a series comes back the way it went in instead of being relabeled UTC at every boundary.

Three of the four name an instant; zoneless does not, and most rules below split on that binary rather than on the four spellings. An unset reference groups with the zoned ones.

Each binding infers the spelling from the input type, so nothing takes a new required argument:

ZoneInfo("UTC") records the zone UTC, not the literal utc. The two render identically forever; the difference shows up only in what the catalog reports back, which is the point of recording a spelling at all.

A spelling is not a grid

A reference records how timestamps were written. It does not change how the grid is stepped: resolution and interval are durations, so an hourly series has hourly instants whatever its reference says. Rendering an hourly America/Denver series across the November fall-back gives 01:00-06:00, 01:00-07:00, 02:00-07:00 — two identical wall clocks, two distinct instants, correctly ordered.

That is the difference between two things "store this in Denver time" can mean:

• Instants, displayed in Denver. Storage is untouched — UTC instants plus a label. This is what a named zone means here.
• A local-clock grid — hourly by the clock, so a 23-hour day in March and a 25-hour one in November. This is inexpressible in SingleTimeSeries and the dense forecasts, whose grid is a Period: a fixed count of milliseconds. Use NonSequentialTimeSeries, which carries an explicit instant per value, so the caller derives those days and the data records them rather than arithmetic implying them.

The store now checks, when you give it something to check

Asserting initial_timestamp + resolution is a claim the store cannot verify — the vector it describes is never supplied. Two rules close that:

• A calendar-scale period on a named zone is refused. A period of a day or more is a span of instants, so on a zone that observes DST it drifts away from the local clock it looks like: a P1D series stepped from local midnight in Denver lands on Nov 3 23:00 after the November transition, on the same calendar day as its predecessor, and stays an hour off forever. P1M is worse — it steps the UTC calendar. Both are InvalidParameter at the write.
• Sub-daily periods stay legal, and that is not a compromise. DST moves the offset, not the length of an hour: Denver has 8784 hours in 2024, exactly as UTC does, every gap exactly one hour, and the 23- and 25-hour days fall out of instant stepping on their own. An hourly grid in a DST zone is the local clock. Refusing it would push callers onto a fixed offset, which is silently wrong for half the year.

A forecast's horizon is exempt: it is a window length, only ever divided (H = horizon / resolution) and never added to an instant, so horizon = P1D — the canonical day-ahead shape — stays legal in any zone. resolution and interval do step, and are covered.

The constructive half is from_timestamps: hand over the timeline you actually have and the store infers the period and proves the instants lie on it, or refuses naming the entry that broke the pattern. This is how a local-clock timeline reaches the store — you materialize it in your own date library, where the policy for a nonexistent or ambiguous wall clock belongs, and the store records what you have rather than what a resolution implies.

Someone with 8760 naive Denver timestamps who localizes only the first and passes resolution = 1h still gets labels shifted by an hour after each transition if their file was not a local-clock walk, and nothing in the values distinguishes that from a correct series — so hand the vector over rather than assert the step.

Months step on the UTC calendar

Period::Months is calendar arithmetic, so unlike a fixed period it has to be told which calendar. It uses the stored UTC one, and the reference does not redirect it. TimeZones.jl steps the local clock instead, so the two disagree by an hour at every DST transition and by up to a day at a month boundary.

Local-frame stepping is refused for three independent reasons: it is the local → instant direction the store deliberately never runs (below); it would let a spelling decide which instants a series contains; and it would need a time-zone database in the core, which would make a stored series' instants depend on which IANA release built the reader. A calendar period on a named zone is therefore refused on write, and on a fixed offset — which has no DST to drift against, only a month boundary — it is warned about. A caller who wants months on a local calendar wants a local-clock grid, and the answer is the one above: NonSequentialTimeSeries, or from_timestamps, which picks between the two for you.

Why a named zone is safe

The ambiguity a named zone is feared for lives in the local → instant direction, and the core never runs it.

• On input that direction has already happened, in the caller's own datetime library. Julia refuses an ambiguous local time outright; Python resolves it through fold. Either way the binding is handed a value that already names one definite instant. The CLI is the exception, because it is handed text — see below.
• On output the store runs only instant → local, which is total and single-valued: one instant maps to exactly one wall clock in a named zone, and converting it back yields the same instant.

So a year-long Denver series stamped -07:00 renders every timestamp after the March transition an hour wrong, while the same series stamped America/Denver renders all of them correctly. Recording "the offset in effect at initial_timestamp" is the one option that is quietly incorrect, which is why it is not among the four spellings.

Two caveats belong here rather than in the type. Rendering a named zone is tz-database-dependent, so a retroactive rule change moves the displayed local time of an already-stored instant — the store records the instant, and the label is a rendering hint. And a zone name's existence is audited, never gated: the core checks only that a name is shaped like an IANA name and cannot be read as an offset or as either literal. Every layer that has a database — the CLI via chrono-tz, Python via zoneinfo, Julia via TimeZones — warns on a name it does not recognize and stores it anyway, and infrastore store-info reports the catalog's distinct spellings with unrecognized zones flagged. Gating would turn a rare read-time error into a write-time error coupled to our release cadence: when IANA adds a zone, a caller whose own database already has it would be refused until they upgraded.

The CLI is where local → instant actually happens

Every other binding is handed an already-resolved datetime. The CLI is handed text, so --assume-timezone America/Denver over a zoneless column is the one place in the system that runs local → instant itself, and chrono-tz answers in three values — each with its own behavior, per row:

Rejecting loudly, per row, with both candidates named is what makes a named zone acceptable here; silently picking one is not. Reading is unaffected: rendering a stored instant in a named zone is the total direction, so --assume-timezone plays no part in it.

Query bounds and mixed selections

A bound must be spelled the way the series is, and a mismatch is refused rather than coerced:

An off-grid bound still names an unambiguous instant, so flooring it is well-defined — that is why time_range snaps. A wall-clock bound against a series that records instants is a category error: there is no defined mapping to fall back on. Bounds stay unconstrained in precision, though: a sub-millisecond bound names a real instant even though a stored one may not.

The same partition drives two rejections and one filter:

1. A ranged bulk read over a selection spanning both groups is refused — no single bound is valid for all of it. An unranged one is unaffected: without a bound there is nothing to disagree about, and each series carries its own spelling back.
2. A StaticReader materializes one timestamp axis, so a mixed cohort is refused at build time, where the error can name the series that disagree. Mixing utc, an offset, and a named zone in one cohort is fine — all three name instants, and the axis is spelled with the cohort's reference when every member agrees and utc when they merely agree on naming instants.
3. ListFilter::zoneless is the constructive half: true selects the wall-clock series, false selects everything that accepts an instant bound — the three zoned spellings and the rows that left the reference unset. It is a binary predicate rather than a match on a specific spelling because an exact match cannot name that second group at all (the trap component_field documents), and here those rows are a coherence group rather than an oversight.

Readers

A reader is the columnar bulk-read surface: build one once over a filter, then walk the timeline and take every matching series' value at each instant. It is the access pattern the on-disk layout is built for, and the one a parent package hands to its users, so it is worth understanding as a concept rather than as two type names.

There are two, and they are described signature-by-signature in each language's reference (Rust, Python, Julia, C ABI):

Neither is exposed over gRPC.

Why They Exist

Static arrays that share a shape are packed as columns of one dataset, chunked across the whole width — (rows, cols, *element_shape), with rows = 1 unless the dataset is narrow enough that one timestamp row would make a chunk too small to be worth one (file format). So a chunk holds one timestamp, or a few consecutive ones, across every column: "every generator's output at hour 4 371" is one chunk read per dataset, while "this one generator's whole year" has to touch every chunk band in the dataset.

That is the asymmetry these readers are built on, and it is a property of the width, not of the row count. A sweep — which is what a reader does — pays the same either way: it visits every chunk regardless, and when a chunk carries several rows they are the next ones the sweep asks for. Scattered access to a single timestamp is the case where extra rows are decompressed and discarded.

That asymmetry is deliberate (Design Choices), and a reader is the API that spends it correctly. A loop of whole-series reads walks the slow direction once per component; a reader walks the fast one once per timestep.

The forecast case is the same argument with a different unit. A dense forecast array is chunked in bounded blocks along the window axis, so reading one window decompresses its whole block. A ForecastReader sizes its cache to that block width, so a sweep over the window timeline decompresses each block exactly once — where independent per-window reads re-decompress overlapping data every step.

A Reader Is a Plan, Not a Cursor

Building one resolves the filter to a fixed set of columns, pins the timeline, and allocates the buffers each read will overwrite in place. It holds no borrow on the store and advances no position of its own: the caller names the instant, the store fills the buffers, the caller walks the columns. A tight simulation loop therefore allocates nothing after the build.

Two consequences worth planning around:

• The column set is frozen at build time. A series added afterwards is not in the reader; build a new one.
• The cost is paid up front. Building resolves metadata for every matching row, so build once outside the loop — never per timestep.

One Timeline Per Reader

A reader materializes one timestamp axis shared by every column, because that is what makes a read a single positional lookup rather than a per-column search. What "one timeline" requires depends on the type the filter names:

A mismatched cohort is refused at build time, where the error can name the series that disagree, rather than at the first read. The same applies to time-reference coherence: a reader whose matched series mix wall clocks with instants is refused, because no single axis can be spelled for both.

The one exception: step functions

A PersistentTimeSeries is the row that does not have to agree with its neighbors, and only because a step function makes that safe: it has a value at every instant from its own first breakpoint onward, so a column need not carry the instant being read in order to answer for it. Such a reader interns the distinct breakpoint vectors its columns sit on, records for each column the vector it resolves against, and takes their sorted union as its public axis. A read then carries values forward per vector rather than once for the whole reader.

Two consequences follow from the union being a public axis rather than a storage layout:

• A position on it is not a storage row. index_at reports a position on the union, which belongs to no column in particular; the values come from each column's own row in force there. Nothing else in the reader surface indexes this way.
• There is still no presence mask. An instant before some column's first breakpoint has no value that column could report, so the read is a hard error naming that column rather than a gap the caller has to test for. The rule the other two types get from a shared timeline — every column has a value at every readable instant — is preserved, just enforced at read time instead of build time.

The motivating data is per-fuel monthly price curves whose breakpoints do not line up. Forcing them onto one cohort would mean either inventing breakpoints or building one reader per fuel, and both lose the single chunk-aligned sweep a reader exists to give.

Naming the span instead of inheriting it

Sharing a grid is a property of whole series, and a simulation usually wants a span they agree on rather than the whole of each. A year of load beside a week of an outage schedule, or one component logged from an hour later than the rest, has no shared grid — and under the rule above, no reader at all. That refusal was correct and not useful.

So a SingleTimeSeries reader can be given its axis: a start, and optionally an extent. Each column then records how many of its own steps precede that anchor, and a read adds that offset to the reader's index. The one-timeline rule is untouched — there is still exactly one axis, and every column still has a value at every instant on it — but the axis is now the caller's span rather than whatever the series happened to have in common. Without an extent the span runs as far from the anchor as every matched column reaches, which is the widest one on which nothing has to be dropped.

The window is where a reader could most easily hand back a full, plausible, wrong row, so all three of its edges are checked at build time rather than smoothed over:

• A column that does not cover the span is an error naming it. Dropping it instead would be invisible at read time: the sweep would return a complete row, one column short, and nothing in the result would say so. Excluding a series is a decision only the caller can make, and the filter is where they make it.
• The anchor is checked against each column's own grid, never floored onto it. This is the one place a reader is stricter than read_by_id, which floors a start inside a step because a value covers its step. A reader answers for a whole cohort at one instant, so flooring per column would shift columns against each other by up to a step.
• A calendar period is still not closed under re-anchoring. A window is a re-anchoring, so it meets the same Period::sub_grid_is_anchorable rule a sliced read does: a monthly grid from Jan-31 read from its own Feb-29 would report Feb-29, Mar-29, Apr-29 against the values of Feb-29, Mar-31, Apr-30 — right values, wrong dates. No anchor fixes it, so the window is refused.

Mechanically the offsets ride the machinery the persistent case already needed: a per-column row index and the scattered backend read. A window whose columns all start at the anchor drops back to the single-index read, so the ordinary uniform sweep costs exactly what it did.

Or dropping the series that are not on the grid

A window assumes every matched series belongs in the sweep. Often one does not: a stray day of data beside a year of it, under the same name, is usually a different component rather than a shorter view of the same thing. Forcing it into a window costs the whole year — the span is capped by the shortest column — for a series nobody asked about.

So the filter vocabulary can name a grid too. ListFilter::initial_timestamp and ListFilter::length join resolution to select the cohort on one grid, and the series that are not on it are not matched at all. This is the same move ListFilter::zoneless makes for spelling: a rule that refuses a divergent selection is only half an answer, and the other half is a way to construct a coherent one.

The two remedies are complementary, not competing, and the question they answer is different:

The last row is the one to keep in mind. A window is a bound, so it is checked and a series that cannot answer it is an error; a filter is a selection, so a grid no row is on is an empty result. It is the same distinction the store draws everywhere between a bound and a predicate.

Both are descriptive rather than identifying: a grid is not part of KeyIdentity, so two series differing only in start or length are one row to the catalog, and an identity probe never narrows by either.

Sharing Is Resolved Once

Where the static side shares storage by packing many components into one dataset, forecasts share it by content addressing: identical arrays are stored once. A ForecastReader inherits that at read time — it reads each distinct backing array a single time per step and fans the result out to every column referencing it, so a forecast shared by a hundred components costs one decompression, not a hundred.

That fan-out is visible to the caller: an entry's slot identifies the underlying array, so per-component work downstream of the read can dedup the same way the read did rather than repeating itself once per referencing component.

When Not to Use One

A reader is the wrong tool for the inverse access — one component's full history, an export, a plot of a single series. Reach for read_by_ids over the ids you want, which reads packed series in one decompress-once pass per dataset. It is still the slow direction against this layout, but it is far cheaper than a read_by_id per series, and much cheaper than building a reader you will step once.

Storage Model

A persistent store is two files that travel together:

system.h5          # HDF5 — the numerical arrays
system.h5.sqlite   # SQLite  — the metadata associations

The catalog path is derived by appending .sqlite to the HDF5 file name. This page explains the split and how the two halves stay consistent. For the exact bytes, dataset names, and table columns, see the On-Disk File Format reference.

Why Split Arrays From Metadata

Arrays and metadata pull in opposite directions:

Forcing both into one format would compromise one of them. Instead, each lives where it is strongest, and the Store layer coordinates them.

The Array Side: HDF5

Arrays live under time_series/single/ in the HDF5 file, in one of two storage modes.

Stored arrays are immutable: a value array is added or deleted as a whole and is never edited in place — there is no API to mutate a row, slice, or column of an array already in the store. Changing data means writing a new array (content-addressed and deduplicated on write) and, if desired, deleting the old one.

Packed mode holds SingleTimeSeries (and the backing array of a DeterministicSingleTimeSeries). Arrays that share a (dtype, element_shape, length, resolution) are packed together as columns of one dataset named sts_{dtype}_{shape}_{length}_{res}, with shape (length, cols, *element_shape). The column count cols is sized to the batch that created the dataset (capped so one chunk stays within a byte budget) — where "the batch" is a bulk add's items or the whole span of an open transaction; only an incremental one-at-a-time write outside a transaction uses the default width of 1,000:

flowchart TB
    subgraph ds["dataset sts_f64_s_8760_PT1H  shape (8760, cols)"]
        direction LR
        C0["col 0<br/>series A"]
        C1["col 1<br/>series B"]
        C2["col 2<br/>(free)"]
        CN["col cols-1<br/>(free)"]
    end
    H["companion sts_f64_s_8760_PT1H_h<br/>cols hash strings"]
    C0 -.hash.-> H
    C1 -.hash.-> H

    style C0 fill:#28a745,color:#fff
    style C1 fill:#28a745,color:#fff
    style C2 fill:#6c757d,color:#fff
    style CN fill:#6c757d,color:#fff
    style H fill:#6f42c1,color:#fff

• Columns are series, rows are timesteps. Chunking spans the whole width, (rows, cols, *element_shape) — one row, so that one chunk is one timestamp across every column, unless the dataset is narrow enough that a single row would make the chunk too small to be worth one (file format). The layout favors bulk writes (a batch fills whole chunks in one pass) and reads across series by timestamp (one timestamp is a chunk per dataset, and a sweep costs the same whatever the row count). The reverse directions are the slow ones, by design: reading a single series in full touches every chunk band, and adding one series at a time rewrites a chunk band per timestep — which is why a run of single adds belongs inside a transaction, where they are buffered and written as one block instead.
• A companion dataset holds the hashes. For each packed dataset there is a sibling {dataset}_h, a (cols, 64) array of u8; row i holds the SHA-256 hex of column i as raw bytes, or is all-zero if the slot is free. This is the on-disk index the backend rebuilds on open.
• Datasets spill when full into …__1, …__2, and so on — when a batch exceeds the per-dataset column cap, or when incremental writes fill a default-width (1,000-column) dataset.
• Compression is configurable at store creation and applies to every data variable (packed and standalone). The default is DEFLATE (zlib) level 3 with the byte-shuffle filter; you may change the level (0–9), disable shuffle, or turn compression off entirely. The choice is recorded in a compression global attribute and restored when the store is reopened for appends, so later writes reuse the same filter. Compression is a storage detail only — arrays decode transparently regardless of the filter, so stores written with different settings stay mutually readable and the data-format version is unaffected. In-memory stores ignore the setting.

Packed mode holds every static series. SingleTimeSeries (and the array behind a DeterministicSingleTimeSeries) pool by resolution into sts_… datasets; the explicit-time-axis types (NonSequentialTimeSeries and PersistentTimeSeries) pool by their timestamp vector into nsts_… datasets, because the chunking is timestamp-major and so only means something for arrays on a common time axis. Such a series carries that axis explicitly, and the store content-addresses it — one tsv_{hash} dataset of unix milliseconds per distinct axis, in the file's own timestamps group — so the interned hash is the cohort key, which is what lets a StaticReader sweep irregular series the same way it sweeps regular ones.

Standalone mode holds the dense forecast arrays (Deterministic, Probabilistic, Scenarios) and any explicit-time-axis series alone on its axis — a pool spreads one array over length chunks, so a cohort of one is not worth packing. Each is its own typed, multi-dimensional variable arr_{hex_hash} — no column packing and no companion hash (the variable name carries the hash). Lone irregular series are shaped [length, *element_shape] and chunked whole; dense forecasts are shaped [H, count, *element_shape] (with extra leading axes for Probabilistic / Scenarios) and chunked in bounded blocks along the count (window) axis, so reading one forecast window decompresses a single block rather than the whole array. The ForecastReader caches a block at a time to match.

The file-format reference gives the precise naming and dimension scheme. Nothing on the array side distinguishes a forecast from a static series of the same physical shape — the type, timestamps, and windowing parameters all live in metadata.

The Metadata Side: SQLite

The catalog holds six tables. The first four describe time series; the last two record relationships between catalog entities and have nothing to do with time series at all.

• time_series_associations — one row per (owner_id, owner_category, time_series_type, name, resolution, interval, features) association, including the data_hash that links it to a packed column or standalone variable, the array typing (dtype, element_shape), the opaque package-owned application_data payload, plus temporal fields, forecast parameters (horizon, interval, count, percentiles), the unit descriptors (units, quantity_kind, unit_system), the time_reference recording how the timestamps were spelled, the component_field the values vary over time, and the features_hash.
• feature_sets — the expanded key/value pairs of a feature map, one row per key, typed by a value_kind discriminator. The table is content-addressed, exactly as arrays are: its primary key is (features_hash, key) — the same hash the association row already carries, so no join column is needed. A feature set is therefore stored once and shared by every association whose hash matches, not copied per association; an empty map stores no rows at all. Because the rows are shared, there is deliberately no foreign key to time_series_associations and no ON DELETE CASCADE (see Compaction below).
• schema_version — a single version column holding the catalog schema revision (CATALOG_SCHEMA_REVISION, currently 2). Its own contract, independent of the artifact's data_format_version: a catalog change the idempotent DDL cannot make to an existing table needs a revision bump and an append-only migration. A catalog predating the stamp reads as revision 1. See Upgrade a store in place.
• supplemental_attribute_associations — which supplemental attributes are attached to which components, as (component_id, component_type, attribute_id, attribute_type). Identity is the (component_id, attribute_id) pair.
• parent_child_associations — directed edges between components, as (parent_id, parent_type, child_id, child_type). Identity is the ordered (parent_id, child_id) pair.

The last two are described in Associations Between Entities. They carry no foreign keys and no cascade, and they are independent of time_series_associations in both directions: removing a time series never touches them, and removing an association never touches a series. They were also added without a data_format_version bump, so a store written before they existed simply gains them on its first writable open — which is why every read of them tolerates the table being absent.

A unique index over (owner_id, owner_category, time_series_type, name, resolution, interval, features_hash) enforces the identity uniqueness invariant at the database level. owner_category is part of the key, so a component and a supplemental attribute that share an owner_id are independent; interval is part of the key too, so forecasts of one variable that differ only by interval are distinct. Because SQLite treats NULL as distinct in a UNIQUE index, a second index folds NULL resolution and interval to a sentinel so series without them (e.g. NonSequentialTimeSeries and PersistentTimeSeries, or any static series, which carry no interval) are still constrained. Indexes on data_hash, (owner_id, owner_category), and resolution keep lookups fast.

Keeping the Two Files Consistent

Because a write touches both files, Store follows a careful ordering so a failure cannot leave a dangling reference:

sequenceDiagram
    participant C as add_time_series
    participant B as HDF5 backend
    participant M as SQLite
    C->>B: put_array(hash, data)
    Note over C,B: idempotent on hash — staged for rollback
    C->>M: BEGIN
    C->>M: INSERT association
    alt insert succeeds
        C->>M: COMMIT
        C-->>C: return TimeSeriesId
    else insert fails (duplicate etc.)
        C->>M: ROLLBACK
        C->>B: remove_array(staged hashes)
        C-->>C: return Err
    end

• Array first, metadata second. put_array is idempotent — calling it with an already-present hash is a no-op — so staging the array before committing metadata is safe.
• Metadata commit is the point of no return. If the SQLite insert fails (most commonly a DuplicateTimeSeries constraint violation), the transaction rolls back and any array column staged in this call is removed, returning the store to its prior state.
• Bulk writes are all-or-nothing. add_time_series_bulk (and the buffered bulk_add session) group packed series by shape and stage each group as one batch-sized block — filling whole chunks — then insert every association in one transaction; any error rolls the whole batch back and removes the staged arrays. A pool the batch gives a single array is the exception: sizing a dataset to it would claim a one-column dataset, so that array fills a shared pool's slot instead, as a single add does — or, for an irregular series, is written as a standalone arr_ dataset unless the file already holds a pool for its time axis (see Packed mode). A one-item batch is the commonest case of it.
• Single adds inside a transaction take the block path too. Nothing a transaction wrote is durable until its outermost commit, so its packed adds are buffered per pool and written together by the same block writer at the commit — bounded per pool by a column width and across all pools by a byte ceiling, and with a span holding one array for a pool falling back to a growth-pool slot by the same rule. See Make Transactions Span Operations.

On delete, the order reverses and is reference-counted: the association rows are removed inside a transaction, then an array column is only zeroed/freed if no remaining association references that hash. This is what lets two keys share one array safely. Feature sets are shared too, but they are not reference-counted — deleting an association never deletes its feature set; the set is left unreachable for a later compact() to sweep.

Persistence and Copying

The HDF5 backend buffers writes. Call flush() (which issues H5Fflush) before copying the files for backup or archival; afterward both system.h5 and system.h5.sqlite can be copied as a pair without closing the handle. The two files must always be kept together — neither is usable alone.

Protecting a Saved Artifact

The dangerous moment for a store is not the crash. Every path that writes an artifact stages to a temporary sibling and renames, so a power loss leaves either the old file or the new one, never a half-written one. What actually destroys a saved store is an ordinary call aimed at a path that already holds one.

Creating over an existing store is refused. Creating truncates the HDF5 file but only opens the catalog beside it, then stamps both halves with one fresh generation. Left unguarded, a build script re-run against last week's output produced an empty array file paired with the old catalog's rows — a store that opens cleanly, reports every series still present, and has nothing behind any of them. Nothing short of verify_integrity() notices. create therefore fails with StoreExists if either half is already at the path; the check covers both, because an orphaned catalog poisons fresh arrays exactly as an orphaned HDF5 file poisons a fresh catalog. Discarding the destination on purpose is a separate, explicitly named call (Store::create_replacing, overwrite=True in Python, overwrite=true in Julia), which removes both halves and the catalog's sidecars first.

Working on a Copy

open() defaults to read-write in every binding. That is the one place the library will damage a file you care about: mutations land in the artifact directly, and HDF5 has neither a journal nor a repair tool, so an interrupted write there is unrecoverable.

open_copy(src, dest) copies both halves and opens the copy, leaving the source byte-for-byte alone. Change the copy, then persist_to(src) — the original is only replaced by the final atomic rename. Both shipped consumers (infrasys and InfrastructureSystems.jl) already work this way; the call exists so the pattern lives in one place rather than being reimplemented per consumer.

Reserve a read-write open() on a user's artifact for when in-place mutation is genuinely what you mean, and prefer read_only=True for anything that only reads.

One writer, and not on a network filesystem

The store assumes a single writer. On an ordinary filesystem HDF5's file lock enforces most of that for you — this build links HDF5 2.0 with locking on. But it is configured best-effort: where the filesystem reports that locking is unavailable, HDF5 proceeds without it and says nothing. Lustre, GPFS, and NFS without lockd all land there, and they are exactly where large runs live. SQLite's WAL journaling is likewise unsafe over NFS.

So: keep a live store on local disk, let one process write it, and copy the finished artifact to shared storage afterwards. Two concurrent writers on a filesystem without working locks will corrupt the HDF5 file, and no amount of care inside this library can prevent it.

Inside one process the library enforces the rule itself, and more strictly than the lock does: a second Store on a path that is already open fails with StoreInUse whatever its mode, and so do create_replacing, open_without_catalog, persist_to, and persist_arrays_to aimed at a path another handle holds. A read-only handle is not exempt because its map from content hash to packed column is built once at open — after the writer removes a series and reuses the slot, that map points a live hash at another series' values, and libhdf5 sharing one file object between the two opens makes the reader's cache agree with it. The HDF5 lock does not see this case at all, and it is the easier mistake to make: an unclosed handle in a notebook or REPL, a fixture and a test body, a read-only handle opened for a report beside the writer. Close the handle you hold before opening another.

verify_integrity() is the backstop. Every array carries a SHA-256 companion, and the check re-reads and re-hashes all of them, so corruption is detectable even when it is not preventable — worth running against an artifact whose history you do not trust. The CLI exposes it as infrastore verify.

Where the Catalog Lives

The catalog does not have to be the .sqlite file while a store is open. CatalogMode picks between two placements, independently of where the arrays live:

Attached is the default and what a long-lived on-disk store wants: the CLI mutates one command per process and relies on each one landing.

The array half has its own moment of durability, because libhdf5 writes its caches back lazily while a catalog commit lands at once. Every write call that put new arrays into the file flushes it before committing the rows that name them, so a process killed right after the call returns leaves both halves agreeing. Inside a transaction the flush is deferred to the outermost commit, and a call that wrote nothing new (a re-add of content the store already holds) skips it. The flush is not free — a caller adding series one at a time pays it per call — and a bulk add or a transaction is how to pay it once for many writes. A transaction also holds that span's packed adds in a per-pool buffer and writes them as one block at the commit, so they land in the file the way a bulk add's do. InMemory suits a consumer that builds a store in a scratch directory beside its own volatile state — a System under construction, say. A crash loses that state regardless, so journaling the scratch catalog buys nothing, and skipping it removes per-commit WAL and fsync work. Arrays still stream to the HDF5 file, so this does not require the data to fit in memory. Nothing is durable until persist_to.

Two caveats. Opening with InMemory reads <path>.sqlite into RAM but still opens the HDF5 half in place, so mutations land in the original file; a caller that means to leave the source untouched until an explicit save wants open_copy (see Working on a Copy). And a scratch store that never reached its first save is a half-artifact: a stamped HDF5 file with no catalog. Reopening it as Attached creates a fresh, unstamped catalog beside it, and the paired stamp rejects that combination — which is the right answer, because the arrays are there but nothing names them.

persist_catalog() is the cheap way to land an in-memory catalog when the arrays are already in their final place. persist_to aimed at another path has to write the arrays again; persist_catalog writes only the .sqlite half, stamped to match the HDF5 file already sitting beside it. That is what makes InMemory usable for what it is good for — skipping per-commit journaling during a bulk load — without paying a full copy of the arrays to land the result. It is a checkpoint, not a mode switch: the catalog stays in RAM, and later changes are again RAM-only until the next call.

Saving: One Pair, Two Renames

persist_to writes both halves to temporary siblings, fsyncs them, and only then renames them into place, so a crash before the first rename leaves the destination untouched.

The renames cannot be made atomic together — POSIX renames one path at a time. A generation stamp covers the gap: each save mints a fresh value and writes it into both the HDF5 catalog_generation root attribute and the catalog's catalog_identity table. A crash between the two renames therefore leaves halves whose stamps disagree, and the next open fails with MismatchedArtifact instead of reading a store that quietly contradicts itself. The same check catches one half being copied without the other.

What the stamp does not give you is a destination that survives a failed save. The renames replace the target, so a crash between them destroys whatever pair was there before. That is a deliberate trade — loud, detectable loss beats silent corruption. Do not assume the destination is intact after a failed persist_to; recover by saving again from the store, which is still live and unchanged.

A store written before the stamp existed carries neither half of it. That is the one legitimate unstamped state, and it opens. One stamped half is not: every path that writes a stamp writes both halves together (create, persist_to, and compact, which carries the existing one across), so a lone stamp means a half was replaced, copied, or created without its partner. It is rejected as a mismatch, which closes the migration-window hole where a save interrupted between its two renames, onto a destination predating the stamp, would otherwise have paired new arrays with an old catalog in silence.

Each save stages through a sibling named uniquely to itself (<target>.persist-<tag>). A fixed name would be a corruption vector rather than a convenience: nothing locks a persist_to destination, so two processes saving to one path would each clear the other's in-flight temp while the stamping and rename that follow still resolve that name — publishing a partially written file as a finished save. The cost of uniqueness is that an interrupted save's temps are no longer swept by the next one; a temp belonging to a live concurrent save cannot be told apart from an abandoned one, so they are left for you to delete once no save is in flight.

The swap also clears any -wal/-shm sidecar beside the destination catalog. A sidecar there belongs to the catalog being replaced — a writer that crashed at that path leaves one — and SQLite would recover it over the database renamed into its place, resurrecting the replaced catalog's pages so the save silently would not take.

Saving an Attached store onto its own path is a no-op: the destination already is that store, and the flush at the start of persist_to has made it durable. The InMemory counterpart is real work rather than a no-op — the arrays are already at path and the save is what writes the catalog beside them, which is exactly the scratch-directory workflow.

Compaction rewrites only the HDF5 half, so it carries the existing stamp into the rewritten file rather than minting a new one — a fresh stamp there would manufacture exactly the mismatch the stamp exists to detect.

Compaction

compact() reclaims space in both halves of the artifact, and the two halves behave differently.

On the array side, compaction rewrites the file. Deleting a series frees its column slot (reused transparently by the next compatible write, via first_free) or unlinks its standalone dataset, but HDF5 cannot hand either back to the filesystem in place, so neither shrinks the .h5. Compaction therefore materializes every array the catalog still references into a fresh sibling file and renames it over the original, reopening the store on the result. What does not survive the trip: freed slots, datasets nothing references, and the slack in packed pools sized for growth rather than for the cohort actually stored. The report says how much went — slots_reclaimed, datasets_dropped, and bytes_reclaimed (how much smaller the file got).

Because the file is replaced, compaction assumes the compacting process is its only user. That is the store's single-writer model in general; the difference is that here a concurrent reader on Unix silently keeps reading the pre-compaction file, and on Windows the rename fails outright.

Compaction also sweeps the shared sets. Because feature sets and timestamp vectors are shared, deleting an association cannot cascade into them: removing the last association that referenced one leaves it unreachable. compact() deletes both — the feature set as a catalog row, the timestamp vector as an unlinked dataset — and reports the counts as feature_sets_reclaimed and timestamp_sets_reclaimed, before the rewrite, so the rewrite's liveness scan sees what survived. (Clearing is the exception on both counts: it orphans every feature set and every axis by construction, so it reclaims them outright rather than waiting for a compaction a cleared store may never get.)

See compact.

Content Addressing

Every array is identified by the SHA-256 hash of its contents, not by a name or an ID. Two series with byte-identical values therefore resolve to the same hash and are stored exactly once. This is the mechanism that lets many series share one underlying array.

The Array Hash

array_hash produces a deterministic 32-byte digest from a TypedArray. The hashed byte stream is, in order:

1. A dtype tag: the dtype name (f64, f32, i64, i32, i16, i8, u64, u32, u16, u8, bool) followed by a NUL byte.
2. The shape: the rank as a little-endian u64, then each dimension as a little-endian u64.
3. The elements in row-major order. Float dtypes canonicalize NaN to a single quiet-NaN bit pattern before hashing; integer and bool bytes are hashed verbatim.

flowchart LR
    T["dtype tag<br/>e.g. f64\\0"] --> S["rank + dims<br/>(u64 LE each)"]
    S --> E["elements<br/>(typed LE, row-major)"]
    E --> H["SHA-256"]
    H --> D["32-byte hash"]

    style T fill:#6f42c1,color:#fff
    style S fill:#6f42c1,color:#fff
    style E fill:#6f42c1,color:#fff
    style H fill:#4a9eff,color:#fff
    style D fill:#28a745,color:#fff

Identity is therefore (dtype, shape, content):

• dtype and shape are part of the identity. Two arrays with the same numbers but a different dtype, or a different shape, never collide. Reshaping or retyping changes the hash.
• NaN is canonicalized (float dtypes only). Any NaN, regardless of its payload bits, hashes as a single canonical quiet-NaN pattern, so semantically equal float arrays never collide on NaN representation.

The Features Hash

Feature maps are hashed by features_hash with the same discipline: a domain tag, the entry count, then, in the BTreeMap's sorted-by-key order, a length-prefixed key plus a kind tag and the value bytes for each entry. Because the map is always sorted, insertion order does not affect the hash:

{"model_year": 2030, "scenario": 1}   # hashes identically to
{"scenario": 1, "model_year": 2030}

The features hash does double duty in the catalog:

• It identifies the association. It is stored in the features_hash column and is part of the metadata uniqueness index — it is how the database distinguishes two otherwise-identical associations that differ only in their features.
• It is the key of the feature set. The feature_sets table is keyed by (features_hash, key), so a feature map is content-addressed exactly as an array is: stored once and shared by every association whose hash matches. The association row already carries that hash, so no join column, foreign key, or association id is needed. An empty map stores no rows at all.

Sharing is not a rare case, it is the common one. Thousands of components typically carry the same feature set (often the empty one, or a single scenario tag), and it collapses to one copy. The sharpest example is a DeterministicSingleTimeSeries: it is a view over the SingleTimeSeries it was derived from and has exactly the same features, so transform_single_time_series writes zero feature rows — it reuses the set the source series already stored, just as it reuses the source's array.

The Timestamps Hash

A NonSequentialTimeSeries carries its timestamps explicitly — as does a PersistentTimeSeries, whose breakpoints are the same thing under a different reading — and those are content-addressed too, by timestamps_hash: a domain tag, the count, then each instant as a unix millisecond count — the same values the store writes, so the hash addresses the stored form rather than a second serialization of it. Two vectors hash equal exactly when they hold the same instants in the same order.

Like the features hash, it does double duty:

• It names the stored vector. Each distinct time axis is one tsv_{timestamps_hash} dataset in the HDF5 file, so an axis shared by a thousand irregular series is stored once. Sharing is again the common case, not the rare one: irregular series in this domain sit on a schedule — outage windows, market intervals, event times — that many components observe together. Unlike the features hash, this one addresses something in the array half of the artifact: a timestamp vector is data, and the catalog holds only its hash.
• It is the cohort key of the packed array layout. Column-packing is only meaningful for arrays on a common time axis, since the chunking is timestamp-major; the interned hash is precisely the answer to "do these two series share one?", so the arrays of a cohort pool into one nsts_{dtype}_{shape}_{length}_{timestamps_hash} dataset. This is what lets a StaticReader sweep irregular series a timestamp at a time, exactly as it does regular ones.

Deduplication on Write

All three hashes deduplicate, by the same mechanism, on the same write.

The array. Store hashes the array and asks the backend whether that hash is already present:

• Present → the existing array is reused; no new array bytes are written. Only a new metadata association row is inserted.
• Absent → the array is written. A packed array (either static type) goes into the first free column of a compatible sts_… / nsts_… dataset, recording its hash in the companion hash variable; a standalone array (a dense forecast, or an irregular series alone on its time axis) becomes a new arr_{hash} variable.

The feature set and the timestamp vector. The association insert (MetadataStore::insert_batched) writes each under its hash with an INSERT OR IGNORE, so one some other association already stored is a no-op — equal hash implies equal content, so an ignored conflict cannot hide a different set behind the same hash. Within one batch a SharedSetCache remembers what has already been written, so a bulk add issues one write per distinct set rather than a no-op statement per row. This is what keeps a transform flat in feature count instead of linear in it, and a cohort of irregular series flat in timestamp count.

So storage cost scales with the number of distinct arrays, feature sets, and time axes, while metadata cost scales with the number of associations. A profile shared by a thousand generators costs one array, one feature set, and a thousand small rows.

Deletion is Reference-Counted

Because arrays are shared, deleting an association cannot blindly delete its array. On remove_by_ids (and clear_time_series), Store:

1. Deletes the matching association rows inside a SQLite transaction, collecting their data_hashes.
2. For each freed hash, counts how many associations still reference it.
3. Only frees the HDF5 column for hashes whose reference count has dropped to zero.

This keeps shared arrays alive until the last referencing key is gone.

The shared catalog rows are the deliberate exception

Feature sets and timestamp vectors are shared by the same mechanism but are not reference-counted, and the symmetry stops there. Counting references on every delete would make deletion pay for a scan the array side already pays for, to reclaim a handful of rows; the schema instead accepts unreachable rows and sweeps them in bulk. Concretely:

• No foreign key, no ON DELETE CASCADE. A cascade would be actively wrong: rows are shared, so deleting one association must not delete a set another association still uses.
• Deleting the last user leaves the row unreachable, rather than deleting it — the same end-state as the HDF5 side's freed packed slots and unlinked datasets, whose space also lingers until a compaction.
• Store::compact sweeps them. It deletes every feature set and every timestamp vector no association references any more, reporting the row counts as feature_sets_reclaimed and timestamp_sets_reclaimed. On an on-disk store the same call then rewrites the .h5 file, so the array side is reclaimed too.
• Clearing the whole store drops them all outright, since a cleared store orphans every row by construction and may never see a compaction.

The practical consequence: an unreachable row is never read (every lookup goes through an association's features_hash or timestamps_hash), so it costs bytes, not correctness — and a re-added association with the same features or timestamps silently adopts the row that was still sitting there.

Discovering Shared Series

Sharing is observable on read, not just an internal write optimization. Every metadata row carries its array's data_hash, so a caller can group series by that hash to learn which ones resolve to the same stored array — without reading any array bytes. Two cases land in the same group: arrays that were deduplicated because their content was identical, and a SingleTimeSeries together with any DeterministicSingleTimeSeries derived from it (the DST shares the backing array).

In the Julia binding, list_metadata returns TimeSeriesMetadata rows, each carrying a data_hash field (the 32 raw bytes, which hash and compare by content and so work directly as a Dict key); group by it to find the shared sets. count_array_references reports, for one hash, how many SingleTimeSeries and DeterministicSingleTimeSeries associations point at it.

groups = Dict{Vector{UInt8}, Vector{TimeSeriesMetadata}}()
for row in list_metadata(store)
    push!(get!(groups, row.data_hash, eltype(values(groups))[]), row)
end
shared = filter(((_, rows),) -> length(rows) > 1, groups)   # arrays referenced by >1 series

This read-side view is what lets a downstream caller collapse work across owners that share data. The ForecastReader builds on the same grouping internally: forecasts that share an array (and read plan) are read from disk once per timestamp no matter how many components reference them — see Window-read deduplication. (This is exactly how InfrastructureSystems.jl backs its own get_shared_time_series and forecast reader.)

Stability is a Contract

These hashes are part of the on-disk format. Any change to a hashing domain above that perturbs a stored hash is a format-breaking change and must bump DATA_FORMAT_VERSION. Treat the hashing rules above as fixed, not as an implementation detail.

The golden_hash_pin integration test guards the array domain by pinning the exact SHA-256 of one fixed input (an f64 array of [0, 1, 2, 3]). That is a tripwire, not a proof: it covers a single dtype, shape, and value set, and it does not pin features_hash at all. A change to the hashing rules can therefore break the format without breaking that test — the reasoning above, not the test, is the contract.

Integrity Verification

Because the stored hash and the stored data are independent on disk, they can be cross-checked. verify_integrity takes the catalog as the statement of what should be there, then checks the HDF5 half against it: it collects every array and every explicit timestamp vector the catalog references, reads each one back, recomputes array_hash / timestamps_hash from the stored values, and reports any mismatch between the recorded hash and the recomputed one — detecting silent corruption. A reference the file cannot satisfy is reported too, as a dangling one. See verify_integrity.

What it does not cover

The check runs from the catalog into the HDF5 file, and only in that direction. It never checks the catalog against itself, so a clean report is a statement about the stored content, not about the store as a whole. Three things fall outside it:

• a catalog row whose dtype, element_shape, or length misdescribes the array it points at. The hash addresses the array's own content, so an array that matches its hash passes the check while the row goes on lying about it.
• a missing catalog entirely. The two artifacts are one logical store, but nothing enforces that: opening read-write with the .sqlite half deleted silently recreates it empty, and the resulting store — zero time series, every array still present and now unreachable — verifies clean, because a catalog that references nothing is a clean bill of health here.
• anything the catalog does not reference. The sweep never reaches it, whatever state it is in; an unreachable array is the expected state after a delete, and compact is what reports it.

This is a deliberate scope rather than an oversight: content corruption is what happens silently, and the catalog has its own purpose-built checks — check_static_consistency for per-resolution grid agreement, and compact for the unreachable arrays and feature sets a delete leaves behind, which it reclaims and reports (an expected state, documented in the file format, not corruption). SQLite also enforces a good deal itself: the NOT NULL and CHECK constraints, and the two unique indexes that guarantee identity uniqueness.

If you need end-to-end assurance that a copied or restored store is intact, the operational rule in the file format is the one that matters: move, copy, and delete the .h5 and its .sqlite together.

Language Bindings

Every interface wraps the same Store. Understanding how each binding bridges to the core explains why the APIs look the way they do, how errors propagate, and what each layer owns.

flowchart TB
    PYAPP["Python code"] --> PYO3["PyO3 classes<br/>(infrastore_py)"]
    JLAPP["Julia code"] --> JLPKG["InfraStore.jl"]
    JLPKG -->|"ccall"| CABI["C ABI<br/>(infrastore_ffi)"]
    RUSTAPP["Rust client code"] --> RC["RemoteClient"]
    RC -->|"gRPC / HTTP2"| GS["gRPC server"]

    PYO3 --> STORE["Store"]
    CABI --> STORE
    GS --> STORE

    style STORE fill:#28a745,color:#fff
    style PYO3 fill:#17a2b8,color:#fff
    style CABI fill:#9558b2,color:#fff
    style JLPKG fill:#9558b2,color:#fff
    style GS fill:#ffc107,color:#000
    style RC fill:#ffc107,color:#000

Python (PyO3)

infrastore-py uses PyO3 to expose Store as native Python classes in a module importable as infrastore. The binding:

• Converts Python datetime/timedelta to chrono types and NumPy arrays (any shape) to TypedArrays at the boundary, supporting the full dtype set (f64, f32, the integer widths, bool).
• Translates the typed TimeSeriesError variants into a Python exception hierarchy rooted at TimeSeriesError (NotFoundError, DuplicateTimeSeriesError, InvalidParameterError, IntegrityError, ReadOnlyStoreError).
• Builds an abi3-py311 wheel, so one wheel works across CPython 3.11+ without recompiling.
• Converts a static series to a pyarrow.Table with to_arrow(), behind the optional arrow extra. This is the one place a binding reaches past numpy, and it is optional for that reason: pyarrow is several times the size of the wheel that would pull it in. What makes Arrow worth the seam is that its timestamp(unit, tz) is the same shape as the store's own model — an instant plus the spelling it was written in — so a table keeps a distinction pandas would flatten.

The metadata side is owned entirely by Rust; Python never touches SQLite directly. See the Python guide and Python API reference.

Julia (C ABI)

Julia does not call Rust directly. Instead, infrastore-ffi compiles a C-compatible cdylib with an opaque-handle API, and InfraStore.jl ccalls into it.

flowchart LR
    JL["InfraStore.jl<br/>structs hold Ptr{Cvoid}"] -->|"ccall infrastore_store_*"| LIB["libinfrastore_ffi"]
    LIB --> STORE["Store"]
    LIB -.->|"infrastore_last_error_message"| JL

    style JL fill:#9558b2,color:#fff
    style LIB fill:#6f42c1,color:#fff
    style STORE fill:#28a745,color:#fff

The conventions that shape the Julia API:

• Opaque handles. InfraStore and InfraStoreKey are pointers; the Julia structs wrap them and register finalizers (close!, _finalize_key) that call the matching ts_*_free function.
• Status codes plus thread-local error messages. Every C function returns an int32_t code. On a non-zero code, Julia calls infrastore_last_error_message to retrieve the detail string and raises the matching Julia exception type.
• Out-parameters and caller-owned buffers. Arrays come back through an out-pointer plus a length and a dtype code; Julia copies them into a Vector{T} for the requested element type and frees the Rust buffer with the deallocator matching the buffer's element type — infrastore_buffer_free_f64, infrastore_buffer_free_u8, infrastore_buffer_free_i64, or infrastore_buffer_free_u64 (shape/dims buffers).
• Features cross as JSON. Julia serializes the feature dict to a JSON string, which the FFI layer parses into a Features map.
• Forecasts are wrapped. InfraStore.jl exposes Deterministic / Probabilistic / Scenarios structs passed to the generic add_time_series!, id-addressed read_by_id getters, and transform_single_time_series!, so all four forecast types are usable from Julia.
• Bulk reads use a result handle. read_by_ids reads many full SingleTimeSeries at once: the FFI fetches them in one decompress-once pass per dataset into a InfraStoreBulkReadHandle (infrastore_store_bulk_read_single), and Julia reads each element out, then frees the handle. Python's store.read_by_ids exposes the same operation directly. read_by_ids addresses the same read by catalog association id and fills the same handle, so both reads decode by the same route: infrastore_bulk_result_item_name hands each item's name back beside its values, as infrastore_bulk_result_item_type does its type. Managed bulk writes already take the fast block-write path through the existing batch / add_time_series_bulk APIs.

InfraStore.jl loads the cdylib from the INFRASTORE_LIB environment variable when it is set, and otherwise from the libinfrastore_ffi artifact its Artifacts.toml pins to the matching GitHub Release (see Integrate with Julia). See the Julia guide, the C ABI reference, and the Julia API reference.

InfrastructureSystems.jl Integration

The model was shaped to drop into InfrastructureSystems.jl: owners are identified by integer component identifiers (i64), owner categories map to Component / SupplementalAttribute, and features accept string values so InfrastructureSystems.jl's feature dictionaries round-trip unchanged. The FFI exposes an attribute-based existence probe (infrastore_store_has_any_by_filter) and removal (infrastore_store_remove_by_ids), a whole-record metadata read (infrastore_store_get_metadata_by_id, reachable from attributes through infrastore_store_list_metadata), and a hash-based array fetch (infrastore_store_get_array_by_hash) so an InfrastructureSystems.jl-side store can keep its own key objects and reach the array layer directly.

gRPC Server and Client

infrastore-server wraps a Store in a tonic gRPC service generated from infrastore-proto. It exposes a read-only slice of the API and adds optional API-key auth. The matching async RemoteClient mirrors the read methods and maps gRPC Status codes back to TimeSeriesError::ConnectionError, so remote calls surface the same error type as local ones.

Writes are deliberately not exposed over gRPC — they require local filesystem access. The server is for fan-out reads of an existing store. See the gRPC Server guide and the gRPC API reference.

CLI (infrastore)

infrastore-cli builds the infrastore binary, a thin wrapper over the core Store for use from a terminal. Unlike the gRPC server it is not read-only: it opens the on-disk .h5 + .h5.sqlite pair directly and supports both reads and writes. Its shape:

• CSV in, store out. Numeric values come from a CSV; the metadata that does not fit a flat grid (owner, name, type, dtype, resolution, timestamps, units, features) is described in a descriptor JSON. All six dtypes and all six writable types are supported, forecasts included.
• A global -f/--format selects table (default), json, jsonl, or csv. Read commands render their results in it; write commands report their outcome in it (prose under table, a one-object status document under json/jsonl). Only template ignores it.
• Store access is isolated. All store opening lives behind one module, so a future remote/gRPC mode can be added without touching the command handlers; today there is no remote mode.

See the CLI guide and the CLI reference.

What Every Binding Shares

A file written by Python reads identically from Julia, Rust, or the server, because none of the bindings reimplement storage — they all funnel through the one core.

Feature Coverage Varies by Binding

The bindings funnel through one core, and the surface is now broadly consistent. Both static series types are available everywhere (read+write, except the read-only gRPC server), and forecasts read back across every interface. The remaining asymmetry is that the read-only gRPC server does not accept any writes:

The only gap is by design: writes (including forecasts added through add_time_series) require local filesystem access, so the read-only gRPC server serves forecast reads but not writes.

show() is Python-only for now: it is a REPL affordance, and the REPL each binding is used from already has one of its own — Julia has Base.show, and the CLI has store-info plus the list family. It composes existing catalog aggregate queries and adds no core API, so any binding that wants it can grow one without a change underneath.

Parquet lives in a crate of its own, infrastore-parquet, which the CLI depends on behind a cargo feature that is on by default — the infrastore binary anyone installs can read and write Parquet, because handing an analyst a file for DuckDB or polars is an ordinary reason to reach for the CLI. The feature stays switchable (--no-default-features --features vendored builds a lean binary), and the line it draws is between the binary and the libraries: infrastore-core, infrastore-py, and infrastore-ffi never link Arrow, which cargo tree --edges normal on each is the check for. The CLI is the only surface that reads and writes Parquet files (export -f parquet, add --parquet): a normalized, partitioned layout of two files per partition — a values file holding each distinct array once and a series file holding the catalog rows that name it, joined on (data_hash, time_axis) — specified in the Parquet layout reference. Python's to_arrow() / from_arrow are a different, in-memory thing — one two-column table per series, with the descriptors in the schema metadata rather than in columns — and are not a reader or writer for the CLI's files; a Python user who wants one writes the per-series table with pyarrow.parquet, or hands the CLI's directory to DuckDB or polars. The relationship between the two is one sentence: a series file's columns are to_arrow()'s metadata keys turned into columns, and the values file is its two columns keyed by the array. Nothing else has it: the C ABI and Julia would need the whole Arrow tree in the cdylib for a format their host languages already have readers for, and the gRPC server serves values, not files.

Materialized timestamps and from_timestamps both run in the core and reach Julia through two stateless ABI entry points, infrastore_grid_timestamps and infrastore_infer_period. That matters more than it looks: Julia is the one binding whose date library has calendar arithmetic of its own, and whose TimeZones overload steps a local clock the core deliberately does not — so a binding-side reimplementation would agree with the core only by luck. There is one implementation of "which instants does this series contain" in the project, and it is Period::add_to. Arrow tables are Python-only because Arrow is where the Python data ecosystem meets; the Julia counterpart would be a Tables.jl interface, which is a different contract and not yet asked for. A Deterministic converts through to_arrow_windows() into one table per window rather than one table, because its two grids — windows stepping by interval, rows stepping by resolution — overlap; Probabilistic and Scenarios wait on a decision about how to spell their third axis.

Developer Guides

These guides are written for developers building on infrastore from a specific language. Each one starts at installation and walks the full workflow — create or open a store, add series, query, read, and persist — with the idioms of that language, then points at the matching reference for exact signatures.

• Rust — Embed infrastore-core directly.
• Python — Use the infrastore PyO3 wheel.
• Julia — Use the InfraStore.jl package over the C ABI.
• CLI — Load and inspect a store from a terminal.
• gRPC Server & Client — Serve a store for remote readers.
• Benchmarks — Measure bulk-add and simulation-loop read performance.

If you are building a package on top of infrastore, read Embedding in a Parent Package first: it collects the contracts and the lifecycle these guides only show the individual calls for.

For the concepts underneath all of them, see the Explanation section — especially Time-Series Types, Data Model, and Readers, which apply equally to every binding.

Rust Developer Guide

This guide covers using infrastore-core from Rust. For exact signatures see the Rust API reference.

For a runnable end-to-end round-trip, crates/infrastore-core/examples/basic.rs creates an in-memory store, adds a SingleTimeSeries, and reads it back:

cargo run --manifest-path crates/infrastore-core/Cargo.toml --example basic

Add the Dependency

The crate is part of this workspace. From another crate in the workspace:

[dependencies]
infrastore-core = { path = "../infrastore-core" }
chrono = "0.4"

You will use chrono::Duration/DateTime<Utc> for time and the crate's TypedArray for values, since those are the types the API speaks.

Open or Create a Store

#![allow(unused)]
fn main() {
use std::path::Path;
use infrastore_core::Store;

// In-memory (tests, scratch work): no filesystem I/O.
let mut store = Store::create(None, true)?;

// On disk: writes system.h5 and system.h5.sqlite.
let mut store = Store::create(Some(Path::new("system.h5")), false)?;

// Reopen later, read-only.
let store = Store::open(Path::new("system.h5"), /* read_only */ true)?;
}

Add a Series

#![allow(unused)]
fn main() {
use chrono::{Duration, TimeZone, Utc};
use infrastore_core::{
    Features, FeatureValue, OwnerCategory, SingleTimeSeries, TimeSeriesData, TypedArray,
};

let initial = Utc.with_ymd_and_hms(2024, 1, 1, 0, 0, 0).unwrap();
let values: Vec<f64> = (0..24).map(|i| 100.0 + i as f64).collect();
let data = TypedArray::from_f64(vec![24], &values);   // shape [length]; or [length, k1, ...]
// The name is part of the series object, not an argument to `add_time_series`.
let ts = SingleTimeSeries::new(initial, Duration::hours(1), data, "load");

let mut features = Features::new();
features.insert("model_year".into(), FeatureValue::Int(2030));

let key = store.add_time_series(
    42,                                     // owner_id
    "Generator",                            // owner_type
    OwnerCategory::Component,
    TimeSeriesData::SingleTimeSeries(ts),
    features,
)?;
}

add_time_series returns a TimeSeriesId: the catalog row's id, which is both how every read and removal addresses the series and the one integer to store in your own object model when it needs to point at it (see Association ids). Adding a series whose identity already exists returns TimeSeriesError::DuplicateTimeSeries.

Descriptors

The descriptive attributes ride on the series object, not on the write call, so a read returns what the write declared:

#![allow(unused)]
fn main() {
use infrastore_core::{TimeReference, UnitSystem};

let ts = SingleTimeSeries::new(initial, Duration::hours(1), data, "load")
    .with_units("MW")
    .with_quantity_kind("ActivePower")
    .with_unit_system(UnitSystem::NaturalUnits)
    .with_component_field("max_active_power")
    .with_time_reference(TimeReference::Zone("America/Denver".into()))
    .with_application_data(r#"{"source": "weather_year_2012"}"#);
}

A native Rust caller declares time_reference itself. Every other binding infers it from the input type — Python from tzinfo, Julia from DateTime versus ZonedDateTime — but DateTime<Utc> is already an instant, so there is nothing to infer from. Leaving it unset means unspecified, which is not a claim the timestamps were written as UTC. It is not inert: a query bound must be spelled the way the series is, and a selection spanning both coherence groups is refused. See Time References.

None of these descriptors is part of a series' identity or of either content hash, so two adds differing only in one are a duplicate.

Bulk inserts

For many series at once, add_time_series_bulk takes a Vec<AddRequest> and commits the whole batch atomically — any error rolls back every array and association in the call:

#![allow(unused)]
fn main() {
use infrastore_core::AddRequest;

let added = store.add_time_series_bulk(vec![
    AddRequest {
        owner_id: 42,
        owner_type: "Generator".into(),
        owner_category: OwnerCategory::Component,
        data: TimeSeriesData::SingleTimeSeries(ts_a),   // the series carries its own name
        features: Features::new(),
        units: Some("MW".into()),
        application_data: None,                             // opaque package-owned payload (e.g. JSON)
        ..AddRequest::new(42, "Generator", OwnerCategory::Component,
                          TimeSeriesData::SingleTimeSeries(ts_b))   // the remaining descriptors unset
    },
    // ...or just AddRequest::new(...).with_features(...)
])?;
// `added` is one TimeSeriesId per request, in push order.
}

A bulk insert is also the fast write path: packed SingleTimeSeries are grouped by shape and written into batch-sized datasets so the timestamp-major HDF5 chunks are filled whole, rather than a slow column-at-a-time write. When you'd rather stream requests instead of building one big Vec, use a buffered session — it accumulates in memory and writes the same way on commit, discarding the buffer if dropped without committing:

#![allow(unused)]
fn main() {
let mut bulk = store.bulk_add();
for (owner_id, ts) in many_series {
    // `add` builds the AddRequest from its parts (application_data = None);
    // `push` takes a prebuilt AddRequest when you need to set application_data.
    bulk.add(owner_id, "Generator", OwnerCategory::Component,
        TimeSeriesData::SingleTimeSeries(ts), Features::new(), Some("MW".into()));
}
println!("staging {} series", bulk.len());   // also: bulk.is_empty()
let keys = bulk.commit()?;                    // consumes the session; keys in push order
}

BulkAdd buffers requests in memory and does no validation or I/O until commit, which is all-or-nothing. Dropping the session without committing writes nothing.

Adding series one at a time with add_time_series instead packs them incrementally into shared default-width datasets; that stays space-efficient but writes each column with a read-modify-write, so prefer a bulk insert or session when loading in volume.

The irregular types

NonSequentialTimeSeries carries an explicit instant per value instead of a grid. Its constructor validates that the timestamps are strictly increasing and match the data length, and returns Result<Self, String>:

#![allow(unused)]
fn main() {
use infrastore_core::NonSequentialTimeSeries;

// Values that exist only at these instants — asking between them is an error.
let outages = NonSequentialTimeSeries::new(instants, data, "forced_outage")?;

store.add_time_series(
    42, "ThermalStandard", OwnerCategory::Component,
    TimeSeriesData::NonSequentialTimeSeries(outages), Features::new(),
)?;
}

PersistentTimeSeries takes the same arguments and stores the same way — the vector is a set of breakpoints — but reads as a step function: the value at an instant is the one belonging to the greatest breakpoint at or before it, held forward past the last. Reaching back before the first breakpoint is an error, not a clamp.

#![allow(unused)]
fn main() {
use infrastore_core::PersistentTimeSeries;

// A monthly fuel price: twelve breakpoints, read at any simulation instant.
let prices = PersistentTimeSeries::new(month_starts, data, "fuel_price")?;

store.add_time_series(
    42, "ThermalStandard", OwnerCategory::Component,
    TimeSeriesData::PersistentTimeSeries(prices), Features::new(),
)?;
}

PersistentTimeSeries::value_at reads one instant — prices.value_at::<f64>(t)? — carrying the last breakpoint's value forward to it, with row_at as the shape-generic form and index_at / breakpoint_at for the row it came from. A time-range read begins at the breakpoint in force at start, so the slice always defines a value there. A columnar sweep is StaticReader, which for this type alone lets its columns sit on independent breakpoint vectors.

Values that are not numbers

A timestep's value can be a function rather than a number — a cost curve re-offered every hour, a pair of coefficients, a fixed-width tuple. The array still holds f64; what those elements mean is the series' element_type, and the values themselves are a DecodedValues.

Build such a series with from_values rather than new. It encodes the values into the array and declares the element type they imply:

#![allow(unused)]
fn main() {
use infrastore_core::{DecodedValues, ElementType, XyPoint};

// One input-output cost curve per hour; point counts may differ per timestep.
let curves = DecodedValues::PiecewiseLinear(vec![
    vec![XyPoint { x: 30.0, y: 1155.0 }, XyPoint { x: 100.0, y: 4120.0 }],
    vec![XyPoint { x: 30.0, y: 1353.0 }, XyPoint { x: 65.0, y: 2730.0 }, XyPoint { x: 100.0, y: 4223.0 }],
]);

let ts = SingleTimeSeries::from_values(initial, Duration::hours(1), &curves, "variable_cost")?;
assert_eq!(ts.element_type, ElementType::PiecewiseLinear);   // derived, not declared
}

That pairing is the point. An element_type and the array it describes are two independent things a caller can get out of step — add_time_series rejects the mismatch, but only after the fact. Deriving both from one input leaves nothing to get out of step, which is why from_values is preferred over new + with_element_type for every composite series.

The read side is decoded_values, which takes the element type and the leading-axis count off the value rather than asking for them:

#![allow(unused)]
fn main() {
let data = store.read_by_id(ts_id, ReadWindow::full())?;
assert_eq!(data.decoded_values()?, curves);
}

A plain numeric series decodes to DecodedValues::Raw — the stored elements already are the values, so the array is the answer. That is a result, not an error.

Every series type has from_values. On a forecast it carries more weight, because the leading axes are derived too: Deterministic::from_values fills [H, count] with H computed from horizon/resolution, so the values are one flat list in row-major order over those axes and nothing is reshaped by hand.

#![allow(unused)]
fn main() {
// H = 2 (a two-hour horizon at hourly resolution) x count = 2 windows = 4 curves.
let forecast = Deterministic::from_values(
    initial, Duration::hours(1), Duration::hours(2), Duration::hours(1), 2, &curves4, "offer",
)?;
}

Probabilistic::from_values and Scenarios::from_values are the same with a third leading axis, taken from percentiles.len() and scenario_count.

See Choosing a Type if you are deciding between these and a SingleTimeSeries.

Read a Series

Every read takes a TimeSeriesId — the one an add returned, or one off a list_metadata row. The row's own type decides what comes back:

#![allow(unused)]
fn main() {
use infrastore_core::ReadWindow;

let data = store.read_by_id(id, ReadWindow::full())?;
let single = data.as_single().expect("this row holds a SingleTimeSeries");
println!("{} values starting {}", single.length, single.initial_timestamp);
}

Slice on the time axis with a range; end is exclusive, it clips to what is there, and the returned series' initial_timestamp/length reflect the slice:

#![allow(unused)]
fn main() {
let start = Utc.with_ymd_and_hms(2024, 1, 1, 6, 0, 0).unwrap();
let end   = Utc.with_ymd_and_hms(2024, 1, 1, 12, 0, 0).unwrap();
let window = store.read_by_ids_range(&[id], TimeRange::new(start, end))?.remove(0);
}

A ReadWindow is the other half of that pair: it names exact steps (start plus a len of timesteps or a count of windows) and is checked, so an over-long request is an error rather than the smaller answer a range would clip to.

To read many whole series at once — say, loading everything for an interactive plot — read_by_ids takes a slice of ids and returns a TimeSeriesData per id in order. It reads packed SingleTimeSeries in one decompress-once pass per dataset, which is much cheaper than a read_by_id each (a single full-series read otherwise touches every chunk under the timestamp-major layout):

#![allow(unused)]
fn main() {
let series = store.read_by_ids(&ids, ReadWindow::full())?;
}

Query Metadata

list_metadata takes a ListFilter builder; every clause is ANDed, and the features clause is a subset match:

#![allow(unused)]
fn main() {
use infrastore_core::{ListFilter, OwnerCategory, TimeSeriesType};

let metas = store.list_metadata(
    ListFilter::new()
        .owner_id(42)
        .owner_category(OwnerCategory::Component)
        .time_series_type(TimeSeriesType::SingleTimeSeries)
        .name("load"),
)?;
for m in &metas {
    println!("{} {:?} units={:?}", m.name, m.resolution, m.units);
}

// Every row for one owner, an existence check, distinct resolutions, counts.
// The owner is the (owner_id, owner_category) pair.
let rows = store.list_metadata(
    ListFilter::new().owner_id(42).owner_category(OwnerCategory::Component),
)?;
let ids: Vec<_> = rows.iter().filter_map(|m| m.id).collect();
let present = store.association_exists(ids[0])?;
let resolutions = store.get_resolutions(Some(TimeSeriesType::SingleTimeSeries))?;
let counts = store.get_time_series_counts()?;
}

Each row carries the 32-byte content hash of the array it resolves to, so grouping a listing by data_hash finds the series that share stored data — one query, where this used to be a second key-shaped listing:

#![allow(unused)]
fn main() {
for m in store.list_metadata(ListFilter::new().owner_id(42))? {
    println!("{} -> {}", m.name, infrastore_core::hash_hex(&m.data_hash));
}
}

The low-level read path

To read values without reconstructing a full SingleTimeSeries — for example when bridging to another store that holds its own keys — resolve metadata and fetch the array by hash:

#![allow(unused)]
fn main() {
let meta = store.get_metadata_by_id(id)?.expect("the id still resolves");
let array = store.get_array_by_hash(&meta.data_hash)?;
}

Forecasts

Dense forecasts are written through the generic add_time_series by wrapping a Deterministic, Probabilistic, or Scenarios object in TimeSeriesData. Each forecast object holds a TypedArray in its native shape (the data model lists the conventional shapes per type); the store content-addresses it and records the windowing parameters:

#![allow(unused)]
fn main() {
use infrastore_core::{Deterministic, TimeSeriesData, TimeSeriesError, TypedArray};

// A Deterministic forecast: a [H, count, *E] array (here scalar steps, so [H, count]).
let (horizon_count, count) = (24, 7);
let values: Vec<f64> = vec![0.0; horizon_count * count];   // row-major, shape (horizon_count, count)
let data = TypedArray::from_f64(vec![horizon_count, count], &values);

// `new` returns Result<_, String>; map it if your function returns TimeSeriesError.
let forecast = Deterministic::new(
    initial, Duration::hours(1),      // initial_timestamp, resolution
    Duration::hours(24),              // horizon
    Duration::hours(24),              // interval
    count,
    data,
    "load_forecast",                  // name
).map_err(TimeSeriesError::InvalidParameter)?;

let key = store.add_time_series(
    42,                               // owner_id (i64)
    "Generator",
    OwnerCategory::Component,
    TimeSeriesData::Deterministic(forecast),
    Features::new(),
    Some("MW".into()),                // units
)?;
}

Probabilistic::new additionally takes the percentiles vector, and Scenarios::new takes a scenario_count; wrap them in TimeSeriesData::Probabilistic / TimeSeriesData::Scenarios the same way.

A DeterministicSingleTimeSeries is not added directly. Call transform_single_time_series to derive one from every stored SingleTimeSeries (it shares the backing array and derives count from the series length); it returns the number of series transformed. The trailing owner_category and resolution arguments are optional filters that restrict which series are transformed:

#![allow(unused)]
fn main() {
let n = store.transform_single_time_series(
    Duration::hours(24),                 // horizon
    Duration::hours(24),                 // interval
    Some(OwnerCategory::Component),      // owner_category filter (None = every category)
    Some(Duration::hours(1).into()),     // resolution filter (Option<Period>; None = every one)
)?;
}

read_by_id reconstructs forecasts too, returning a TimeSeriesData::Deterministic, Probabilistic, or Scenarios variant (a DeterministicSingleTimeSeries is synthesized into a Deterministic). Match on the variant or use the as_deterministic / as_probabilistic / as_scenarios accessors:

#![allow(unused)]
fn main() {
if let Some(d) = store.read_by_id(id, ReadWindow::full())?.as_deterministic() {
    // d.data is the TypedArray; d.horizon, d.interval, d.count carry the forecast parameters
}
}

The low-level path is still available when you only need the raw array: get_metadata_by_id exposes horizon, interval, count, and percentiles, and get_array_by_hash returns the TypedArray in its stored shape (to_f64_vec returns a Result<_, String>, so map the error if your function returns TimeSeriesError):

#![allow(unused)]
fn main() {
let meta = store.get_metadata_by_id(id)?.expect("the id still resolves");
let arr = store.get_array_by_hash(&meta.data_hash)?;   // arr.shape == [horizon_count, count]
let values = arr.to_f64_vec().map_err(TimeSeriesError::InvalidParameter)?;
}

When a forecast is known by its attributes rather than by an id you already hold, list_metadata is the identify half. Its type filter reads a request: TimeSeriesType::Deterministic matches a stored Deterministic or a DeterministicSingleTimeSeries, so a caller need not know which form the store holds, and each row still reports the concrete time_series_type that matched. Its id is what every read and removal takes:

#![allow(unused)]
fn main() {
use infrastore_core::TimeSeriesType;

let rows = store.list_metadata(
    ListFilter::new()
        .owner_id(42)
        .owner_category(OwnerCategory::Component)
        .name("load_forecast")
        .time_series_type(TimeSeriesType::Deterministic),
)?;
// A caller wanting exactly one checks that it got one -- there is no separate
// resolver, so ambiguity is the caller's to name rather than the store's.
let [meta] = &rows[..] else { panic!("expected exactly one match: {rows:?}") };
}

Per-Timestamp Reads (Simulation Loop)

The layout is optimized for "every component's value at one timestamp", and a reader is the API for that. Build one once outside the loop, then step it; the buffers are reused, so a tight loop allocates nothing. A reader is a passive plan — it does not borrow the Store, so the read goes through Store::static_read / Store::forecast_read, which fill it. See Readers for the concepts.

Static series

#![allow(unused)]
fn main() {
use infrastore_core::{ListFilter, TimeSeriesType};

let mut reader = store.build_static_reader(ListFilter::new().resolution(Duration::hours(1)))?;
for at in reader.timestamps().collect::<Vec<_>>() {
    store.static_read(&mut reader, at)?;
    for group in reader.groups() {
        let bytes = group.values();   // [num_columns, *element_shape], row-major LE
        // group.keys()[j] identifies column j; group.dtype(), group.element_shape()
    }
}
}

build_static_reader covers both static types, and the filter decides what must hold: SingleTimeSeries (the default) must pin a resolution and share one grid; NonSequentialTimeSeries must pin no resolution — reader.resolution() is then None — and its columns must lie on one timestamp vector:

#![allow(unused)]
fn main() {
let mut reader = store.build_static_reader(
    ListFilter::new().time_series_type(TimeSeriesType::NonSequentialTimeSeries),
)?;
}

Whatever the kind, reader.timestamps() walks the timeline, so the loop body above is unchanged. Coherence is validated at build time, where the error can name the series that disagree — there is no presence mask, and static_read errors rather than clamps on an off-grid instant.

When the matched SingleTimeSeries share no grid, name the span instead of inheriting one. build_static_reader_over gives each column an offset of its own, so ragged series sweep together:

#![allow(unused)]
fn main() {
use infrastore_core::ReadWindow;

let mut reader = store.build_static_reader_over(
    ListFilter::new().resolution(Duration::hours(1)),
    ReadWindow::from(anchor),   // .with_len(n) to pin the extent
)?;
}

Without a len the reader runs as far from the anchor as every matched series reaches. With one, a series that does not cover the span is an error naming it.

When the odd series out should not take part at all, filter to one grid instead — ListFilter::initial_timestamp and ListFilter::length match only the series already on it:

#![allow(unused)]
fn main() {
let mut reader = store.build_static_reader(
    ListFilter::new().resolution(Duration::hours(1)).initial_timestamp(anchor).length(8784),
)?;
}

See the Rust API reference for how the two remedies differ.

Forecasts

#![allow(unused)]
fn main() {
let mut reader = store.build_forecast_reader(
    ListFilter::new()
        .time_series_type(TimeSeriesType::Deterministic)
        .resolution(Duration::hours(1)),
)?;
for k in 0..reader.count() {
    let at = reader.interval().add_to(reader.initial_timestamp(), k as i64).unwrap();
    store.forecast_read(&mut reader, at)?;
    for (i, entry) in reader.entries().iter().enumerate() {
        let slot = reader.entry_slot(i);
        let bytes = slot.window();    // window of slot.window_shape(), row-major LE
        // entry.id() names the forecast; get_metadata_by_id resolves its owner
    }
}
}

Entries that reference the same array and slice it the same way share one WindowSlot, and forecast_read performs one backend read per slot — so a forecast shared by a hundred owners is decompressed once per step. Dedup your own per-entry work by slot the same way. Note that entry_slot(i) takes the entry index, while entry.slot() is that slot's index into slots().

Copy, Remove, and Maintain

copy_time_series re-points an existing association at another owner, optionally renaming it. It writes a metadata row only — arrays are content-addressed, so no data is duplicated — and it preserves the source's time_series_type (a DeterministicSingleTimeSeries stays one instead of being materialized into a dense Deterministic, which is what a read-then-write copy would produce):

#![allow(unused)]
fn main() {
let copied = store.copy_time_series(
    id,
    99,                    // dst_owner_id
    "Generator",           // dst_owner_type
    Some("load_copy"),     // new_name; None keeps the source name
)?;                        // returns the copy's own id; the source's is untouched
}

#![allow(unused)]
fn main() {
store.remove_by_ids(&[id])?;           // one series, or many in one transaction
// The owner is the (owner_id, owner_category) pair.
store.clear_time_series(Some((42, OwnerCategory::Component)))?;  // all series for an owner
store.clear_time_series(None)?;        // everything

let report = store.compact()?;         // reports reusable slots
let integrity = store.verify_integrity()?;  // stored content vs. the hashes the catalog names
assert!(integrity.errors.is_empty());
}

Removal is reference-counted: a shared array survives until its last referencing row is gone. count_array_references(&hash) returns the (SingleTimeSeries, DeterministicSingleTimeSeries) association counts on one array, which is how you tell whether removing a SingleTimeSeries would orphan a forecast derived from it.

Associations

Two catalog tables record relationships between entities the store does not otherwise model, wholly independently of time series: which supplemental attributes are attached to which components, and directed parent/child edges between components. Removing a time series never touches either, and vice versa — see Associations Between Entities.

Attachments are keyed on the (component_id, attribute_id) pair. The type names ride along for filtering and are not part of identity, so re-attaching the same pair under different type names is a duplicate and fails with DuplicateAssociation:

#![allow(unused)]
fn main() {
use infrastore_core::{SupplementalAttributeAssociation, SupplementalAttributeFilter};

store.add_supplemental_attribute_association(SupplementalAttributeAssociation {
    component_id: 42,
    component_type: "Generator".into(),
    attribute_id: 100,
    attribute_type: "GeographicInfo".into(),
})?;

// Bulk add is one all-or-nothing transaction.
store.add_supplemental_attribute_associations(vec![SupplementalAttributeAssociation {
    component_id: 43,
    component_type: "Generator".into(),
    attribute_id: 100,
    attribute_type: "GeographicInfo".into(),
}])?;
}

Filters are all-optional and ANDed; the default matches everything, which is what makes a bulk export/import round trip. Queries run in both directions:

#![allow(unused)]
fn main() {
// The attributes on one component...
let attrs =
    store.list_supplemental_attribute_ids(&SupplementalAttributeFilter::new().component_id(42))?;
assert_eq!(attrs, vec![100]);

// ...and the components carrying one attribute.
let owners =
    store.list_components_with_attributes(&SupplementalAttributeFilter::new().attribute_id(100))?;
assert_eq!(owners, vec![42, 43]);

// `*_types` filters take CONCRETE type names, rendered as SQL `IN (…)`. Expanding an
// abstract type into its subtypes is the caller's job — the store has no type hierarchy.
// An empty list is a deliberate "none of these" and matches nothing.
let geo = SupplementalAttributeFilter::new().attribute_types(["GeographicInfo"]);
assert_eq!(store.count_supplemental_attributes(&geo)?, 1);   // distinct attributes
assert_eq!(store.count_components_with_attributes(&geo)?, 2); // distinct components

for row in store.supplemental_attribute_summary()? {
    println!("{} on {}: {}", row.attribute_type, row.component_type, row.count);
}

// Removal returns a count. Matching nothing is `Ok(0)`, not an error: assert on the
// count yourself if you expected a hit.
let removed = store
    .remove_supplemental_attribute_associations(&SupplementalAttributeFilter::new().component_id(43))?;
assert_eq!(removed, 1);
}

Parent/child edges work the same way, except that identity is the ordered pair — the reverse of an edge is a different edge — and both endpoints are always components, so there is no category:

#![allow(unused)]
fn main() {
use infrastore_core::{ParentChildAssociation, ParentChildFilter};

store.add_parent_child_association(ParentChildAssociation {
    parent_id: 42,
    parent_type: "Generator".into(),
    child_id: 7,
    child_type: "Bus".into(),
})?;

assert_eq!(store.list_children(&ParentChildFilter::new().parent_id(42))?, vec![7]);
assert_eq!(store.list_parents(&ParentChildFilter::new().child_id(7))?, vec![42]);

// Renumbering a component rewrites both ends of every edge in one statement, so an
// edge that names it twice is counted once.
let updated = store.replace_parent_child_component_id(42, 99)?;
assert_eq!(updated, 1);
}

Neither table is reachable over gRPC or the infrastore CLI.

Persist to Disk

The HDF5 backend buffers writes. Call flush before copying the files for backup:

#![allow(unused)]
fn main() {
store.flush()?;   // H5Fflush; afterwards system.h5 + system.h5.sqlite can be copied as a pair
}

persist_to writes the whole store to a new path — both halves of the artifact, path and <path>.sqlite, overwriting anything already there. It works for an on-disk store (it flushes and copies the pair) and for an in-memory store, which is how you materialize a scratch store built with Store::create(None, true):

#![allow(unused)]
fn main() {
let mut store = Store::create(None, true)?;   // in-memory
// ... add series ...
store.persist_to(Path::new("system.h5"))?;   // writes system.h5 + system.h5.sqlite
}

Always keep the .h5 and .h5.sqlite files together — neither is usable alone.

Error Handling

Every fallible method returns Result<T, TimeSeriesError>. Match on the variant to react:

#![allow(unused)]
fn main() {
use infrastore_core::TimeSeriesError;

match store.read_by_id(id, ReadWindow::full()) {
    Ok(data) => { /* ... */ }
    Err(TimeSeriesError::NotFound) => { /* missing */ }
    Err(TimeSeriesError::ReadOnlyStore) => unreachable!("this is a read"),
    Err(e) => return Err(e.into()),
}
}

Threading

Store is Send but not Sync: the SQLite catalog holds a rusqlite::Connection, which is not shareable across threads. A store can therefore be moved into another thread, but it cannot be shared by reference — not even for concurrent reads.

To use one store from several threads, wrap it in external synchronization and serialize every access, reads included: Arc<Mutex<Store>> (this is exactly what the gRPC server does). The library does not coordinate multiple processes writing the same files either.

Python Developer Guide

This guide covers building on the infrastore PyO3 module, from installing the wheel to the calls a consumer package makes. For exact signatures and return shapes, see the Python API reference.

For complete programs rather than isolated snippets, use the repository's runnable Python examples. They cover static, non-sequential, deterministic, probabilistic, and scenario data; fixed tuples; every function-valued element type; feature-based selection; and conversion to Polars data frames.

Install

Python 3.11 or newer. The wheels are prebuilt and statically linked, so a consumer package such as infrasys needs nothing else:

pip install infrastore

The wheel is built against the abi3-py311 stable ABI, so one wheel works on CPython 3.11 and every newer 3.x without recompiling.

to_arrow() needs pyarrow, which is not installed by default — it is several times the size of everything else here, and nothing but that one method uses it:

pip install 'infrastore[arrow]'

From a checkout

Building from source needs the build tools (cmake, a C compiler, protobuf) — but no system HDF5. The binding is built with maturin; maturin develop compiles the extension and installs it into the active virtual environment:

cd crates/infrastore-py
python3 -m venv .venv && source .venv/bin/activate
pip install maturin pytest numpy tzdata  # tzdata: zoneinfo on Windows
pip install netCDF4 h5py                 # only for the HDF5-interop tests
maturin develop

python -c "import infrastore; print(infrastore.__version__)"
pytest ../../python/tests

To produce a wheel you can install elsewhere:

maturin build --release
# -> target/wheels/infrastore-<version>-cp311-abi3-<platform>.whl

Installing an unreleased core into a consumer's environment is the same maturin develop, run with that consumer's venv active.

If it does not import

• ImportError for the extension — Ensure you ran maturin develop in the active venv, or that you pip install-ed the wheel into the interpreter you are running.
• HDF5 build errors with HDF5_DIR set — Unset it. The vendored build compiles its own HDF5 and the variable redirects it at an external install while static libraries are still requested (see Build Prerequisites).
• InvalidParameterError on add — Pass a NumPy array (any shape) whose dtype is one of float64, float32, int64, int32, int16, int8, uint64, uint32, uint16, uint8, or bool; any other dtype (e.g. complex128 or a string dtype) raises. Feature values must be int/float/bool/str. Timestamps for a NonSequentialTimeSeries must be strictly increasing.

Import

from datetime import datetime, timedelta, timezone
import numpy as np
from infrastore import Store, SingleTimeSeries, OwnerCategory, TimeSeriesType

The module exposes Store and Transaction; the static series classes SingleTimeSeries and NonSequentialTimeSeries, and PersistentTimeSeries; the forecast classes Deterministic, Probabilistic, and Scenarios; the readers StaticReader and ForecastReader; the association records SupplementalAttributeAssociation and ParentChildAssociation; the TimeSeriesType and OwnerCategory enums; the init_tracing, encode_element_values, and decode_element_values functions; __version__; and an exception hierarchy rooted at TimeSeriesError.

If you are building a package on top of infrastore — the way infrasys does — read Embedding in a Parent Package alongside this guide: it covers the lifecycle (scratch store, persist_to, open_copy), id mapping, and lookup semantics that this page only shows the calls for.

Open or Create a Store

# In-memory: no filesystem I/O.
store = Store.create(in_memory=True)

# On disk: writes system.h5 and system.h5.sqlite.
store = Store.create(path="system.h5")

# Reopen read-only.
store = Store.open("system.h5", read_only=True)

Build a Series

SingleTimeSeries takes a timezone-aware datetime, a resolution (a timedelta or an ISO 8601 duration string such as "PT1H" — the string form is required for calendar periods like "P1M"), and a NumPy array:

ts = SingleTimeSeries(
    datetime(2024, 1, 1, tzinfo=timezone.utc),
    timedelta(hours=1),
    np.arange(24, dtype=np.float64) + 100,
    "load",  # name (required)
)

Use timezone-aware datetimes (UTC is stored). The binding is dtype-generic — it accepts and returns NumPy arrays of float64, float32, int64, int32, int16, int8, uint64, uint32, uint16, uint8, or bool, and whatever dtype you pass round-trips unchanged. The array may be multi-dimensional: shape (length,) for scalar steps, or (length, k1, …) to attach a per-step element shape (such as cost-curve coefficients). The required name is an association attribute carried on the object — the same array can be added under different names. Use NonSequentialTimeSeries(timestamps, data, name) for explicitly timestamped series, and PersistentTimeSeries(timestamps, data, name) for a sparse step function whose value holds forward between breakpoints (worked through in Step Functions).

Add a Series

series_id = store.add_time_series(
    owner_id=42,
    owner_type="Generator",
    owner_category=OwnerCategory.Component,
    time_series=ts,   # name and descriptors come from ts
    features={"model_year": 2030, "scenario": "high"},
)
# `series_id` is the catalog row's id: how every read and removal
# addresses the series, and one integer to keep in your own model.

features is a plain dict whose values are int, float, bool, or str. Adding a series whose identity already exists raises DuplicateTimeSeriesError. The add returns the id and nothing else. To see the rest of the row — owner_id, owner_category, time_series_type, name, resolution, interval, features, and the descriptors below — ask store.get_metadata_by_id(series_id), or store.list_metadata(...) for a set of them (resolution and interval come back as ISO 8601 duration strings or None).

Descriptors

Beyond units, a series can carry quantity_kind (what the values measure — "ActivePower"; the one record of what per-unit values mean), unit_system ("natural_units" or "component_base"; unset means unspecified, not natural units), component_field (the field on the owning component these values vary — "max_active_power"; also a filter), and application_data (an opaque string the store returns verbatim — the package-owned slot). All of them are set on the series object, not on the add, and each is a read-only property there:

ts = SingleTimeSeries(
    datetime(2024, 1, 1, tzinfo=timezone.utc), timedelta(hours=1), values, "load",
    units="MW", quantity_kind="ActivePower", unit_system="natural_units",
    component_field="max_active_power",
    application_data='{"source": "weather_year_2012"}',
)
series_id = store.add_time_series(
    owner_id=42, owner_type="Generator", owner_category=OwnerCategory.Component,
    time_series=ts,
)
assert store.read_by_id(series_id).quantity_kind == "ActivePower"

Keeping them on the object is what makes a read-then-add lossless: a series read from one store can be added to another unchanged, with no descriptor to re-supply and none the write could silently replace.

A series also records a time_reference — how its timestamps were spelled — inferred from the datetime it was built with: timezone.utc gives "utc", a fixed-offset tzinfo gives "-07:00", a ZoneInfo gives its name, and a naive datetime gives "zoneless". A naive datetime is accepted (it names a wall clock, not an instant) precisely because the read hands one back — naive and aware datetimes are never equal in Python, so returning the other kind would break every == a caller writes.

None of them is part of the key or of either content hash, so two adds that differ only in a descriptor are a duplicate. See Optional Descriptors and Time references.

Add many series at once

add_time_series_bulk takes a list of dicts mirroring add_time_series's keyword arguments and commits them in one catalog transaction, taking the block-sized HDF5 write path, so same-shaped series land in the same packed dataset.

An item carries exactly the keys add_time_series takes as parameters — owner_id, owner_type, owner_category, time_series, and optionally features — and any other key raises, as the misspelled keyword it almost always is. Everything that describes the values (units, quantity_kind, unit_system, component_field, application_data, element_type, time_reference) rides on the time_series object, exactly as it does on the single-series path.

ids = store.add_time_series_bulk([
    {"owner_id": i, "owner_type": "Generator", "owner_category": OwnerCategory.Component,
     "time_series": series[i]}
    for i in range(len(series))
])   # one catalog id per item, in input order; all-or-nothing

This is an order of magnitude faster than a bare loop of single adds, which pays one catalog transaction and one HDF5 flush per series. It is not faster than that same loop inside a transaction, which buffers and writes the identical datasets. What separates the two is one thing: this call writes the batch as a single block whatever its size, because you are already holding it, where the transaction holds its buffered adds to write_buffer_bytes and spills past it. Reach for this when the whole cohort is in hand as a list; reach for the loop when you would rather build the series one at a time than materialize every one of them first, and raise the budget if you want the single dataset back.

Transactions

Several operations that must take effect together — replacing a series is an add plus a remove — go inside a transaction. Removals are reversible only there; outside one the array bytes are reclaimed immediately.

with store.transaction():
    new_id = store.add_time_series(owner_id=42, owner_type="Generator",
                                   owner_category=OwnerCategory.Component,
                                   time_series=updated)
    store.remove_by_ids([old_id])
# committed on a clean exit, rolled back if the block raised

Blocks nest (each level is a savepoint), and the store holds the SQLite write lock until the outermost one ends. begin_transaction / commit_transaction / rollback_transaction are the explicit form.

A transaction is also where a run of single adds belongs. Nothing it writes is durable until the outermost commit, so packed adds inside one are buffered per shape group and written as one block at the commit — the datasets add_time_series_bulk of the same series would produce, without having to hold the batch yourself. Two things qualify it. The buffer spills a block early — an extra dataset, nothing else, the same split a batch that wide gets — once a group reaches the narrower of two widths, or once the unwritten arrays across every group cross 128 MiB:

• the columns one chunk row holds: 131,072 for a scalar float64;
• 128 MiB divided by one column's bytes, which for anything but a short series is the ceiling that actually binds — a 30,500-step float64 series is 244 KB a column, so its group spills at 550.

And a span holding a single array fills a shared-pool slot rather than claiming a dataset one column wide.

How wide a dataset a span writes

The 128 MiB is a default, not a law. write_buffer_bytes sets it, and through it how wide a dataset a run of single adds can produce:

store.write_buffer_bytes = 1 << 30      # 1 GiB
with store.transaction():
    for s in series:
        store.add_time_series(owner_id=..., owner_type="Generator",
                              owner_category=OwnerCategory.Component, time_series=s)
# one dataset, however many series that was

Raised far enough, the loop writes exactly what add_time_series_bulk of the same series writes — and the memory it costs is the memory the bulk call's caller was holding anyway. Measured on 2,000 hourly year-long float64 series: at the default they are ~0.26 s as one bulk call against ~0.31 s as a loop, one (8760, 2000) dataset against a (8760, 1915) and a (8760, 85); raise the budget and the loop lands the single dataset too.

The figure belongs to the Store object, not to the artifact — nothing is persisted, and a store reopened elsewhere is back to 128 MiB. It is a budget for the writing process, so a machine that cannot afford another machine's choice does not inherit it. Lowering it mid-transaction writes out whatever the buffer already holds beyond the new figure; zero raises InvalidParameterError, since a pool's width floors at one column and a zero budget would mean a dataset per array rather than no buffering at all. The chunk-row ceiling above it does not move.

Read a Series

got = store.read_by_id(series_id)
assert np.array_equal(np.asarray(got.data), np.asarray(ts.data))
print(got.length, got.initial_timestamp, got.resolution)

Slice on the time axis with a (start, end) tuple of datetimes (end exclusive). A range clips to what is there:

(window,) = store.read_by_ids_range(
    [series_id],
    (
        datetime(2024, 1, 1, 6, tzinfo=timezone.utc),
        datetime(2024, 1, 1, 12, tzinfo=timezone.utc),
    ),
)

read_by_id takes the other kind of slice: start_time plus a len of timesteps or a count of windows, checked rather than clipped, so an over-long request raises rather than quietly returning less.

To read many whole series at once — e.g. loading everything for a plot — read_by_ids takes a list of ids and returns the typed series objects in the same order. Packed SingleTimeSeries are read in one decompress-once pass per dataset, which is much faster than a read_by_id each:

series = store.read_by_ids(ids)
window = store.read_by_ids_range(ids, (start, end))   # the same clip on every series

As a table

A read hands back the values as a numpy array with the timeline beside it, not fused into it. When you want the two together — to plot, to write Parquet, to hand to pandas or polars — to_arrow() builds a two-column pyarrow.Table of timestamp and value:

table = store.read_by_id(series_id).to_arrow()
table.to_pandas()

It works on all three static types, and needs the arrow extra. The timestamp column is typed in the series' own spelling — timestamp[ms, tz=America/Denver] for a zoned series, an unzoned timestamp[ms] for a zoneless one — and the descriptive attributes ride in table.schema.metadata. A SingleTimeSeries grid is materialized calendar-aware, so a monthly series lands on month ends rather than on a multiple of 30 days. See to_arrow().

Without pyarrow, timestamps is the same timeline as a plain list of datetimes:

got = store.read_by_id(series_id)
list(zip(got.timestamps, got.data))

A Deterministic converts to one table per window instead, keyed by issue time:

windows = store.read_by_id(forecast_id).to_arrow_windows()
windows[datetime(2024, 1, 2, tzinfo=timezone.utc)]   # that window's forecast

Each value looks exactly like a static series' table. It is a dict rather than one table because a forecast has two grids that overlap — windows step by interval, rows inside a window by resolution — so a day-ahead forecast reissued hourly shares 23 of every 24 instants between neighbouring windows. The dict iterates chronologically. See to_arrow_windows().

Back from a table

from_arrow is the inverse, on the same three types — and it reads a table anything wrote, not just one to_arrow() produced:

import pyarrow.parquet as pq

series = SingleTimeSeries.from_arrow(pq.read_table("load.parquet"))
store.add_time_series(42, "Generator", OwnerCategory.Component, series)

A table to_arrow() wrote round-trips with no arguments, because the metadata is the descriptor. A foreign table — from a dataframe, or a Parquet file someone else wrote — needs a name= at minimum, since a name is part of a series' identity; everything else is inferred from the Arrow schema. Note what a table does not carry: the owner and the catalog id, because to_arrow() is a method on a value object and a series built here is not filed anywhere. You supply the owner to add_time_series, as you would for any other series.

The inference rules and the four things that are refused rather than coerced (nulls, sub-millisecond timestamps, rows that leave a declared grid, decoded struct/list value columns) are in the reference.

to_arrow() and from_arrow() are per-series, in-memory conveniences -- one series, one table. They are not the CLI's file format: infrastore export -f parquet writes a values/series file pair per partition, each distinct array once and one catalog row per series, which from_arrow does not read. To move a whole store through Parquet, use the CLI at both ends.

Datetimes and precision

Every datetime must be timezone-aware (any zone; converted to UTC on the way in, UTC on the way out), and a naive one raises InvalidParameterError. A stored instant — an initial timestamp, a NonSequentialTimeSeries timestamp or PersistentTimeSeries breakpoint — must also be a whole number of milliseconds, so quantize datetime.now(timezone.utc) before storing it; query bounds such as time_range are unconstrained. See Datetimes.

Per-Timestamp Reads (Simulation Loop)

read_by_id hands back a whole series or forecast. Simulations instead walk the timeline and, at each timestamp, want the value of every series at that instant. For that, build a reader once and drive it in a loop — it pins one resolution and reuses its output buffers, so the loop allocates almost nothing. StaticReader serves SingleTimeSeries; ForecastReader serves forecasts. (Full signatures: Python API reference.)

Static series

Series are grouped by (dtype, element_shape); each group's group_values is one dense (num_columns, *element_shape) array whose columns line up with that group's ids. All matched series must share one grid (initial_timestamp + length), validated at build.

reader = store.build_static_reader(timedelta(hours=1))
grid = reader.grid()               # {"initial_timestamp", "resolution", "length", "time_series_type"}
groups = reader.groups()           # each: {"dtype", "element_type", "element_shape", "ids"}
for ts in reader.timestamps():
    store.static_read(reader, ts)
    for i, g in enumerate(groups):
        vals = reader.group_values(i)   # (num_columns, *element_shape); column j ↔ g["ids"][j]

When the series do not share a grid

Most real systems do not meet that requirement — a year of load beside a week of an outage schedule, or one component logged from an hour later than the rest — and the build then raises, naming the series that diverges. Give the reader a span instead of letting it inherit one:

reader = store.build_static_reader(
    timedelta(hours=1),
    window_start=datetime(2024, 1, 1, 7, tzinfo=timezone.utc),
    window_length=8760,   # optional: without it, as far as *every* matched series reaches
)

Each column then reads at an offset of its own, so ragged series sweep together. The span is checked, not clamped: a matched series that does not cover it raises InvalidParameterError naming that series rather than dropping its column, and the anchor must land on each series' own step boundaries. See reader windows.

Sometimes the odd series out should not take part at all — a stray day of data beside a year of it is a different component, not a shorter view of the same sweep. Then filter to one grid instead, with initial_timestamp and length:

reader = store.build_static_reader(
    timedelta(hours=1),
    initial_timestamp=datetime(2024, 1, 1, 7, tzinfo=timezone.utc),
    length=8784,
)

The window sweeps a span across whatever matched; the filter matches only the series already on that grid. static_summary() shows which grids a store holds, and the filter reaches every other filter-taking call too — list_metadata, remove_by_filter, and the rest. See selecting one grid.

Forecasts

entry_values(i) returns the window backing entries()[i], shaped (horizon, *element_shape) for Deterministic/DeterministicSingleTimeSeries, (num_percentiles, horizon, *element_shape) for Probabilistic, and (scenario_count, horizon, *element_shape) for Scenarios. A Deterministic reader is abstract — it also includes any DeterministicSingleTimeSeries (read into identical windows).

reader = store.build_forecast_reader(TimeSeriesType.Deterministic, timedelta(hours=1))
tl = reader.timeline()             # {"initial_timestamp", "resolution", "interval", "count", ...}
entries = reader.entries()         # list[int]: catalog ids, parallel to entry_values
for ts in reader.timestamps():
    store.forecast_read(reader, ts)
    for i, entry_id in enumerate(entries):
        window = reader.entry_values(i)   # the window for that id's series

Shared forecasts are read once

Forecasts that share a backing array (deduplicated identical data, or several DeterministicSingleTimeSeries over one SingleTimeSeries) collapse to a single window slot. forecast_read reads each slot from the .h5 file once per timestamp, so a forecast shared by 10 components costs one read, not ten. reader.num_slots() is the physical read count, and reader.entry_slot(i) says which slot an entry uses — group by slot to materialize each unique window only once on the Python side too:

store.forecast_read(reader, ts)
windows: dict[int, np.ndarray] = {}
for i, key in enumerate(entries):
    window = windows.setdefault(reader.entry_slot(i), reader.entry_values(i))

Step Functions (PersistentTimeSeries)

A PersistentTimeSeries is a sparse step function: a strictly increasing vector of breakpoints plus one value each, where the value at an arbitrary instant is the one belonging to the greatest breakpoint at or before it. The motivating data is a monthly fuel or gas price curve — a dozen breakpoints a simulation reads at timestamps that almost never land on one. The same curve stored as a NonSequentialTimeSeries would raise at nearly every step, because an irregular series has no value between its timestamps; that difference in read semantics is the whole reason this is a separate type. See Time series types for the model.

Build and add

Construction is identical to NonSequentialTimeSeries — same arguments, same validation, same spelling inference from tzinfo:

from infrastore import PersistentTimeSeries

breakpoints = [datetime(2024, m, 1, tzinfo=timezone.utc) for m in (1, 4, 7, 10)]
prices = PersistentTimeSeries(
    breakpoints,
    np.array([3.5, 4.25, 5.0, 4.75]),
    "gas_price",
    units="USD/MMBtu",
    component_field="fuel_cost",
    # Whether a curve is expanded to a full series or collapsed to one scalar is
    # your application's policy, and rides here where the store never reads it.
    application_data='{"as_time_series": false, "force_scalar_mode": "midpoint"}',
)

price_id = store.add_time_series(
    owner_id=7,
    owner_type="ThermalStandard",
    owner_category=OwnerCategory.Component,
    time_series=prices,
)

read_by_id(price_id) hands back a PersistentTimeSeries whose timestamps are the breakpoints and whose data holds one value each — the same dtype and shape rules as every other static series, multi-dimensional per-breakpoint values included.

Read a window

A range read slices on the step function's own terms: the result begins at the breakpoint in force at start, so it always defines a value at the start of the window you asked for.

(window,) = store.read_by_ids_range(
    [price_id],
    (datetime(2024, 4, 10, tzinfo=timezone.utc), datetime(2024, 9, 1, tzinfo=timezone.utc)),
)
window.timestamps   # [2024-04-01, 2024-07-01] — the April step, not the first one inside the window
window.data         # array([4.25, 5.  ])

Past the last breakpoint the last value holds forever, so a window opening after the end comes back with that one row. Before the first breakpoint a step function is undefined: a non-empty window starting there raises InvalidParameterError rather than clamping. (A zero-width range, end == start, selects nothing — here as for every type.) read_by_id's start_time + len window is checked rather than sliced, so it must name one of the breakpoints; reach for read_by_ids_range when the instant is arbitrary.

Sweep step functions in the simulation loop

A StaticReader filtered to the type is the per-timestamp path, and it is the one place the one-timeline-per-reader rule bends: a step function has a value at every instant from its own first breakpoint on, so the columns need not share a breakpoint vector. Per-fuel curves whose breakpoints do not line up still build one reader.

reader = store.build_static_reader(
    time_series_type=TimeSeriesType.PersistentTimeSeries,   # no resolution — passing one raises
    component_field="fuel_cost",
)
grid = reader.grid()                   # grid["resolution"] is None: a step function has no step
groups = reader.groups()
for at in reader.timestamps():         # the sorted union of every column's breakpoints
    store.static_read(reader, at)
    for i, g in enumerate(groups):
        vals = reader.group_values(i)  # the value in force at `at`; column j ↔ g["ids"][j]

timestamps() is the union of the columns' breakpoints, so a position on it is not a storage row for any one column — each column independently reports the value in force there. There is still no presence mask: reading at an instant before some column's first breakpoint raises InvalidParameterError naming that column's association id. Either filter the reader down to columns that start early enough, or begin the sweep at the latest first breakpoint among them.

The sweep need not follow that union axis at all. static_read accepts any instant every column defines a value at, so driving this reader at your SingleTimeSeries grid's timestamps — the simulation's own clock — works and is usually what an application wants:

for at in load_reader.timestamps():        # the hourly grid the simulation runs on
    store.static_read(load_reader, at)
    store.static_read(reader, at)          # each fuel price, held forward to this hour

These rows do not travel in an OpenAPI document

PersistentTimeSeries is an infrastore-local extension, and the vendored wire contract is a oneOf over six canonical Sienna types with no schema for a seventh. So export_time_series_associations_openapi omits persistent rows — a mixed store still exports its six-type rows — a filter naming the type raises InvalidParameterError rather than answering with an empty array, and an import refuses a document that carries one. The series themselves are unaffected: they live in the artifact, which holds them in full. Ask the catalog what a document leaves behind:

left_behind = store.list_metadata(time_series_type=TimeSeriesType.PersistentTimeSeries)

Custom Element Types

By default an array's elements are plain numbers of its dtype. An element_type says otherwise: what the trailing per-step dimension of the array actually means. It is metadata, not a different storage format — the array is still the same typed HDF5 dataset — but it is what lets a reader turn the raw floats back into the values you meant. See Element types for the full grammar and the byte layout each kind produces; this section works through each one from Python.

from_values and decoded_values

Every series type has a from_values classmethod that takes the values themselves. It encodes them and records the element type they imply, so the two cannot disagree:

curves = [
    [{"x": 0.0, "y": 1.0}, {"x": 1.0, "y": 3.0}],
    [{"x": 0.0, "y": 2.0}],
]
ts = SingleTimeSeries.from_values(
    datetime(2024, 1, 1, tzinfo=timezone.utc), timedelta(hours=1), curves, "cost_curve",
)
assert ts.element_type == "piecewise_linear"     # nobody declared it

series_id = store.add_time_series(
    owner_id=42, owner_type="Generator", owner_category=OwnerCategory.Component, time_series=ts,
)
assert store.read_by_id(series_id).decoded_values() == curves

decoded_values() is the read-side half: the element type and the number of leading axes both come off the series, so there is nothing left to pass. It returns None for a scalar element type and for any array whose dtype is not float64 — there the stored elements already are the values, and .data is the answer.

Which element type a payload implies is read off the shape of a row; the five shapes are disjoint:

element_type= is still accepted on from_values, as an assertion rather than an override: it raises InvalidParameterError if it disagrees with the values. Where the values name nothing it is the only thing to go on — an empty values, or rows that are all empty and read equally as a curve with no points or a tuple with no fields.

Underneath sit encode_element_values(values, element_type, leading_dims) and decode_element_values(array, element_type, leading_dims), which the rest of this section uses to show what each element type packs into. Reach for them directly when there is no series to hang the values on — decoding an array that arrived on its own — or for the one series from_values cannot name: an empty tuple(N,f64), whose arity lives in rows it does not have.

Composite values: tuple(N,dtype)

A tuple(N,dtype) is bytes-identical to a plain array shaped (length, N) — declaring it changes nothing about what is stored, only how a reader should group the trailing N values: as one composite value (three cost-curve coefficients), not N independent samples. Build it like any other multi-dimensional series and declare the type on the constructor:

coeffs = np.array([[1.0, 0.5, 12.0], [1.1, 0.4, 11.5]])  # (length=2, N=3)
ts = SingleTimeSeries(
    datetime(2024, 1, 1, tzinfo=timezone.utc), timedelta(hours=1), coeffs, "cost_coeffs",
    element_type="tuple(3,f64)",
)

This is the one element type where the constructor stays the natural call: there is no packing to build, so an array you already hold in numpy needs no encoding step. from_values accepts the same values as a list of list[float] rows, and is the better fit when that is the shape you have.

decoded_values() and decode_element_values only unpack f64 arrays — for any other dtype (tuple(3,i32), say) they return None, because there is nothing to unpack: the stored rows already are the tuples, and you read them straight off .data. encode_element_values is a convenience for the f64 case only; it also always builds f64, so neither it nor from_values can produce a tuple of any other dtype.

Fixed-width coefficients: linear_function, quadratic_function

These give every timestep a small, fixed number of function coefficients — a proportional and a constant term for a line, plus a quadratic term for a parabola — packed as f64 regardless of the rest of the series. The values are a list of per-timestep dicts, matching keys to the function's coefficients:

from infrastore import encode_element_values, decode_element_values

curves = [
    {"proportional": 1.0, "constant": 2.0},
    {"proportional": 1.2, "constant": 1.8},
]
array = encode_element_values(curves, "linear_function")   # shape (2, 2), f64
assert decode_element_values(array, "linear_function") == curves

Or, without naming the type or holding the array at all:

ts = SingleTimeSeries.from_values(
    datetime(2024, 1, 1, tzinfo=timezone.utc), timedelta(hours=1), curves, "marginal_cost",
)
series_id = store.add_time_series(
    owner_id=7, owner_type="ThermalStandard", owner_category=OwnerCategory.Component,
    time_series=ts,
)
assert store.read_by_id(series_id).decoded_values() == curves

quadratic_function is the same shape, with a "quadratic" key added and row width 3 instead of 2. Both raise if a row doesn't match the required width exactly — there is no padding for these two, because every timestep genuinely has the same number of coefficients.

Ragged curves: piecewise_linear, piecewise_step

These are the element types built for a variable number of points per timestep — a case covered end to end in the runnable single_custom_elements.py example. encode_element_values finds the widest row across the whole array, then packs every timestep as a leading count n followed by its points, zero-padded out to that common width; decoding reads n back off each row and returns exactly that many points, ignoring the padding.

A piecewise_linear timestep is a list of {"x", "y"} knots:

curves = [
    [{"x": 0.0, "y": 1.0}, {"x": 1.0, "y": 3.0}, {"x": 2.0, "y": 5.0}],                      # 3 points
    [{"x": 0.0, "y": 2.0}, {"x": 1.0, "y": 4.0}, {"x": 2.0, "y": 6.0}, {"x": 3.0, "y": 8.0}], # 4 points
]
ts = SingleTimeSeries.from_values(
    datetime(2024, 1, 1, tzinfo=timezone.utc), timedelta(hours=1), curves, "cost_curve",
)
assert ts.data.shape == (2, 1 + 2 * 4)   # widest row wins; the 3-point row is padded
series_id = store.add_time_series(
    owner_id=42, owner_type="Generator", owner_category=OwnerCategory.Component, time_series=ts,
)

back = store.read_by_id(series_id)
assert back.decoded_values() == curves   # 3- and 4-point rows both exact

A piecewise_step timestep decodes to a different shape — one dict of parallel arrays rather than a list of points, since a step function has one fewer y than x: each y is the value between two adjacent x's, so n coordinates bound n - 1 steps and the last coordinate is the right-hand end of the curve rather than the start of an open final step. Nothing is held forward past it — that is a PersistentTimeSeries, which is a time series type rather than an element type:

steps = [
    {"x": [0.0, 1.0, 2.5], "y": [10.0, 20.0]},   # 3 x's, 2 steps
    {"x": [0.0, 5.0], "y": [7.5]},               # 2 x's, 1 step
]
array = encode_element_values(steps, "piecewise_step")

The padded width is fixed by whatever encode_element_values saw in that one call. Add a timestep with more points later than any seen so far, and the wider row width makes it a different packed HDF5 dataset (element_shape differs), not an in-place resize of the array you already wrote — the same packing rule every other same-shaped-series pooling in this store follows.

Forecasts (Deterministic, Probabilistic, Scenarios) use these same element types over their extra leading axes. This is where from_values saves the most: the leading dimensions come from arguments the forecast constructor already takes, so nothing is computed by hand.

# H = horizon / resolution = 2, so [H, count] = [2, 2] wants four curves, entry
# `i * count + j` being window `j`'s step `i`.
forecast = Deterministic.from_values(
    start, timedelta(hours=1), timedelta(hours=2), timedelta(hours=1), 2, curves * 2,
    "cost_curve",
)
assert forecast.data.shape == (2, 2, 9)

Scenarios.from_values takes scenario_count explicitly, where its constructor reads it off the array's first axis — there is no array yet to read it from. Going through encode_element_values instead means passing leading_dims=[horizon, count] (or [percentiles, horizon, count] / [scenarios, horizon, count]) yourself, in place of the default single-axis case. See Forecasts above for the shapes those types read back as.

Query Metadata

list_metadata returns a list of plain dicts, filtered by any combination of arguments (the features argument is a subset match):

for m in store.list_metadata(
    owner_id=42,
    owner_category=OwnerCategory.Component,
    time_series_type=TimeSeriesType.SingleTimeSeries,
):
    print(m["name"], m["resolution"], m["units"], m["features"])

# The owner is the (owner_id, owner_category) pair.
rows = store.list_metadata(owner_id=42, owner_category=OwnerCategory.Component)
ids = [r["id"] for r in rows]
exists = store.association_exists(ids[0])
resolutions = store.get_resolutions()          # list[str] (ISO 8601 durations)
counts = store.get_time_series_counts()        # dict

What is in here?

Before querying anything in particular, show() prints the shape of the whole store — the time-series associations by type, the arrays behind them, the owners, and both association catalogs:

store.show()
# Store: system.h5 (read-write)
# Time series: 128 associations over 128 distinct arrays
#   SingleTimeSeries      100
#   PersistentTimeSeries    8
#   Deterministic          20
# Owners with time series: 108 components, 0 supplemental attributes
# Supplemental attribute attachments: 12
# Parent/child edges: 5

It is aggregate catalog queries only, so it stays fast on a large store, and it takes file= like print does. For the numbers themselves rather than the rendering, use counts_by_type(), time_series_counts_detailed(), num_distinct_arrays(), and the count_* methods — see show().

Remove and Maintain

store.remove_by_ids([series_id])   # one series, or many in one transaction
# The owner is the (owner_id, owner_category) pair.
n = store.clear_time_series(owner_id=42, owner_category=OwnerCategory.Component)  # one owner; returns count
store.clear_time_series()                                  # remove everything

# Reassign every series from one owner to another; returns the number moved.
moved = store.replace_owner(42, 43, OwnerCategory.Component)

report = store.compact()            # rewrites the .h5 from the live set; the report includes
                                   #  "slots_reclaimed", "datasets_dropped",
                                   #  "feature_sets_reclaimed", "timestamp_sets_reclaimed",
                                   #  "bytes_reclaimed"
integrity = store.verify_integrity()   # {"ok": True, "errors": []} when every array and time
                                   # axis the catalog names matches its recorded hash

Associations

Two catalog tables record relationships between entities the store does not otherwise model, wholly independently of time series: which supplemental attributes are attached to which components, and directed parent/child edges between components. Removing a time series never touches either, and vice versa — see Associations Between Entities.

Filter arguments are keyword-only, all optional, and ANDed; passing none matches everything.

from infrastore import (
    SupplementalAttributeAssociation,
    ParentChildAssociation,
    DuplicateAssociationError,
)

store.add_supplemental_attribute_association(
    SupplementalAttributeAssociation(42, "Generator", 100, "GeographicInfo")
)

# Bulk add is one all-or-nothing transaction.
store.add_supplemental_attribute_associations([
    SupplementalAttributeAssociation(43, "Generator", 100, "GeographicInfo"),
    SupplementalAttributeAssociation(43, "Generator", 101, "Outage"),
])

# Queries run in both directions, returning distinct ids in ascending order.
assert store.list_supplemental_attribute_ids(component_id=43) == [100, 101]
assert store.list_components_with_attributes(attribute_id=100) == [42, 43]
assert store.has_supplemental_attribute_association(component_id=42, attribute_id=100)

# `*_types` filters take CONCRETE type names. Expanding an abstract type into its
# subtypes is the caller's job — the store has no type hierarchy. An empty list is a
# deliberate "none of these" and matches nothing.
assert store.list_supplemental_attribute_ids(
    component_id=43, attribute_types=["Outage"]
) == [101]

assert store.count_supplemental_attributes() == 2        # distinct attributes
assert store.count_components_with_attributes() == 2     # distinct components
store.supplemental_attribute_counts_by_type()
# [('GeographicInfo', 2), ('Outage', 1)]
store.supplemental_attribute_summary()
# [{'component_type': 'Generator', 'attribute_type': 'GeographicInfo', 'count': 2}, ...]

Identity is the (component_id, attribute_id) pair. The type names ride along for filtering and are not part of it, so re-attaching the same pair under different type names is still a duplicate:

try:
    store.add_supplemental_attribute_association(
        SupplementalAttributeAssociation(42, "Load", 100, "Outage")
    )
except DuplicateAssociationError as e:
    print(e)   # attribute 100 is already attached to component 42

# Removal returns a count. Matching nothing returns 0 rather than raising, so assert on
# the count yourself if you expected a hit.
assert store.remove_supplemental_attribute_associations(component_id=43) == 2

Parent/child edges work the same way, except that identity is the ordered pair — the reverse of an edge is a different edge — and both endpoints are always components:

store.add_parent_child_association(ParentChildAssociation(42, "Generator", 7, "Bus"))
store.add_parent_child_associations([ParentChildAssociation(43, "Generator", 7, "Bus")])

assert store.list_children(parent_id=42) == [7]
assert store.list_parents(child_id=7) == [42, 43]
assert store.count_parent_child_associations() == 2

# Renumbering a component rewrites both ends of every edge.
assert store.replace_parent_child_component_id(42, 99) == 1
assert store.list_parents(child_id=7) == [43, 99]

Neither table is reachable over gRPC or the infrastore CLI.

Persist to Disk

store.flush()   # sync buffered writes; afterwards system.h5 + system.h5.sqlite can be copied

Keep the two files together — the .h5 and .h5.sqlite pair is a single logical store.

To change a store you did not build in this process, open a copy: Store.open defaults to read-write, and HDF5 has no journal, so an interrupted in-place write is unrecoverable.

store = Store.open_copy(src, scratch / "time_series.h5")   # src is never opened for writing
...
store.persist_to(src)                                       # one atomic rename replaces it

Store.open(path, read_only=True) is the right call when nothing will be written.

Where the Catalog Lives

By default the catalog is system.h5.sqlite, and every commit is durable. Passing catalog="memory" keeps it in RAM instead, so it reaches disk only when you call persist_to():

# Build in a scratch directory; nothing is durable until the explicit save.
store = Store.create(scratch / "time_series.h5", catalog="memory")
store.add_time_series(...)
store.persist_to(destination)     # writes both halves as a matched pair
store.persist_catalog()           # or: land only the .sqlite half beside the arrays already at path

Arrays still stream to the HDF5 file, so this does not require the data to fit in memory. It suits building a store beside volatile in-process state — a crash loses that state anyway, so journaling the scratch catalog buys nothing. Read store.catalog to see which mode a store is in.

Store.open(path, catalog="memory") loads an existing catalog into RAM the same way. Note that the HDF5 half is still opened in place, so mutations land in the original file; open a copy if you mean to leave the source untouched until an explicit save.

persist_to() stages both halves and renames them into place, and stamps the pair so that a save interrupted between the two renames is caught on the next open rather than read as a valid store. It does replace the destination, though, so a failed save may have destroyed what was there — recover by calling persist_to() again on the still-live store rather than assuming the target survived.

Error Handling

The store's own exceptions inherit from TimeSeriesError, so you can catch broadly or narrowly:

from infrastore import NotFoundError, DuplicateTimeSeriesError, TimeSeriesError

try:
    store.add_time_series(...)
except DuplicateTimeSeriesError:
    ...                       # key already exists
except TimeSeriesError as e:
    ...                       # anything else from the store

Argument validation stays inside the hierarchy: a malformed ISO 8601 duration string, a naive datetime, a sub-millisecond stored timestamp, and an unsupported NumPy dtype all raise InvalidParameterError. The one exception is a period argument that is neither a timedelta nor a str, which raises a plain TypeError that except TimeSeriesError will not catch.

One gotcha: because Python's bool is a subclass of int, the binding deliberately checks bool first, so True/False feature values are stored as booleans (not as 1/0 integers).

A Complete Round-Trip

from datetime import datetime, timedelta, timezone
import numpy as np
from infrastore import Store, SingleTimeSeries, OwnerCategory

store = Store.create(in_memory=True)
ts = SingleTimeSeries(
    datetime(2024, 1, 1, tzinfo=timezone.utc),
    timedelta(hours=1),
    np.arange(24, dtype=np.float64) + 100,
    "load",
    units="MW",
)
series_id = store.add_time_series(
    owner_id=42, owner_type="Generator",
    owner_category=OwnerCategory.Component,
    time_series=ts,
    features={"model_year": 2030},
)
got = store.read_by_id(series_id)
assert got.name == "load"
assert np.array_equal(np.asarray(got.data), np.asarray(ts.data))

Diagnostics and tracing

The store emits structured tracing spans for every significant operation. To see them, initialize a subscriber before your first store call.

Via environment variable — set RUST_LOG before starting Python. The module auto-initializes a subscriber on import when this variable is set:

RUST_LOG=debug python myscript.py
# or, to limit output to the store core only:
RUST_LOG=infrastore_core=debug python myscript.py

Programmatically — call init_tracing with a filter directive string:

from infrastore import init_tracing

init_tracing("infrastore_core=debug")

store = Store.create(in_memory=True)
store.add_time_series(...)   # spans appear on stderr

init_tracing is a no-op if a subscriber is already registered (including the automatic one from RUST_LOG). The filter syntax is the same as RUST_LOG: comma-separated target=level pairs, or a bare level such as "debug" to match everything. Useful targets:

Julia Developer Guide

This guide covers building on InfraStore.jl, the Julia package that wraps the C ABI, from installing it to the calls a consumer package makes. For exact signatures see the Julia API reference.

For complete programs rather than isolated snippets, use the repository's runnable Julia examples. They cover static, non-sequential, persistent, deterministic, probabilistic, and scenario data; fixed tuples; every function-valued element type; feature-based selection; and conversion to DataFrames.

Install

Julia 1.11 or newer. InfraStore.jl is registered in General, and the native library comes with it:

using Pkg
Pkg.add("InfraStore")

Pkg downloads the libinfrastore_ffi artifact for your platform from the matching GitHub Release (Artifacts.toml in the package pins its URL and hash). The library is linked statically against HDF5 and zlib, so there is no HDF5_jll and no system HDF5 involved, and the HDF5 version behind the on-disk format is the one infrastore pinned. Artifacts exist for Linux x86_64 and aarch64 (glibc), macOS x86_64 and Apple Silicon, and Windows x86_64; on any other platform, build the library yourself and use the override below.

That is the whole recipe for a consumer package.

From a checkout

InfraStore.jl resolves the cdylib at first use, in this order:

1. The INFRASTORE_LIB environment variable — the development override.
2. The libinfrastore_ffi artifact Pkg downloaded at install time.

So developing against a working tree means building the library and exporting the variable. This needs the build tools (cmake, a C compiler, protobuf), but no system HDF5:

cargo build -p infrastore-ffi --release
export INFRASTORE_LIB=$PWD/target/release/libinfrastore_ffi.dylib  # .so on Linux

julia --project=julia/InfraStore.jl -e 'using Pkg; Pkg.instantiate()'
julia --project=julia/InfraStore.jl julia/InfraStore.jl/test/runtests.jl

using InfraStore always works; the resolution happens on the first call that reaches the native library, and the path is cached for the rest of the session — so set the variable before that first call, in the same shell that launched Julia. To use the checkout from your own project, Pkg.develop(path="/path/to/infrastore/julia/InfraStore.jl") with the variable set.

If it does not load

• Could not locate libinfrastore_ffi at <path>. — The resolved path is not a file. If <path> is under .julia/artifacts, the artifact download did not complete: Pkg.instantiate() (or Pkg.add("InfraStore") again) fetches it. If it is your own path, export INFRASTORE_LIB before the first store call.
• could not load library — Check the path exists and has the right extension for your OS (.dylib on macOS, .so on Linux, .dll on Windows), and that you built with --release if your variable points at target/release.
• InvalidParameterError on add — owner_id must be an integer (e.g. 42, an Int64), and features values must be JSON scalars.

Load the Package

using Dates, InfraStore

Exported names include Store, SingleTimeSeries, NonSequentialTimeSeries, PersistentTimeSeries, the forecast structs (Deterministic, Probabilistic, Scenarios), OwnerCategory (Component, SupplementalAttribute), the add_time_series! / read_by_id / list_metadata family, and transform_single_time_series!. The store type is named Store.

Open or Create a Store

# In-memory.
store = Store(in_memory=true)

# On disk: writes system.h5 and system.h5.sqlite.
store = Store(in_memory=false, path="system.h5")

# Reopen read-only.
store = open_store("system.h5"; read_only=true)

A Store registers a finalizer, so nothing leaks if you never close one — but the GC decides when, and on disk that means an open file handle and a held SQLite write lock for an unbounded time. Prefer the do-block forms, which release the store on the way out including on a throw:

Store(in_memory=true) do store
    add_time_series!(store, 42, "Generator", Component, ts)
end

open_store("system.h5"; read_only=true) do store
    only(list_metadata(store; owner_id=42, owner_category=Component, name="load"))
end

open_copy has one too. close!(store) is the explicit form when a do-block does not fit — a long-lived store held by a consumer package, say — and is idempotent, so closing a store the finalizer later reaps is fine.

Add a Series

# `name` ("load") is a required field on the struct.
ts = SingleTimeSeries(DateTime(2024, 1, 1), Hour(1), collect(100.0:123.0), "load")

A bare DateTime carries no zone, so this package reads one as a wall clock — the store holds its fields unchanged and records the spelling as ZonelessReference(). If your timestamps are genuinely zoned, using TimeZones and pass a ZonedDateTime instead — it names an instant on its own, and is accepted anywhere a DateTime is:

using TimeZones
ts = SingleTimeSeries(
    ZonedDateTime(DateTime(2024, 1, 1), tz"America/Denver"),   # = 2024-01-01T07:00Z
    Hour(1), collect(100.0:123.0), "load",
)

Reads return a bare DateTime holding the instant either way, with the recorded spelling beside it (zoned_timestamp fuses the two back together); see Time and resolution conversions.

id = add_time_series!(
    store,
    42,
    "Generator",
    Component,
    ts;                                   # name and descriptors come from ts
    features = Dict("model_year" => 2030),
)
# `id` is the catalog row's id: how every read and removal addresses
# the series, and one integer to keep in your own model.

Notes:

• owner_id is an integer (Int64) — the component identifier, e.g. 42.
• resolution is a Period such as Hour(1) or Minute(5).
• features is a Dict serialized to JSON, so values must be JSON scalars (Int, Float64, Bool, String). String features are supported and round-trip unchanged.
• Adding a duplicate identity throws DuplicateTimeSeriesError.
• For a sparse step function — a monthly fuel price, say — use PersistentTimeSeries, worked through in Step Functions.

add_time_series! returns the catalog row's id as an Int64 (see Association ids). Every read, removal and copy takes that id; list_metadata is how you recover one for a series you did not just write.

Two more rules worth knowing up front. Stored instants and periods are millisecond-precision: a Microsecond(1) resolution, a Millisecond(0) one, or a negative period is refused with InvalidParameterError (query bounds are unconstrained). And a Store is not thread-safe: confine it, and any reader built from it, to one task, or guard every call with your own lock — concurrent calls are undefined behavior, not just a race on results.

Descriptors

Beyond units, an association can carry quantity_kind (what the values measure — "ActivePower"; the one record of what per-unit values mean), unit_system (NaturalUnits or ComponentBase; nothing means unspecified, not natural units — the store rescales nothing), component_field (the field on the owning component these values vary — "max_active_power"; also a filter), and application_data (an opaque string returned verbatim — the package-owned slot). They can be set on the struct, where they become the add_time_series! defaults, or passed as keywords:

ts = SingleTimeSeries(DateTime(2024, 1, 1), Hour(1), collect(100.0:123.0), "load";
                      units = "MW", quantity_kind = "ActivePower",
                      unit_system = NaturalUnits, component_field = "max_active_power",
                      application_data = "{\"source\": \"weather_year_2012\"}")
id = add_time_series!(store, 42, "Generator", Component, ts)   # keeps all five

A series also records a time_reference — how its timestamps were spelled — inferred from the timestamp it was built with. A bare DateTime is a wall clock (ZonelessReference()); a ZonedDateTime keeps its zone, so a Denver series renders correctly on both sides of every DST transition. Reads still return a DateTime holding the instant, with the reference beside it; using TimeZones adds zoned_timestamp to fuse them back together losslessly.

None of them is part of a series' identity or of either content hash, so two adds that differ only in a descriptor are a duplicate. See Optional Descriptors and Time references.

Add many series at once

AddBatch accepts the same add_time_series! calls as a Store but only accumulates them; add_time_series_bulk! commits the whole batch in one catalog transaction and takes the block-sized HDF5 write path, so same-shaped series land in the same packed dataset.

batch = AddBatch()
for (id, ts) in series
    add_time_series!(batch, id, "Generator", Component, ts)
end
ids = add_time_series_bulk!(store, batch)   # Vector{Int64}, in input order; all-or-nothing

This is an order of magnitude faster than a bare loop of single adds, which pays one catalog transaction and one HDF5 flush per series. It is not faster than that same loop inside a transaction, which buffers and writes the identical datasets. What separates the two is one thing: the batch is written as a single block whatever its size, because you are already holding it, where the transaction holds its buffered adds to write_buffer_bytes and spills past it. Reach for the batch when the whole cohort is in hand; reach for the loop when you would rather add each series as you build it than hold them all first, and raise the budget if you want the single dataset back.

Transactions

Several operations that must take effect together — replacing a series is an add plus a remove — go inside a transaction. Removals are reversible only there; outside one the array bytes are reclaimed immediately.

transaction(store) do
    new_id = add_time_series!(store, 42, "Generator", Component, updated)
    remove_by_ids!(store, [old_id])
end   # committed if the block returns, rolled back if it throws

Blocks nest (each level is a savepoint), and the store holds the SQLite write lock until the outermost one ends. begin_transaction! / commit_transaction! / rollback_transaction! are the explicit form.

A transaction is also where a run of single adds belongs. Nothing it writes is durable until the outermost commit, so packed adds inside one are buffered per shape group and written as one block at the commit — the datasets add_time_series_bulk! of the same series would produce, without having to hold the batch yourself. Two things qualify it. The buffer spills a block early — an extra dataset, nothing else, the same split a batch that wide gets — once a group reaches the narrower of two widths, or once the unwritten arrays across every group cross 128 MiB:

• the columns one chunk row holds: 131,072 for a scalar Float64;
• 128 MiB divided by one column's bytes, which for anything but a short series is the ceiling that actually binds — a 30,500-step Float64 series is 244 KB a column, so its group spills at 550.

And a span holding a single array fills a shared-pool slot rather than claiming a dataset one column wide.

How wide a dataset a span writes

The 128 MiB is a default, not a law. set_write_buffer_bytes! moves it, and with it how wide a dataset a run of single adds can produce:

set_write_buffer_bytes!(store, 1 << 30)   # 1 GiB
transaction(store) do
    for (id, ts) in series
        add_time_series!(store, id, "Generator", Component, ts)
    end
end
# one dataset, however many series that was

Raised far enough, the loop writes exactly what add_time_series_bulk! of the same series writes — and the memory it costs is the memory the batch was holding anyway. Measured on 400 hourly year-long Float64 series, the two are already the same file in comparable time (~0.09 s either way, and ~0.12–0.14 s for a NonSequentialTimeSeries cohort on one axis); at 2,000 they part only over that extra dataset, ~0.30 s as one batch against ~0.38 s as a loop, and raising the budget closes it.

write_buffer_bytes(store) reads the figure back. It belongs to the handle, not to the artifact — nothing is persisted, and a store reopened elsewhere is back to 128 MiB. It is a budget for the writing process, so a machine that cannot afford another machine's choice does not inherit it. Lowering it mid-transaction writes out whatever the buffer already holds beyond the new figure; zero throws, since a pool's width floors at one column and a zero budget would mean a dataset per array rather than no buffering at all. The chunk-row ceiling above it does not move.

Values that are not numbers

A timestep's value can be a function rather than a number — a cost curve re-offered every hour, a pair of coefficients, a fixed-width tuple. Hand the constructor those values and it does the rest: it packs them into the array the store holds and names the element_type they imply.

curves = [
    PiecewiseLinear([(x = 30.0, y = 1155.0), (x = 100.0, y = 4120.0)]),
    PiecewiseLinear([(x = 30.0, y = 1353.0), (x = 65.0, y = 2730.0), (x = 100.0, y = 4223.0)]),
]

ts = SingleTimeSeries(t0, Hour(1), curves, "variable_cost")
ts.element_type            # "piecewise_linear" -- derived, not declared

id = add_time_series!(store, 42, "Generator", Component, ts)
read_by_id(store, id).data == curves      # true

There is no separate constructor for this and nothing to declare. A Vector{PiecewiseLinear} says what it is, so element_type= is only for the numeric case, where the numbers alone cannot say what they mean — and one that contradicts the values is an error rather than an override. The struct goes on holding the values you gave it; encoding happens at the ABI boundary, which is why a read hands back the same thing a write was given.

A case covered end to end in the runnable single_custom_elements.jl example. InfraStore.jl ships four value types plus NTuple{N,Float64}:

Every series type takes them, including the irregular ones and the forecasts. A forecast's values keep their window shape rather than arriving flat, since a Julia array carries its own:

# [H = 2, count = 2]: a curve for every (horizon step, window) pair.
det = Deterministic(t0, Hour(1), Hour(2), Hour(1), 2, reshape(curves4, 2, 2), "offer")

Two things follow from Julia arrays carrying their element type that are worth knowing. An empty series still names itself — NTuple{3,Float64}[] is a tuple(3,f64) series with no rows, which a language whose empty list is untyped cannot express. And a metadata row's time_series_type is the full parameterized type, SingleTimeSeries{PiecewiseLinear, 1}, so it describes the values rather than their packing.

raw = true on a read hands back the packing instead, and the per-timestamp readers are deliberately never decoded — they are the simulation path, and their numbers are physical. encode_element_values / decode_element_values are the same two directions as free functions, for an array with no series around it, and the door a consumer extends to encode and decode its own domain types with no conversion step. See Element values for both.

Read a Series

got = read_by_id(store, id)
@assert got.data == ts.data
println(got.initial_timestamp, " ", got.resolution)   # resolution comes back as Millisecond

read_by_id also takes a window — start_time plus a len of timesteps or a count of windows — which is checked: an over-long request throws rather than quietly returning less.

To read many whole series at once — e.g. loading everything for a plot — read_by_ids takes a vector of ids and returns one struct per id in the same order (each of its stored type, so the result is a Vector{Any}), reading each packed dataset's column span once instead of re-reading every chunk per series. Its time_range keyword is the other kind of slice, which clips:

series = read_by_ids(store, ids)
window = read_by_ids(store, ids; time_range = (t0, t1))   # the same clip on every series

Attribute-Based Lookups

Beyond key handles, InfraStore.jl can resolve a series directly from its attributes — convenient when a caller keeps its own identifiers (as an InfrastructureSystems.jl-side store does):

meta = get_metadata_by_id(
    store,
    42,
    Component,        # owner_category; the owner is the (owner_id, owner_category) pair
    "load";
    resolution = Hour(1),
    features = Dict("model_year" => 2030),
)
# meta :: TimeSeriesMetadata — the whole record: owner_id/owner_type/owner_category,
#          name, time_series_type, data_hash, initial_timestamp, resolution, length,
#          horizon/interval/count, percentiles, element_type, element_shape, features,
#          units, quantity_kind, unit_system, component_field, application_data

values = get_array_by_hash(store, meta.data_hash)     # Vector{Float64}; pass ::Type{T} for other dtypes

# Identify, then act. `list_metadata` answers which series exist and hands back
# the id; every read and removal takes that id.
row = only(list_metadata(store, owner_id = 42, name = "load",
                         resolution = Hour(1),
                         exact_features = Dict("model_year" => 2030)))
got = read_by_id(store, row.id)
remove_by_ids!(store, [row.id])

# `has_time_series` stays attribute-addressed: it is an index probe that reads no
# row, so routing it through a resolution would cost more than it answers.
present = has_time_series(store, 42, Component, "load";
                          resolution = Hour(1), features = Dict("model_year" => 2030))

exact_features matches the feature map exactly — the series above was added with model_year = 2030, so features (a subset match) would also select a sibling carrying more. The plain features keyword is the right one for "every series tagged scenario = high"; a package that resolves partial user queries lists, then decides what more than one match means.

Forecasts

InfraStore.jl exposes Deterministic, Probabilistic, and Scenarios structs that wrap a native AbstractArray in the type's logical shape (the wrapper derives the dtype and dims and serializes the buffer row-major). Construct one and add it through the generic add_time_series!:

data = zeros(Float64, 24, 7)   # (horizon_count, count)
fc = Deterministic(
    DateTime(2024, 1, 1), Hour(1), Hour(24), Hour(24), 7, data, "load_fc"; units = "MW"
)
id = add_time_series!(
    store,
    42,
    "Generator",
    Component,
    fc,                         # name and units come from fc
)

got = read_by_id(store, id)   # the id add_time_series! returned
values = got.data             # Float64 matrix, shape (24, 7)

Probabilistic(initial_timestamp, resolution, horizon, interval, count, percentiles, data, name) carries the percentile vector, and Scenarios(initial_timestamp, resolution, horizon, interval, count, data, name) takes scenario_count from data's leading axis. Every forecast constructor also accepts a application_data= keyword. A read names only an id, so the row's own type decides what comes back — there is no requested type to disagree with it.

If two forecasts of one owner/name/type differ only by interval (say day-ahead and intra-day), pass interval= to the listing to pin the one you want; without it it returns both:

row = only(list_metadata(store; owner_id = 42, name = "load_fc",
                         resolution = Hour(1), interval = Hour(6)))

A DeterministicSingleTimeSeries is not added directly — derive one from the stored SingleTimeSeries with transform_single_time_series!, which returns a TransformOutcome whose transformed field is the count. It optionally restricts the transform to one owner_category and/or one resolution, and dry_run = true runs every check without writing:

n = transform_single_time_series!(store, Hour(24), Hour(24)).transformed
outcome = transform_single_time_series!(store, Hour(24), Hour(24);
                                        owner_category = Component, resolution = Hour(1),
                                        normalize_single_window = true,
                                        require_uniform_forecast_grid = true)

The two policy flags reproduce InfrastructureSystems.jl's rules (it passes both as true); see the reference for what each enforces.

Filtering for Deterministic also matches a transformed DeterministicSingleTimeSeries, so you find a forecast the same way whether it was added densely or derived — and either reads back as a Deterministic, since a DST has no materialized struct. Each row still reports the concrete form it is, so transform_single_time_series! needs no separate enumeration path:

for row in list_metadata(store; owner_id = 42, time_series_type = Deterministic)
    row.time_series_type <: DeterministicSingleTimeSeries   # derived, or densely stored?
    series = read_by_id(store, row.id)   # a Deterministic either way
end

transform_single_time_series! also reports the ids it wrote on its TransformOutcome.written, so a caller can reference a view it just derived without listing the store to find it again.

has_time_series takes the time series type as its first argument to address anything other than a SingleTimeSeries (and takes the same resolution / interval / features keywords). copy_time_series! takes the source id and re-points that series at another owner without duplicating data — it writes one association row against the same content-addressed array, preserving the stored type (a DST stays a DST) — and returns the copy's own id:

has_time_series(Scenarios, store, 42, Component, "wind"; resolution = Hour(1))
src = only(list_metadata(store; owner_id = 42, name = "load")).id
copy_time_series!(store, src, 43, "Generator")

Every time_series_type filter keyword takes the Julia type as well:

list_metadata(store; time_series_type = Deterministic)
get_resolutions(store; time_series_type = SingleTimeSeries)

A metadata row's time_series_type is the full type — SingleTimeSeries{Float64,1}, Deterministic{Float32,3} — so a row names what a read of it hands back rather than only which of the six kinds it is, and a consumer holding InfrastructureSystems.jl-style parameterized types gets them back intact:

md = get_metadata_by_id(store, id)
md.time_series_type == typeof(read_by_id(store, id))   # every stored type but DST
md.time_series_type <: SingleTimeSeries                # ask for the kind with <:, not ==

A derived DeterministicSingleTimeSeries is the exception: its row keeps the DST tag while a read of it hands back a Deterministic with the same {T,N}. Dispatch on the read's type when the two have to agree, and on the row's when you mean "was this derived?".

That type passes straight back into any filter, has_time_series, or reader. The parameters are ignored there — a series is addressed by identity, which carries no element type — so they never narrow a match; they are accepted so a row you just read round-trips without being taken apart first.

The low-level get_metadata_by_id + get_array_by_hash path is still available for raw access. See the Julia API reference.

Per-Timestamp Reads (Simulation Loop)

read_by_id hands back a whole series or forecast. Simulations instead walk the timeline and, at each timestamp, want the value of every series at that instant. For that, build a reader once and drive it in a loop — it pins one resolution and reuses its output buffers, so the loop allocates almost nothing. StaticReader serves SingleTimeSeries; ForecastReader serves forecasts. (Full signatures: Julia API reference.)

Static series

reader = build_static_reader(store; resolution = Hour(1))
grid = static_grid(reader)                 # StaticGrid: initial_timestamp, resolution, length
for k in 0:(grid.length - 1)
    static_read!(reader, grid.initial_timestamp + grid.resolution * k)
    for (gi, g) in enumerate(static_groups(reader))
        vals = static_values(reader, gi)   # (num_columns, element_dims...); column j ↔ g.ids[j]
    end
end

Series are grouped by (dtype, element_shape); each group's static_values is one dense array whose columns line up with the group's ids. All matched series must share one grid (initial_timestamp + length), validated at build.

Most real systems do not meet that — a year of load beside a week of an outage schedule — and the build then errors, naming the series that diverges. Give the reader a span instead of letting it inherit one:

reader = build_static_reader(store; resolution = Hour(1),
                             window_start = DateTime(2024, 1, 1, 7),
                             window_length = 8760)  # optional: without it, as far as all reach

Each column then reads at an offset of its own, so ragged series sweep together. The span is checked rather than clamped: a series that does not cover it errors, naming that series, and the anchor must land on each series' own step boundaries. See reader windows.

When the odd series out should not take part at all — a stray day of data beside a year of it is a different component, not a shorter view of the same sweep — filter to one grid instead:

reader = build_static_reader(store; resolution = Hour(1),
                             initial_timestamp = DateTime(2024, 1, 1, 7), length = 8784)

The window sweeps a span across whatever matched; the filter matches only the series already on that grid, and reaches list_metadata and remove_by_filter! the same way. See selecting one grid.

Forecasts

reader = build_forecast_reader(store, Deterministic; resolution = Hour(1))
tl = forecast_timeline(reader)             # ForecastTimeline: initial_timestamp, resolution, interval, count
for k in 0:(tl.count - 1)
    forecast_read!(reader, tl.initial_timestamp + tl.interval * k)
    for (i, e) in enumerate(forecast_entries(reader))
        window = forecast_values(reader, i)   # shape e.window_shape, for e.key
    end
end

A Deterministic reader is abstract — it also includes any DeterministicSingleTimeSeries (read into identical windows).

Shared forecasts are read once

Forecasts that share a backing array (deduplicated identical data, or several DeterministicSingleTimeSeries over one SingleTimeSeries) collapse to a single window slot. forecast_read! reads each slot from the .h5 file once per timestamp, so a forecast shared by 10 components costs one read, not ten. forecast_num_slots(reader) is the physical read count, and each ForecastEntry.slot says which slot an entry uses — group by slot to materialize each unique window only once:

forecast_read!(reader, t)
windows = Dict{Int, Any}()
for (i, e) in enumerate(forecast_entries(reader))
    w = get!(() -> forecast_values(reader, i), windows, e.slot)
    # apply w to e.key's owner
end

Step Functions (PersistentTimeSeries)

A PersistentTimeSeries is a sparse step function: a strictly increasing vector of breakpoints plus one value each, where the value at an arbitrary instant is the one belonging to the greatest breakpoint at or before it. The motivating data is a monthly fuel or gas price curve — a dozen breakpoints a simulation reads at timestamps that almost never land on one. The same curve stored as a NonSequentialTimeSeries would throw at nearly every step, because an irregular series has no value between its timestamps; that difference in read semantics is the whole reason this is a separate type. See Time series types for the model.

Build and add

The struct takes the same arguments as NonSequentialTimeSeries — and, like every series struct here, its descriptors as keywords. A bare DateTime is read as a wall clock and a ZonedDateTime as the instant it names, exactly as for the other types:

breakpoints = [DateTime(2024, m, 1) for m in (1, 4, 7, 10)]
prices = PersistentTimeSeries(
    breakpoints,
    [3.5, 4.25, 5.0, 4.75],
    "gas_price";
    units = "USD/MMBtu",
    component_field = "fuel_cost",
    # Whether a curve is expanded to a full series or collapsed to one scalar is
    # your application's policy, and rides here where the store never reads it.
    application_data = """{"as_time_series":false,"force_scalar_mode":"midpoint"}""",
)
id = add_time_series!(store, 7, "ThermalStandard", Component, prices)

got = read_by_id(store, id)              # a PersistentTimeSeries; `timestamps` are the breakpoints
get_metadata_by_id(store, id).time_series_type <: PersistentTimeSeries   # `<:`, not `==`

A metadata row's time_series_type is the full parameterized type (PersistentTimeSeries{Float64, 1}), so ask which kind a row is with <:. Every type-taking call — a time_series_type filter, has_time_series, either reader — accepts the bare spelling and the parameterized one alike.

Read a window

read_by_ids' time_range slices on the step function's own terms: the result begins at the breakpoint in force at the start, so it always defines a value at the start of the window you asked for.

sliced = only(
    read_by_ids(store, [id]; time_range = (DateTime(2024, 4, 10), DateTime(2024, 9, 1))),
)
sliced.timestamps   # [2024-04-01T00:00:00, 2024-07-01T00:00:00] — April, not the first one inside
sliced.data         # [4.25, 5.0]

Past the last breakpoint the last value holds forever, so a window opening after the end comes back with that one row. Before the first breakpoint a step function is undefined: a non-empty window starting there throws InvalidParameterError rather than clamping. (A zero-width range selects nothing, here as for every type.) read_by_id's start_time + len window is checked rather than sliced, so it must name one of the breakpoints; reach for the time_range form when the instant is arbitrary.

Sweep step functions in the simulation loop

A StaticReader filtered to the type is the per-timestamp path, and it is the one place the one-timeline-per-reader rule bends: a step function has a value at every instant from its own first breakpoint on, so the columns need not share a breakpoint vector. Per-fuel curves whose breakpoints do not line up still build one reader.

reader = build_static_reader(
    store;
    time_series_type = PersistentTimeSeries,   # no resolution — passing one throws
    component_field = "fuel_cost",
)
grid = static_grid(reader)                 # grid.resolution === nothing: no constant step
for at in static_timestamps(reader)        # the sorted union of every column's breakpoints
    static_read!(reader, at)
    for (gi, g) in enumerate(static_groups(reader))
        vals = static_values(reader, gi)   # the value in force at `at`; column j ↔ g.ids[j]
    end
end

static_timestamps is the union of the columns' breakpoints, so a position on it is not a storage row for any one column — each column independently reports the value in force there. There is still no presence mask: reading at an instant before some column's first breakpoint throws InvalidParameterError naming that column's association id. Either filter the reader down to columns that start early enough, or begin the sweep at the latest first breakpoint among them.

The sweep need not follow that union axis at all. static_read! accepts any instant every column defines a value at, so driving this reader at your SingleTimeSeries grid's timestamps — the simulation's own clock — works and is usually what an application wants:

for at in static_timestamps(load_reader)   # the hourly grid the simulation runs on
    static_read!(load_reader, at)
    static_read!(reader, at)               # each fuel price, held forward to this hour
end

These rows do not travel in an OpenAPI document

PersistentTimeSeries is an infrastore-local extension, and the vendored wire contract is a oneOf over six canonical Sienna types with no schema for a seventh. So export_time_series_associations_openapi omits persistent rows — a mixed store still exports its six-type rows — a filter naming the type throws InvalidParameterError rather than answering with an empty array, and an import refuses a document that carries one. The series themselves are unaffected: they live in the artifact, which holds them in full. Ask the catalog what a document leaves behind:

left_behind = list_metadata(store; time_series_type = PersistentTimeSeries)

Store-Wide Operations

counts = get_counts(store)        # TimeSeriesCounts: components_with_time_series, static_time_series, forecasts
nerr   = verify_integrity(store)  # 0 == every referenced array and time axis matches its hash
report = compact!(store)          # CompactionReport; rewrites the .h5 from the live set, so a
                                  # delete actually shrinks the file. Nothing else may have the
                                  # store open while it runs.

Associations

Two catalog tables record relationships between entities the store does not otherwise model, wholly independently of time series: which supplemental attributes are attached to which components, and directed parent/child edges between components. Removing a time series never touches either, and vice versa — see Associations Between Entities.

Filter keywords are all optional and ANDed; passing none matches everything.

add_supplemental_attribute_association!(
    store, SupplementalAttributeAssociation(42, "Generator", 100, "GeographicInfo"))

# Bulk add is one all-or-nothing transaction.
add_supplemental_attribute_associations!(store, [
    SupplementalAttributeAssociation(43, "Generator", 100, "GeographicInfo"),
    SupplementalAttributeAssociation(43, "Generator", 101, "Outage"),
])

# Queries run in both directions, returning distinct ids in ascending order.
list_supplemental_attribute_ids(store; component_id=43)      # [100, 101]
list_components_with_attributes(store; attribute_id=100)     # [42, 43]
has_supplemental_attribute_association(store; component_id=42, attribute_id=100)  # true

# `*_types` filters take CONCRETE type names, so expand an abstract type yourself —
# `get_all_subtype_names` in InfrastructureSystems.jl is the usual source. An empty
# vector is a deliberate "none of these" and matches nothing.
list_supplemental_attribute_ids(store; component_id=43, attribute_types=["Outage"])  # [101]

count_supplemental_attributes(store)         # 2, distinct attributes
count_components_with_attributes(store)      # 2, distinct components
supplemental_attribute_counts_by_type(store)
# [SupplementalAttributeTypeCount("GeographicInfo", 2), SupplementalAttributeTypeCount("Outage", 1)]
supplemental_attribute_summary(store)
# [SupplementalAttributeSummaryRow("Generator", "GeographicInfo", 2), ...]

Identity is the (component_id, attribute_id) pair. The type names ride along for filtering and are not part of it, so re-attaching the same pair under different type names is still a duplicate:

try
    add_supplemental_attribute_association!(
        store, SupplementalAttributeAssociation(42, "Load", 100, "Outage"))
catch e
    e isa InfraStore.DuplicateAssociationError || rethrow()
    @info e.msg   # attribute 100 is already attached to component 42
end

# Removal returns a count. Matching nothing returns 0 rather than throwing, so assert on
# the count yourself if you expected a hit.
remove_supplemental_attribute_associations!(store; component_id=43)   # 2

Parent/child edges work the same way, except that identity is the ordered pair — the reverse of an edge is a different edge — and both endpoints are always components:

add_parent_child_association!(store, ParentChildAssociation(42, "Generator", 7, "Bus"))
add_parent_child_associations!(store, [ParentChildAssociation(43, "Generator", 7, "Bus")])

list_children(store; parent_id=42)      # [7]
list_parents(store; child_id=7)         # [42, 43]
count_parent_child_associations(store)  # 2

# Renumbering a component rewrites both ends of every edge.
replace_parent_child_component_id!(store, 42, 99)   # 1
list_parents(store; child_id=7)                     # [43, 99]

Neither table is reachable over gRPC or the infrastore CLI.

Persist to Disk

flush!(store)   # sync HDF5 + SQLite; afterwards system.h5 + system.h5.sqlite can be copied

Keep the .h5 and .h5.sqlite files together.

To change a store you did not build in this process, open a copy: open_store defaults to read-write, and HDF5 has no journal, so an interrupted in-place write is unrecoverable.

store = open_copy(src, joinpath(scratch, "time_series.h5"))   # src is never opened for writing
...
persist!(store, src)                                          # one atomic rename replaces it

open_store(path; read_only=true) is the right call when nothing will be written.

Where the Catalog Lives

By default the catalog is system.h5.sqlite, and every commit is durable. Passing catalog=:memory keeps it in RAM instead, so it reaches disk only via persist!:

# Build in a scratch directory; nothing is durable until the explicit save.
store = Store(; in_memory=false, path=joinpath(scratch, "time_series.h5"), catalog=:memory)
add_time_series!(store, 42, "Generator", Component, ts)
persist!(store, destination)     # writes both halves as a matched pair
persist_catalog!(store)          # or: land only the .sqlite half beside the arrays already at path

Arrays still stream to the HDF5 file, so this does not require the data to fit in memory. It suits building a store beside volatile in-process state — a crash loses that state anyway, so journaling the scratch catalog buys nothing. catalog_mode(store) reports which mode a store is in.

open_store(path; catalog=:memory) loads an existing catalog into RAM the same way. Note that the HDF5 half is still opened in place, so mutations land in the original file; open a copy if you mean to leave the source untouched until an explicit save.

persist! stages both halves and renames them into place, and stamps the pair so that a save interrupted between the two renames is caught on the next open rather than read as a valid store. It does replace the destination, though, so a failed save may have destroyed what was there — recover by calling persist! again on the still-live store rather than assuming the target survived.

Error Handling

Errors subtype InfraStore.TimeSeriesException. The exception types are not exported, so reference them module-qualified. Catch broadly or narrowly:

try
    add_time_series!(store, 42, "Generator", Component, ts)
catch e
    if e isa InfraStore.DuplicateTimeSeriesError
        @warn "already present"
    else
        rethrow()
    end
end

The available types are InfraStore.NotFoundError, InfraStore.DuplicateTimeSeriesError, InfraStore.DuplicateAssociationError, InfraStore.InvalidParameterError, InfraStore.IntegrityError, InfraStore.ReadOnlyStoreError, InfraStore.IOError, InfraStore.StoreExistsError (creating over an existing artifact), InfraStore.MismatchedArtifactError (the .h5 and .sqlite halves came from two saves), InfraStore.IncompatibleFormatError (the on-disk store was written by an incompatible data format version), and InfraStore.GenericError (which carries the raw FFI status code). See the reference for the full table.

InfrastructureSystems.jl Integration Notes

Embedding in a Parent Package is the language-neutral version of this section — the store lifecycle, id mapping, and lookup semantics a package like InfrastructureSystems.jl has to honor. The Julia-specific points:

• Owners are integer component identifiers (Int64), matching InfrastructureSystems.jl component/attribute IDs.
• OwnerCategory distinguishes Component from SupplementalAttribute and is part of the owner identity: the owner is the (owner_id, owner_category) pair, so a component and a supplemental attribute may share a numeric id and stay distinct. Owner-scoped calls take the category alongside the id.
• The attribute-based existence probes (has_time_series, has_any_time_series) plus get_metadata_by_id and get_array_by_hash let an InfrastructureSystems.jl-side store keep its own object model — holding only the catalog id — and reach the array layer directly.
• For the simulation read pattern — iterate every component's value at each timestamp, reading a forecast shared across components only once — use the readers (Per-Timestamp Reads). The ForecastEntry.slot / forecast_num_slots surface lets the wrapping store dedup its own per-component work, mirroring the store's one-read-per-shared-array behavior.

See Language Bindings for how this maps onto the FFI.

Diagnostics and tracing

The store emits structured tracing spans for every significant operation. To see them, initialize a subscriber before your first store call.

Via environment variable — set RUST_LOG before loading the package. The module's __init__ hook calls init_logging("") automatically, which reads RUST_LOG if set:

# shell
export RUST_LOG=infrastore_core=debug
julia --project=. myscript.jl

Programmatically — call init_logging with a filter directive string:

using InfraStore

init_logging("infrastore_core=debug")

store = Store(in_memory=true)
add_time_series!(store, ...)   # spans appear on stderr

init_logging is a no-op if a subscriber is already registered (including the automatic one from RUST_LOG). The filter syntax is the same as RUST_LOG: comma-separated target=level pairs, or a bare level such as "debug" to match everything. Useful targets:

CLI Developer Guide

infrastore loads time series from CSV files and inspects a store, talking directly to the on-disk .h5 + .h5.sqlite pair — no gRPC server, and unlike the server it is not read-only. This guide walks the whole workflow; for every flag, the descriptor schema, and the CSV layouts, see the CLI Reference. For a 60-second round trip, see the CLI Quick Start.

It is also the fastest way to see what a consumer package actually wrote: infrastore list and infrastore info read any artifact your package produces, and infrastore diff compares two by hash without reading arrays.

Install

Grab a prebuilt archive from the Releases page — the executables inside are statically linked against HDF5, so there is nothing else to install:

VERSION=v0.14.0    # pick a release from the Releases page
curl -fsSLO https://github.com/NatLabRockies/infrastore/releases/download/$VERSION/infrastore-aarch64-apple-darwin.tar.gz
tar xzf infrastore-aarch64-apple-darwin.tar.gz

Or install from crates.io, which builds HDF5 from source and so needs cmake and a C compiler:

cargo install infrastore-cli       # installs the `infrastore` binary

See Installation for the per-platform archive list and checksum verification.

Working in a checkout instead:

cargo build -p infrastore-cli      # debug build at target/debug/infrastore
cargo build -p infrastore-cli --release

The examples below assume infrastore is on your PATH (or use ./target/debug/infrastore).

Describe the Data

Numeric values live in a CSV; everything that does not fit a flat grid (owner, name, type, element_type, resolution, initial timestamp, units, features) lives in a descriptor JSON. Print a starting point for any type with template:

infrastore template SingleTimeSeries > load.json   # print an example descriptor to edit

Edit it to point at your data and metadata:

{
  "owner_id": 42,
  "owner_type": "Generator",
  "owner_category": "Component",
  "name": "load",
  "type": "SingleTimeSeries",
  "element_type": "f64",
  "units": "MW",
  "csv": "load.csv",
  "initial_timestamp": "2024-01-01T00:00:00Z",
  "resolution": "PT1H",
  "features": {
    "model_year": 2030
  }
}

# load.csv
value
100.0
101.5
103.0
104.2
102.8
101.0

Every data CSV needs a header row. It is not decoration: add reads it to tell a hand-written value-only file from one export wrote (see Reading back, and re-adding). A file whose first row is data is rejected rather than quietly losing that row to the header.

The descriptor rejects unknown keys, so a typo (resolutionn) is a hard error rather than a silently ignored setting.

Timestamps must name an instant. A timestamp column (or initial_timestamp) written as 2024-01-01 00:00:00 — no offset, the way most spreadsheets and databases export — is rejected, because it names no instant. Rather than rewrite the file, say what zone it was written in with the global --assume-timezone (UTC, or a fixed offset like -07:00; named zones such as America/Denver are deliberately not accepted, because a DST fold would make some rows ambiguous):

infrastore --store demo.h5 --assume-timezone UTC add --descriptor load.json

A timestamp that carries its own offset is never overridden. See Zoneless timestamps.

Add It to a Store

infrastore --store demo.h5 add --descriptor load.json --dry-run   # check first
infrastore --store demo.h5 add --descriptor load.json

The store (demo.h5 and its demo.h5.sqlite catalog) is created on first add, or explicitly with init when you want to pin a compression policy up front:

infrastore --store demo.h5 init --compression deflate:6                 # default is deflate:3
infrastore --store demo.h5 init --compression none --catalog in-memory   # see below

A descriptor may also be a JSON array of objects to add many series in one transaction. --csv overrides the descriptor's csv path, but only when the descriptor is a single object: with an array of two or more it errors (--csv cannot be used with an array descriptor).

--dry-run is worth running first on anything large. It resolves every descriptor and reads every CSV in full, then prints the resolved (owner, type, name, element type, shape) table without opening the store — which catches the "I got the shape wrong" class of mistake before a multi-GB load starts. --replace makes a re-run after fixing the data idempotent, and --descriptor - reads the JSON from stdin so a generator script can pipe descriptors straight in:

infrastore --store demo.h5 add --descriptor batch.json --replace
generate.py | infrastore --store demo.h5 add --descriptor - --quiet

For a load too large to hold in one transaction, --batch-size N commits every N series (at the cost of the load's atomicity), and --catalog in-memory skips the per-commit journaling of the SQLite catalog while the command runs — the CLI writes it out at the end of the command either way:

infrastore --store demo.h5 add --descriptor batch.json --batch-size 500 --catalog in-memory

Every command carries worked examples in its own help — infrastore add --help — and infrastore --help is the grouped index.

For a one-off, the descriptor fields are also add flags:

infrastore --store demo.h5 add --csv load.csv --owner-id 42 --owner-type Generator \
  --name load --type SingleTimeSeries --element-type f64 --units MW \
  --resolution PT1H --initial-timestamp 2024-01-01T00:00:00Z

One file, many components

The canonical power-systems CSV is one column per component, which is the opposite shape from the descriptor's one-object-one-series default:

# gen_profiles.csv
timestamp,gen_001,gen_002,gen_003
2024-01-01T00:00:00Z,101.5,88.2,44.0
2024-01-01T01:00:00Z,102.1,87.4,44.6

"layout": "wide" loads that as one scalar series per column. The store keys on an integer owner_id while the headers are component names, so the mapping is an input — a column,owner_id[,owner_type] sidecar CSV, an inline object, or "owner_id_from": "header" when the headers already are ids:

{
  "csv": "gen_profiles.csv",
  "layout": "wide",
  "type": "SingleTimeSeries",
  "name": "max_active_power",
  "owner_type": "ThermalStandard",
  "element_type": "f64",
  "units": "MW",
  "initial_timestamp": "2024-01-01T00:00:00Z",
  "resolution": "PT1H",
  "owner_map": "components.csv"
}

infrastore grid writes that same shape back out, so the two are an inverse pair — see below.

Read It Back

infrastore follows an output convention: a global -f/--format with table (default), json, jsonl, and csv. The read commands render their results in it; the write commands (add, remove, transform, …) report their outcome in it — prose under table, a one-object status document such as {"added": 3, "store": "demo.h5"} under json/jsonl — so a scripted mutation pipes into jq the same way a query does. Only template ignores it. jsonl is json line-delimited — one compact object per line, which streams into jq where a single pretty array cannot. Under -f json an error is a {"status": "error", "message": …} document on stderr.

Before you can write a selector you need to know what values exist, which is what the discovery commands are for:

infrastore --store demo.h5 names                                      # distinct series names
infrastore --store demo.h5 owner-types                                # distinct owner types
infrastore --store demo.h5 owners --type SingleTimeSeries             # distinct owner ids
infrastore --store demo.h5 exists --name load                         # exit 0 = yes, 1 = no

infrastore --store demo.h5 list                                       # what's in the store
infrastore --store demo.h5 list --name-glob 'load_*'                  # name pattern (SQLite GLOB)
infrastore --store demo.h5 list --limit 20 --wide                     # bounded, all columns
infrastore --store demo.h5 get  --owner-id 42 --name load             # pretty table
infrastore --store demo.h5 get  --name load --tail --limit 24         # the last day
infrastore --store demo.h5 get  --name load --plot                    # a terminal sparkline
infrastore --store demo.h5 -f csv  get  --owner-id 42 --name load     # timestamped CSV
infrastore --store demo.h5 -f jsonl list                              # one JSON object per line
infrastore --store demo.h5 -f json info --owner-id 42 --name load     # metadata + hash + stats
infrastore --store demo.h5 -f csv  export --dir out/                  # one file per series

To see many series side by side against one time axis — the read-direction inverse of the wide ingest above — use grid:

infrastore --store demo.h5 -f csv grid --name max_active_power --resolution PT1H

Every column in a grid shares one timeline, which is what makes the rows line up without a presence mask; that is why SingleTimeSeries needs --resolution. When every column shares one series name the headers are bare owner ids, which is exactly the wide form add reads back.

When the matched series do not share one — a year of load beside a shorter schedule — the build fails, naming the series that diverges. --window-start sweeps a span you name instead, with each column reading at an offset of its own:

infrastore --store demo.h5 -f csv grid --resolution PT1H --window-start 2024-01-01T02:00:00Z

Add --window-length N to pin the extent; without it the sweep runs as far as every matched series reaches. A series that does not cover the span is an error naming it, never a column quietly left out of the table.

Sometimes the odd series out should not be a column at all. Then select one grid instead — with --resolution, the --initial-timestamp and --length selectors name a whole grid, and every series not on it drops out:

infrastore --store demo.h5 -f csv grid --resolution PT1H \
    --initial-timestamp 2024-01-01T02:00:00Z --length 6

They are ordinary selectors, so list, remove, and the rest take them too — which is how you find the grids a store holds (list --length 24) and retire a stray cohort without naming its owners.

export is the bulk read-direction inverse of add: every series the selector matches is written to its own CSV or JSON file under --dir (or to stdout when exactly one matches), optionally sliced with --time-range; -f parquet instead writes one file pair per partition. Setting INFRASTORE_STORE in the environment stands in for --store, every destructive command except compact accepts --dry-run to preview its effect, and the global -y/--yes answers every confirmation prompt so a script does not have to know which commands ask:

export INFRASTORE_STORE=demo.h5
infrastore remove --name-glob 'scratch_*' --dry-run      # what would go
infrastore -y remove --name-glob 'scratch_*'             # no prompt
infrastore -f csv export --name load --time-range 2024-01-01T00:00:00Z..2024-01-08T00:00:00Z

info reports metadata, the array's content hash and where it lives in the HDF5 file, and stats over the values: min/max/mean/stddev, the p5–p95 percentiles, first/last, and a separate non_finite count — or true_count/false_count when the dtype is bool. The stats are the only part that reads the array — --no-stats skips it for a purely catalog-side query.

list shows every field that is part of a series' identity, features included, so two series that differ only by a feature never render as the same row. Its Hash column is the first 12 characters of the array's content hash: rows with equal hashes share one array on disk.

get/info/remove select a single series with --owner-id, --owner-category, --name, --name-glob, --component-field, --type, --resolution, and repeated --feature key=value (--feature is the only repeatable one); if more than one series matches, infrastore lists the candidates so you can narrow the query. The owner is the (owner_id, owner_category) pair, so a component and a supplemental attribute may share a numeric id — add --owner-category (Component / SupplementalAttribute) to disambiguate. Large series truncate in table output — pass --limit N, --full, or --tail to read from the end. --stride N keeps every Nth row and, unlike the display bounds, applies to a -f csv pipe too: it selects data rather than shortening a view, and a silently short pipe is a bug in whatever consumes it.

--time-range START..END on get takes two timestamps (RFC3339 or epoch-ms), not a duration:

infrastore --store demo.h5 get --owner-id 42 --name load \
  --time-range 2024-01-01T01:00:00Z..2024-01-01T03:00:00Z

Selectors accept either spelling: --type single and --type SingleTimeSeries mean the same thing, as do --owner-category component and --owner-category Component. What the CLI prints — in list/get/info output and in what template writes — is always the canonical CamelCase name, so descriptors, rendered rows, and -f json output all string-match each other. The lowercase forms are a typing shortcut on the command line, not a second vocabulary.

Chart It

infrastore --store demo.h5 get --name load --plot                        # sparkline, no file
infrastore --store demo.h5 plot --name load --out load.svg               # the profile
infrastore --store demo.h5 plot --name load --kind duration --out ldc.svg
infrastore --store demo.h5 plot --name load --kind heatmap --out heat.html
infrastore --store demo.h5 plot --name load_prob --type Probabilistic --kind fan --window 0 --out fan.svg
infrastore --store demo.h5 plot --name load --type Deterministic --kind overlay --out forecast.svg

plot writes one self-contained file — no external fonts, scripts, or images, and both light and dark themes inside it — so it opens in a browser and drops into a report. The five --kind values are line, duration (the load duration curve), heatmap (time-of-day against day, which is how you catch a timezone or DST shift), fan (percentile bands or scenario traces for one forecast window), and overlay (a Deterministic's windows over the actuals it came from).

Find the Bytes on Disk

Arrays are content-addressed, so identical values are stored once and shared. arrays shows what collapsed onto what, and where each array actually lives:

infrastore --store demo.h5 store-info    # both file paths, format version, compression
infrastore --store demo.h5 arrays        # one row per distinct array + the series sharing it
infrastore --store demo.h5 arrays --data-hash 2018057b   # narrow to one (any prefix, any case)

info resolves a single series the same way, reporting data_hash, hdf5_dataset, and hdf5_column. You need all three to open the data with an outside tool: a packed array is one column of a dataset shared with other same-shaped arrays, and a packed dataset that fills up spills into suffixed siblings, so neither the column nor the dataset name can be worked out from metadata alone.

Opening the catalog directly, use the time_series_readable view — sqlite3 prints the raw BLOB hashes as garbage bytes, and in .mode box it mangles the table borders:

sqlite3 demo.h5.sqlite 'SELECT name, data_hash FROM time_series_readable;'

Hand It to Something Else: Parquet

CSV is the interchange format add and export default to, and it has one real cost: a float goes out as decimal text and comes back as whatever parsing that text gives. Parquet does not have that problem, and every analysis tool worth the name reads it:

infrastore --store demo.h5 -f parquet export --name-glob 'load_*' --dir parquet/
infrastore --store other.h5 add --parquet parquet/

What comes out is a handful of partitions, not a file per series -- a store with thousands of series would otherwise become thousands of files. Each partition is a pair:

parquet/
  SingleTimeSeries.f64.utc.values.parquet            every distinct array once, one row per value
  SingleTimeSeries.f64.utc.series.parquet            one catalog row per series
  SingleTimeSeries.f64.America_Denver.values.parquet
  SingleTimeSeries.f64.America_Denver.series.parquet
  Deterministic.f64.utc.values.parquet
  Deterministic.f64.utc.series.parquet

One pair per (type, value type, time reference) triple, because those three cannot vary inside one table without nullable or ill-typed columns -- a forecast has an issue_time and a static series does not, and a table has one Arrow type per column. The payoff is that every column is required, which is worth more to whoever queries the files than the files it costs.

The split is the other half of it. The store is content-addressed, so a thousand components sharing one profile hold one array; writing the catalog row beside every value would write that profile a thousand times, and Parquet's compression does not find repeats across pages. So the values file holds each array once, keyed by the pair (data_hash, time_axis), and the series file carries that same pair beside each catalog row. They join on it.

add --parquet takes a file, a whole directory, or a partition stem, and commits one transaction per partition, so a partition that fails leaves the ones already committed alone. A pair export wrote re-adds with no other flag; a foreign file -- one from a dataframe, or a values file whose partner was not copied -- carries less, and is told what it is missing:

infrastore --store demo.h5 add --parquet from_pandas.parquet \
    --owner-id 42 --owner-type Generator --name load

Three things to know. -f parquet requires --dir, because Parquet's footer sits at the end of the file and a writer has to seek back to it -- a pipe cannot. The data_hash half of the key is a checksum on the way back in: if you edited values in DuckDB, recompute it or pass --no-checksum. And a series with no values fails the export, naming every one -- an empty series would be a catalog row whose key matches no values rows, which is also what a truncated export looks like.

The full layout -- both column sets, the array key, the partition rules, the filenames, the footer, the merge join, and what a foreign file has to supply -- is in Parquet Layout.

Parquet is on by default, in the released binaries and in cargo install infrastore-cli alike. It is a cargo feature, so --no-default-features --features vendored builds a binary without the Arrow dependency tree; that binary still accepts the flags and tells you which feature to rebuild with, rather than reporting parquet as an unknown format.

Querying it with DuckDB

The reason to write Parquet rather than to embed a query engine here. Every question starts with the same join, on the pair that keys both halves:

SELECT s.name, s.owner_id, s.units, max(v.value) AS peak
FROM 'parquet/SingleTimeSeries.f64.utc.values.parquet' v
JOIN 'parquet/SingleTimeSeries.f64.utc.series.parquet' s USING (data_hash, time_axis)
GROUP BY s.name, s.owner_id, s.units;

Write it once as a view and nothing after has to think about it:

CREATE VIEW load AS
SELECT s.*, v.timestamp, v.value
FROM 'parquet/SingleTimeSeries.f64.America_Denver.values.parquet' v
JOIN 'parquet/SingleTimeSeries.f64.America_Denver.series.parquet' s USING (data_hash, time_axis);

-- One component's day, in its own spelling -- the timestamp column is zoned.
SELECT timestamp, value FROM load
WHERE owner_id = 42 AND name = 'load'
ORDER BY timestamp;

That view is the denormalized table the format deliberately does not write. Materializing it costs exactly what the split saves; keeping it as a view costs nothing.

A forecast joins the same way, and its window is a column of the values half:

SELECT v.issue_time, count(*) AS steps, max(v.value) AS peak
FROM 'parquet/Deterministic.f64.utc.values.parquet' v
JOIN 'parquet/Deterministic.f64.utc.series.parquet' s USING (data_hash, time_axis)
WHERE s.name = 'load_det'
GROUP BY v.issue_time
ORDER BY v.issue_time;

Note what a glob does not mix: 'parquet/*.values.parquet' only works across partitions whose columns agree, which is to say within one time_series_type. Reading a static partition and a forecast one together needs the columns named, since only the latter has issue_time.

The catalog is still there when you want what the files leave out -- the data_hash in its binary form, the feature sets, the association tables:

INSTALL sqlite; LOAD sqlite;
ATTACH 'demo.h5.sqlite' AS catalog (TYPE sqlite);

-- Two joins: the array key pairs the halves, then `id` reaches the catalog.
SELECT s.name, s.owner_id, max(v.value) AS peak, c.timestamps_hash
FROM 'parquet/SingleTimeSeries.f64.utc.values.parquet' v
JOIN 'parquet/SingleTimeSeries.f64.utc.series.parquet' s USING (data_hash, time_axis)
JOIN catalog.time_series_readable AS c ON c.id = s.id
GROUP BY s.name, s.owner_id, c.timestamps_hash;

time_series_readable is the catalog's hand-inspection view -- it hex-encodes the two content hashes and decodes the integer type codes, so the rows read as text (see Reading the SQLite catalog by hand). The id column on the series half is what reaches it; it is provenance only, and add assigns fresh ids rather than reusing it.

Stamp Provenance on the Artifact

A store built by a model run should say so. store-attr records free-form key/value provenance about the whole artifact -- who built it, from what source system, under which of your own schema versions:

infrastore --store demo.h5 store-attr set creator sienna-build
infrastore --store demo.h5 store-attr set source_system WECC-2032-ADS
infrastore --store demo.h5 store-attr list
infrastore --store demo.h5 store-attr get creator     # bare value on stdout
infrastore --store demo.h5 store-attr remove creator

The store never interprets a value, so structure rides in the text -- store JSON if you need it. The attributes live in the catalog, which means they travel with persist, survive compact, and show up under store_attributes in infrastore -f json store-info.

Three things to know. set replaces rather than appending: an artifact records one creator, not a history of them. get exits 1 when the key is unset, so a script can branch on it, while remove reports removed: false and exits 0. And keys beginning with infrastore. are reserved.

merge brings a source's attributes across without overwriting: a key the destination lacks is copied, a key both sides agree on is a no-op, and a disagreement is reported and left as the destination has it. diff gives them a section of their own and gates on them -- an artifact whose recorded source system changed is not the artifact you expected, even when every series is identical.

Note the name. attributes (below) is a different command about a different thing.

Associations

Two association catalogs live alongside the time series, readable and writable here:

infrastore --store demo.h5 attributes                 # component <-> supplemental attribute
infrastore --store demo.h5 attributes --summary       # counts by (component type, attribute type)
infrastore --store demo.h5 links --parent-id 42       # directed parent -> child edges
infrastore --store demo.h5 attach --from attachments.csv
infrastore --store demo.h5 link --parent-id 42 --parent-type Generator \
  --child-id 7 --child-type Bus
infrastore --store demo.h5 reassign --old 42 --new 43 # both catalogs follow a renumbered component

The store holds only the relationship — the components and attributes themselves live in the consumer's object graph — so the flags are bare ids and type names. attach --from and link --from import a whole table in one all-or-nothing transaction; their header is checked, because the four columns are two interchangeable-looking (id, type) pairs and a swapped file would silently invert every relationship. detach and unlink are the inverses, and take --dry-run.

Forecasts

All six writable types work (SingleTimeSeries, NonSequentialTimeSeries, PersistentTimeSeries, Deterministic, Probabilistic, Scenarios). infrastore template Deterministic prints a descriptor to edit, but it is plain JSON and says nothing about the data layout, so here is the rule:

Forecast CSVs are a flat, row-major stream of values with no structure of their own. The count must equal the product of the type's shape:

H = horizon / resolution — with the template's "horizon": "PT24H", "resolution": "PT1H", and "count": 7, a scalar Deterministic needs exactly 24 * 7 = 168 values, plus the header row. Use -f json to read the flat values back at full fidelity. get -f csv and export -f csv on a forecast emit timestamped analysis rows instead — one row per (window, step) with issue_time/target_time columns and one value column per percentile or scenario. add recognizes that header too and transposes the rows back, so an exported forecast re-adds exactly (see Reading back, and re-adding).

DeterministicSingleTimeSeries is not added from CSV — store a SingleTimeSeries, then derive it. transform takes no selector: it rewrites every SingleTimeSeries in the store. --horizon must fit inside each one (horizon / resolution steps must not exceed its length), so with the 6-row hourly load above, a 24-hour horizon fails and a 3-hour one works:

infrastore --store demo.h5 transform --horizon PT3H --interval PT1H

The derived series keeps the source's owner, name, and resolution, so load now matches two entries and a bare selector becomes ambiguous. Disambiguate with --type:

infrastore --store demo.h5 get --owner-id 42 --name load --type single
infrastore --store demo.h5 get --owner-id 42 --name load --type deterministic_single

A forecast's table view is the same structured one -f csv writes — issue_time, target_time, and one column per percentile or scenario — and --window N (or --issue-time <TS>) narrows it to a single window instead of dumping all of them:

infrastore --store demo.h5 get --name load --type deterministic_single --window 0

Compare and Move Stores

infrastore --store run.h5 diff --against baseline.h5      # exits 1 when they differ
infrastore --store demo.h5 merge --from other.h5 --name-glob 'load_*'
infrastore --store demo.h5 persist --dest backup.h5 --force

diff compares catalog identities and content hashes without reading either store's arrays, which makes it cheap enough for a CI gate: two series hold the same numbers exactly when they carry the same hash. merge moves arrays as bytes, so nothing is lost to a CSV round trip. persist is the one write that refuses an existing destination without --force: a save that fails partway may already have destroyed what was there.

Maintain It

infrastore --store demo.h5 verify                 # re-hash every array; exit 1 on a mismatch
infrastore --store demo.h5 check-consistency      # every SingleTimeSeries of a resolution on one grid
infrastore --store demo.h5 stats                  # association, owner, and distinct-array counts
infrastore --store demo.h5 remove --owner-id 42 --name load --dry-run
infrastore --store demo.h5 compact --force        # rewrite the .h5 so deletions actually shrink it

Deleting frees a column or unlinks a dataset, but HDF5 cannot give the space back in place, so the file only shrinks when compact rewrites it from the live set — nothing else may have the store open while it runs, and it is the one destructive command with no --dry-run. verify and diff use exit status 1 as an answer, not a failure; 2 is a usage error (Exit Status).

Shell completion for bash, zsh, fish, elvish, and PowerShell comes from the binary itself:

infrastore completions zsh > "${fpath[1]}/_infrastore"

Notes

• The CLI writes locally; there is no remote/gRPC mode yet (store access is isolated so one can be added later).
• Output is colored (green table headers) only when stdout is a terminal; it is plain when piped/redirected or when NO_COLOR is set, so -f json/-f csv stay clean for other tools.
• --log-level (or RUST_LOG) controls logging; the default is quiet (warn).
• The .h5 and .h5.sqlite files are one artifact — move, copy, and delete them together.

gRPC Server & Client Guide

The gRPC server exposes a store for remote, read-only access. Writes always require local filesystem access, so the service offers only list/get/keys/resolutions/counts/exists/verify. This guide covers running the server and talking to it from Rust. For the wire contract see the gRPC API reference; for the config file see Server Configuration.

When to Use It

Use the server to fan out reads of an existing store to many clients or across the network — for example, serving a published dataset to analysis jobs. A single writer produces the .h5 + .h5.sqlite pair locally; the server then reads it and answers queries. It never modifies the files.

Run the Server

1. Produce a store with any binding and flush() it to disk.

2. Write a config (start from examples/server.toml):

   [server]
   host = "0.0.0.0"
   port = 50051
   
   [data]
   file = "./system.h5"   # the .h5.sqlite catalog must sit beside it
   
   [authentication]
   method = "none"
   

3. Launch:

   cargo run -p infrastore-server -- --config my_server.toml
   # or, from a release build:
   ./target/release/infrastore-server --config my_server.toml
   

On startup the server validates the auth section, opens [data].file read-only, and serves the CatalogStore service on host:port. Set RUST_LOG=debug for verbose logs.

Check that it is up

With grpcurl and the proto file:

grpcurl -plaintext -proto proto/infrastore/v1/store.proto \
  127.0.0.1:50051 infrastore.v1.CatalogStore/GetCounts

Add -H 'x-api-key: replace-me-with-a-secret' when authentication is enabled. The equivalent from Rust is RemoteClient::connect(...).get_counts() — see The Rust Client below.

The Rust Client

RemoteClient mirrors the read methods of Store and returns the same core types. gRPC status codes are mapped back onto TimeSeriesError, so remote and local calls surface the same error taxonomy.

use infrastore_core::OwnerCategory;
use infrastore_server::client::RemoteClient;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = RemoteClient::connect("http://127.0.0.1:50051".into()).await?;

    let counts = client.get_counts().await?;
    println!("{} static series", counts.static_time_series);

    // Identify, then act. The owner is the (owner_id, owner_category) pair, and
    // every row carries the catalog id that addresses it.
    let rows = client
        .list_metadata(
            Some(42),
            Some(OwnerCategory::Component),
            None, None, None, None, None, None, None, None,
        )
        .await?;
    if let Some(id) = rows.first().and_then(|m| m.id) {
        let data = client.read_by_id(id, None).await?;
        println!("read {} values", data.as_single().unwrap().length);
    }
    Ok(())
}

A series is addressed by its catalog association id, never by a key: list_metadata (or list_metadata_by_ids, for ids a caller already recorded) hands back rows carrying id, and read_by_id / read_by_ids / get_metadata_by_id take it. association_exists answers whether a stored reference still resolves without fetching its row — the cheap way to validate a whole model on load.

Available methods: connect, from_channel, list_metadata, list_metadata_by_ids, get_metadata_by_id, association_exists, has_any_time_series, read_by_id, read_by_ids, get_resolutions, get_intervals, get_counts, counts_by_type, time_series_counts_detailed, get_forecast_parameters, list_owner_ids, static_summary, forecast_summary, check_static_consistency, verify_integrity.

Authentication

To require an API key, configure the server:

[authentication]
method = "api_key"
keys = ["replace-me-with-a-secret-1", "replace-me-with-a-secret-2"]

method = "api_key" with an empty keys list is rejected at startup. Clients must then send the key in the x-api-key metadata header; a missing or wrong key is rejected with Unauthenticated before the RPC runs. The comparison against the configured keys does not early-exit — every key of the same length as the supplied one is checked — so which key matched is not leaked by timing. The supplied key's length is not blinded; keys of a different length are rejected without a byte-wise compare, on the assumption that length is not secret.

RemoteClient::connect does not attach auth metadata, so against an authenticated server use the generated client with an interceptor that injects the header:

#![allow(unused)]
fn main() {
use infrastore_proto::pb::catalog_store_client::CatalogStoreClient;
use infrastore_proto::pb::CountsReq;
use tonic::metadata::MetadataValue;
use tonic::transport::Channel;

let channel = Channel::from_shared("http://127.0.0.1:50051")?.connect().await?;
let key: MetadataValue<_> = "replace-me-with-a-secret-1".parse()?;
let mut client = CatalogStoreClient::with_interceptor(channel, move |mut req: tonic::Request<()>| {
    req.metadata_mut().insert("x-api-key", key.clone());
    Ok(req)
});

let counts = client.get_counts(CountsReq {}).await?.into_inner();
}

Clients in Other Languages

The proto file at proto/infrastore/v1/store.proto is a standard proto3 definition. Generate a client for any gRPC-supported language from it, sending the x-api-key metadata header when the server requires authentication.

Benchmarks

infrastore-bench is a standalone binary (infrastore-bench) for measuring two critical performance dimensions: bulk ingestion and simulation-loop reads.

Build

# Debug build (fast to compile, slower to run)
cargo build -p infrastore-bench

# Release build (recommended for any real measurement)
cargo build --release -p infrastore-bench

The binary is placed at target/release/infrastore-bench.

Subcommands

Common flags

The read and all subcommands also accept --steps T (default: --length) to control how many simulation timesteps are benchmarked.

Add benchmark

# 10 000 components, 1 week hourly, in-memory
infrastore-bench add --count 10000 --length 168 --in-memory

# 100 000 components, same length, on-disk (tests real I/O + OS page cache)
infrastore-bench add --count 100000 --length 168

Reports for both SingleTimeSeries and Deterministic:

• Build requests — time to construct AddRequest objects in memory (array allocation only; excluded from the add throughput measurement).
• add_time_series_bulk — wall time, items/s, and data MB/s.

SingleTimeSeries arrays are column-packed into HDF5 datasets of up to 1 000 columns each and wrapped in a single SQLite transaction; the add benchmark stresses both. Deterministic arrays are standalone HDF5 variables; their add cost is dominated by per-variable write overhead and the same single-transaction SQLite commit.

Read benchmark

# 1 000 components, 168 timesteps, in-memory (pure CPU / metadata cost)
infrastore-bench read --count 1000 --length 168 --in-memory

# Same, but on-disk — store is written, dropped, then reopened read-only
infrastore-bench read --count 1000 --length 168

# Benchmark only the first 24 simulation steps of a 168-step series
infrastore-bench read --count 10000 --length 168 --steps 24

The read benchmark simulates the access pattern of an energy-simulation step loop:

for t in 0..T:
    for each component key:
        store.read_by_ids_range(ids, (t₀ + t·Δt, t₀ + (t+1)·Δt))

For on-disk stores the binary flushes, drops, and reopens the store read-only before the timed loop. This rebuilds the HDF5 in-memory index and starts the HDF5 chunk cache cold, reflecting real simulation startup conditions. The OS page cache may still be warm.

Reports per-step min / median / p95 / max and total component-reads/s.

Deterministic note. The current read_by_id implementation reads the full [H × count] array from storage and then slices to the requested window in memory. For large count (e.g. 168 windows) each single-step read transfers H × count × 8 bytes, which is 24 × 168 × 8 = 32 KB per component. The benchmark output flags this explicitly so the overhead is visible.

Interpreting results

The two metrics to watch:

• add throughput drops sharply when the number of items exceeds a SQLite transaction or HDF5 dataset threshold. If adding 100 000 items is notably slower per-item than adding 10 000, the bottleneck is likely the SQLite commit or HDF5 file growth, not array construction.

• Per-step time for the read benchmark scales linearly with --count. If the cost per step is much higher for on-disk than in-memory, the bottleneck is HDF5 chunk reads or SQLite metadata queries rather than Rust overhead. The packed SingleTimeSeries layout chunks (rows, cols) across the whole width — one timestamp per chunk, or a few for a narrow dataset (file format) — so reading one timestamp across many series is a chunk read per dataset, while reading one full series touches every chunk band (the slow direction, expected for exploration/plotting rather than hot loops).

Tracing spans for deeper diagnosis

When the benchmark numbers show a problem but don't tell you which layer is slow, enable the built-in tracing spans with --log-level:

# Show all debug-level spans from the store core only (least noise)
infrastore-bench --log-level infrastore_core=debug add --count 100 --in-memory

# Show everything — useful when the bottleneck might be in infrastore-bench itself
infrastore-bench --log-level debug add --count 100

RUST_LOG is also accepted as a fallback when --log-level is not provided.

The key spans emitted by infrastore-core:

Spans nest: a single add_time_series_bulk call groups packed series by shape and emits one put_packed_block span per group (filling whole chunks), plus a put_array → put_standalone span per standalone item; a single add_time_series call instead emits one put_array → put_packed span. This makes it straightforward to see whether time is spent in metadata insertion, HDF5 I/O, or the debug_span overhead itself.

The infrastore CLI supports the same --log-level flag for diagnosing a live store.

Reference

Information-oriented listings: exact signatures, schemas, and on-disk layouts. Reach for these pages when you already know what you want and need the precise shape of it. For narrative guidance, see the Developer Guides; for concepts, the Explanation section.

• On-Disk File Format — The HDF5 layout and SQLite schema, byte for byte.
• Parquet Layout — The values/series file pairs export -f parquet writes and add --parquet reads.
• Element Types — What element_type means, the row layouts it names, and the conformance corpus every binding's codec is tested against.
• Rust API — infrastore-core public types and Store methods.
• Python API — The infrastore module.
• Julia API — The InfraStore.jl package.
• C ABI — The infrastore_ffi cdylib functions.
• gRPC API — The infrastore.v1 service.
• Server Configuration — The server TOML file.

On-Disk File Format

A persisted store is a pair of files that must be kept together:

<name>.h5          # HDF5   — numerical arrays
<name>.h5.sqlite   # SQLite — metadata associations

The SQLite catalog path is the HDF5 path with .sqlite appended to the file name. This page is the authoritative description of both. For the rationale behind the split, see the Storage Model.

The array file is a plain HDF5 file, written directly against libhdf5. Store::open accepts only files it wrote itself (see storage_backend below). The .h5 extension is a convention — the store never inspects it.

Format Version

The HDF5 root carries four global attributes:

data_format_version = "0.19.0"
compression         = "deflate:3:shuffle"
storage_backend     = "hdf5"
catalog_generation  = "9f2c1ab4e70d5836c41b9e2af0d7c358"

data_format_version is the semver of the on-disk format (DATA_FORMAT_VERSION). It is bumped when the HDF5 layout, the SQLite schema, or the hashing domain changes in a backward-incompatible way. A purely additive SQLite table does not qualify: it costs old readers nothing and old stores pick it up from the idempotent DDL on their first writable open. The associations table landed that way.

compression records the filter policy the store was created with, so that appends made after reopening reuse the same filter. It is not part of the compatibility contract — see Compression below.

storage_backend marks the file as one infrastore wrote. Store::open checks it before reading anything else and rejects a file that lacks it with InvalidParameter — this is what distinguishes an infrastore store from an arbitrary HDF5 file, so a foreign file is refused rather than misread.

catalog_generation pairs this file with exactly one catalog; the same value lives in the catalog's catalog_identity table. Store::open compares the two and rejects a mismatch with MismatchedArtifact. It is written at creation and re-minted by every persist_to, which is what makes a save interrupted between its two renames detectable — see Saving. It is additive, so it carries no data_format_version change: a store written before it existed has neither the attribute nor the table, which reads as "unstamped" and skips the check rather than failing. Compaction preserves the existing value rather than minting a new one, since it rewrites only the HDF5 half.

Opening a store sorts its recorded version into one of three tiers, not two:

• Current — the stamp equals this build's DATA_FORMAT_VERSION. Opened as-is.

• Upgradable — the stamp is at least MIN_UPGRADABLE_VERSION and older. The array layout is compatible, so the file is read as it stands; a writable open runs the catalog migration ladder and then re-stamps the file. A read-only open changes nothing and leaves the stamp alone.

  Whether a read-only open succeeds is decided by the catalog half, not by this tier. CatalogMigrationRequired comes from a stale CATALOG_SCHEMA_REVISION; an upgradable stamp sitting over an already-current catalog opens fine. That is not a hypothetical combination — it is what an interrupted writable open leaves behind, and the step ordering is chosen to make it the harmless outcome.

• Incompatible — older than MIN_UPGRADABLE_VERSION, newer than this build, or unparseable (including an unstamped file). Fails with IncompatibleFormat, naming both versions. There is no upgrade path: regenerate the store with the matching build.

The catalog carries its own revision in schema_version (CATALOG_SCHEMA_REVISION), independent of the artifact version above. Any catalog change the idempotent DDL cannot make to an existing table — a new column, a changed CHECK, a rebuilt table, a backfill — needs a CATALOG_SCHEMA_REVISION bump plus an append-only migration, not a re-created store. A catalog written by a newer build is CatalogTooNew and is refused in both directions. Catalog revision 2, which widens the time_series_type CHECK from BETWEEN 0 AND 5 to >= 0, is the first such change and takes no DATA_FORMAT_VERSION bump at all — nothing in the HDF5 file moves, so a 0.19.0 store upgrades in place on its first writable open. See Upgrade a store in place.

PersistentTimeSeries (storage code 6) is the first type to arrive through that door, and it likewise takes no version bump. The HDF5 layout is unchanged — a persistent series pools into the same nsts_… datasets as a NonSequentialTimeSeries on the same breakpoints, and its breakpoints are an ordinary timestamp vector — so the widened CHECK is the whole of what it needed from the catalog, and every 0.19.0 store already has it.

Version history

Newest first. Each entry is the change that forced the bump; 0.19.0 is both the current DATA_FORMAT_VERSION and the current MIN_UPGRADABLE_VERSION, so only the top entry describes stores this build can open.

0.19.0 — moved a NonSequentialTimeSeries's timestamps out of the SQLite catalog and into the HDF5 file: the timestamp_sets table is gone, and each distinct time axis is now an i64 dataset of unix milliseconds under time_series/timestamps/, named by the same content hash the association row already carried. The hash domain changed with it — timestamps_hash is now the SHA-256 of those milliseconds rather than of the delta-varint blob the table held — so the nsts_… pool names change too. The timestamps are data, and belong in the half of the artifact built for data: a store may hold many distinct axes, each of them long, and a JSON document round trip needs them to travel with the artifact rather than be locked inside the catalog.

0.18.0 — made the id column of all three catalog tables (time_series_associations, supplemental_attribute_associations, and parent_child_associations) an INTEGER PRIMARY KEY AUTOINCREMENT rather than a bare rowid alias, so an id is never reissued once its row is deleted. Recycled ids matter because the id is an external reference — a consumer stores it in its own model — and reuse makes a stale reference resolve to a different, valid row.

This is the one bump that adds no column and changes no value. It was taken because AUTOINCREMENT is part of a table's declaration and there is no ALTER TABLE for it, so re-running the DDL over an older catalog would have left all three tables on recycled ids while reporting success — and, before the migration ladder existed, stranding that catalog was the only way to keep it out. The version floor is what enforces the guarantee, and it holds: MIN_UPGRADABLE_VERSION is 0.19.0, so every catalog predating this entry is Incompatible and refused on open. No store this build can open lacks AUTOINCREMENT. The silent-recycling failure above is what the bump prevented; it is not a state a store can be found in, and nothing needs to check for it at runtime (every_association_table_declares_autoincrement in crates/infrastore-core/tests/association_ids.rs holds a fresh catalog to the property, and sqlite_sequence is what proves the keyword took effect rather than merely being present).

The ladder has since changed what a repeat would cost, so do not reason from this entry to the next one. A table rebuild is now an ordinary migration — revision 2 is exactly that, and carries the AUTOINCREMENT high-water mark across by hand — so the same change made today would be a CATALOG_SCHEMA_REVISION bump that upgrades existing catalogs in place, not a DATA_FORMAT_VERSION bump that strands them.

0.17.0 — added the metadata column time_reference, which records how a series' timestamps were spelled: an instant in UTC, an instant at a fixed offset, an instant in a named IANA zone, or a wall clock naming no instant. It also changes how stored timestamps are interpreted — a row marked zoneless holds wall clocks the store keeps as if UTC, which an older reader would hand back as instants. See Time References.

0.16.0 — added the metadata column component_field.

0.15.0 — renamed the metadata column ext to application_data and added the quantity_kind and unit_system columns. Unlike a new table, new columns are not picked up by the idempotent CREATE TABLE IF NOT EXISTS DDL, so a store one version behind is rejected on open.

0.14.0 — moved a NonSequentialTimeSeries's timestamps out of the association row: the timestamps_json TEXT column became a timestamps_hash BLOB resolving into a then-new content-addressed timestamp_sets catalog table (which 0.19.0 replaced with the HDF5 datasets described above), and irregular arrays sharing a time axis became column-packed into nsts_… datasets keyed by that hash instead of one standalone arr_… dataset each.

0.13.0 — replaced the dtype column with element_type, which names the logical element type and derives the physical dtype from it. See Element types.

0.12.0 — changed owner_category and time_series_type from TEXT names to small INTEGER codes. See Discriminant encoding below.

0.11.0 — renamed the metadata column logical_type to ext, an opaque, package-owned extension payload (typically JSON) the store stores verbatim and never interprets.

0.10.0 — replaced the per-association features table with the content-addressed feature_sets table below, so a feature map is stored once and shared by every association that uses it, dropping the association_id foreign key and its ON DELETE CASCADE.

0.9.0 — changed the packed-dataset chunking to timestamp-major (1, cols, *element_shape) and made the column count cols per-dataset (sized to the writing batch) instead of a fixed 1,000, optimizing reads across series by timestamp and bulk writes.

0.8.0 — added the forecast interval to the association uniqueness key, so two forecasts of one variable differing only by interval became distinct series. This widened both unique indexes (the NULL-folding index now COALESCEs interval as well as resolution).

0.7.0 — made resolution/horizon/interval calendar-aware periods, encoded as ISO-8601 duration strings (e.g. PT1H, P1M, P1Y) rather than integer milliseconds, in both the packed dataset names and the SQLite columns, so irregular periods (Month/Quarter/Year) can be represented distinctly from fixed spans.

0.6.0 — added owner_category to the association uniqueness key, making the owner identity the pair (owner_id, owner_category) and widening the unique indexes and idx_owner.

0.5.0 — changed the owner identifier to a signed 64-bit integer (owner_id).

0.4.0 — the baseline the Rust port of InfrastructureSystems.jl shipped with, and the version that introduced DATA_FORMAT_VERSION itself.

0.3.0 — switched the time unit from nanoseconds to milliseconds, renaming the SQLite *_ns columns to *_ms and encoding the packed dataset name's {res} field in milliseconds instead of whole seconds.

0.2.0 — introduced typed, multi-dimensional arrays and the two-mode array layout below.

0.1.0 — stored only 1-D f64.

Arrays Are Typed and N-Dimensional

Every stored array is a TypedArray: an element dtype, a shape [length, k1, k2, …] whose first axis is time and whose trailing axes are a fixed per-step element shape, and the row-major element values.

Byte order differs between the file and everything outside it. In the HDF5 file each dataset's datatype records its byte order, which is the native order of the host that wrote it, and libhdf5 converts on every read. Outside the file, a TypedArray's buffer is always little-endian: that is what the content hash covers and what every binding and the C ABI exchange. A reader using its own HDF5 tools gets correct values from any host's file without knowing either fact.

The supported dtypes and their stable integer codes (shared with the bindings and the C ABI). Codes 0–5 are the original set and never move; new widths are appended:

A scalar-per-step series has an empty element shape; a per-step tuple (e.g. the 3 coefficients of a quadratic cost curve) has element shape [3].

The dtype says how wide an element is. What those elements mean — and how a ragged piecewise curve is packed into a fixed-width row — is the association's element_type; see Element types. The dtype above is derived from it.

The HDF5 file does not describe its own element typing, and is not meant to: bool and u8 are the same byte on disk, and nothing in a dataspace says whether three f64s are a quadratic curve or three independent samples. Every read takes the dtype from the catalog's element_type instead. The two artifacts are one logical store — an .h5 without its .sqlite is not readable in any case — so this removes a second, weaker source of truth rather than adding a dependency. Where a backend does know a dtype independently (a packed dataset's name encodes it), the catalog's value is checked against it and a mismatch is an integrity error.

HDF5 Layout

Arrays live under a two-level group hierarchy, in one of two storage modes, with the explicit time axes in a sibling group of their own:

<name>.h5
├── attribute  data_format_version = "0.19.0"
├── attribute  compression         = "deflate:3:shuffle"
├── attribute  storage_backend     = "hdf5"
└── group      time_series/
    ├── group  single/
    │   ├── dset  sts_{dtype}_{shape}_{length}_{res}      packed  (length, cols, *element_shape)
    │   ├── dset  sts_{dtype}_{shape}_{length}_{res}_h    u8  (cols, 64)   # per-column hex hashes
    │   ├── dset  sts_{dtype}_{shape}_{length}_{res}__1   packed spill dataset
    │   ├── dset  nsts_{dtype}_{shape}_{length}_{tshash}  packed irregular cohort
    │   ├── dset  nsts_{dtype}_{shape}_{length}_{tshash}_h  u8  (cols, 64)
    │   ├── dset  arr_{hex_hash}                          standalone  [length, *element_shape]
    │   └── ...
    └── group  timestamps/
        ├── dset  tsv_{hex_hash}                          i64  [n]   # unix milliseconds
        └── ...

Datasets carry no dimension scales and no dimension names — shape is read straight off the HDF5 dataspace. On open the backend recovers each packed dataset's column count from the second extent of its dataspace, which is how per-dataset widths round-trip.

The hash companion {dataset}_h is a (cols, 64) array of u8 holding each column's 64 lowercase hex characters as raw bytes — not an HDF5 string dataset. An all-zero row marks a free slot.

Packed mode

Used for every static series: SingleTimeSeries, the underlying array of a DeterministicSingleTimeSeries, NonSequentialTimeSeries, and PersistentTimeSeries. Many arrays that share a (dtype, element_shape, length) and a time axis are column-packed into one dataset:

The trailing element is what pins the time axis, which is what makes packing meaningful at all: the chunking is timestamp-major, so row t of a dataset is "every column at the same instant". A regular series takes its axis from the resolution; an irregular one carries the axis explicitly, and two of them share one exactly when their timestamp vectors are equal — which the store already answers by content-addressing those vectors (see Timestamp vectors). The pool key is the time axis and never the series type, so a PersistentTimeSeries and a NonSequentialTimeSeries on the same instants share one nsts_… dataset, and identical values on them share one content-addressed array. The full 64-char hash is used, not a prefix: the name is parsed back into the pool key when the index is rebuilt at open, so a truncated form could let two distinct time axes collide into one pool.

The dataset shape is (length, cols, *element_shape) and chunking is (rows, cols, *element_shape) with rows = 1 in the ordinary case, so one HDF5 chunk holds a single timestamp across every column — making a read across series by timestamp one chunk, and a buffered bulk write fill whole chunks. cols is chosen per dataset: a managed write sizes it to the batch it has in hand, while an incremental one-at-a-time write path uses a default width (DEFAULT_COLS_PER_DATASET = 1000) and fills one of its slots per call. In both cases cols is capped so one chunk stays within a byte budget (MAX_CHUNK_BYTES = 1 MiB); a batch wider than the cap spills across datasets.

"The batch it has in hand" is the whole of an add_time_series_bulk call — or the whole span of an open transaction, for irregular cohorts as well as regular pools: whether an irregular series packs with others on its axis or stands alone is decided from the block's final membership at the commit, not from the one request an add can see. Nothing a transaction writes is durable until its outermost commit, so a single add inside one is buffered per pool rather than dropped into a growth-pool slot, and the buffered arrays are written with the same block writer at the commit, or when a read needs one of them to have a physical position. A loop of single adds inside one transaction therefore produces exactly the datasets one bulk add of the same items produces — so long as the span reaches its commit without materializing early; only an un-transactioned single add takes the default width.

The buffer is the store's, not the file's: nothing on disk records it, and a reopen never sees one. The byte ceiling below and an early materialization are what can break that equivalence: crossing the ceiling, or asking a buffered array for its physical location (Store::locate_array, which takes &mut self for that reason), writes the span out as it stands. Each split costs an extra dataset and nothing else — the datasets are still block-written and chunk-aligned.

Two limits keep that buffer from being unbounded, and both simply write a block out early — the same spill a too-wide batch already performs, costing an extra dataset and nothing else:

A pool's block width is the lower of two ceilings, and the buffer as a whole has a third:

• Per pool, one chunk row of columns — MAX_CHUNK_BYTES over the element block, exactly the width a bulk add's block spills at, so the span's datasets are the bulk add's datasets. It is deliberately not DEFAULT_COLS_PER_DATASET: a bulk add issued inside a transaction is buffered too, and a thousand-column cap cut a 100,000-series batch into a hundred datasets where the same batch outside a transaction writes ten, which every per-timestep read then paid for chunk by chunk.
• Per pool, MAX_PENDING_BYTES over one column's bytes — length × element_block. A chunk row is a count of columns and says nothing about how long they are, so for anything but a short series this is the ceiling that actually binds: scalar f64 is 131,072 columns by the rule above, but 30,500 steps is 244,000 bytes a column, so the pool spills at 550. The two together are min(MAX_CHUNK_BYTES / element_block, MAX_PENDING_BYTES / (length × element_block)).
• Across every pool, MAX_PENDING_BYTES = 128 MiB of unwritten arrays. Per-pool shares do not add up to a global one, so a span touching several shapes is held to the total as well; crossing it writes out the widest block.

MAX_PENDING_BYTES is the only one of the three a caller can move: Store::set_write_buffer_bytes in the Rust core, store.write_buffer_bytes = n in Python, set_write_buffer_bytes!(store, n) in Julia, infrastore_store_set_write_buffer_bytes across the C ABI. Raising it is how a run of single adds gets the dataset the bulk add of the same series writes, since a batch handed over as a list is written as one block with no budget applied at all — the caller is already holding it. The chunk-row ceiling above it does not move, and neither does the figure travel: it belongs to the handle that set it, not to the artifact, because it is a budget for the writing process rather than a property of the file.

A block of one is not a block. If a span ends up holding a single array for a pool, it fills a growth-pool slot rather than claiming a dataset sized to one column — chunked (1, 1), that would give a scalar f64 series an eight-byte chunk per timestep, whose per-chunk overhead dwarfs the data. It is the same rule add_time_series_bulk applies to a pool its batch gives one array. For an irregular pool the fallback is a standalone arr_ dataset instead of a slot, because an nsts_ pool is shared only by the series on that exact axis: a cohort of one is a dataset spread over length chunks for no reason, where a regular pool is shared by every series on the resolution. From two columns up the block is what the bulk add of those items writes.

This is a write-time policy like every other choice on this page: the layouts it produces are ones the format already had, so it does not affect data_format_version and stores written either way stay mutually readable.

rows rises above one when a single timestamp row would leave the chunk under MIN_CHUNK_BYTES = 32 KiB. A narrow dataset is where that bites hardest, and a dataset is narrow when the width had to be small — which above is MAX_PENDING_BYTES / length for a span, so a long series got a 512-byte chunk. That is small enough to cost on both sides: deflate has too little to work with, and HDF5's fixed per-chunk cost stops being rounding error.

Measured on 512 series of 262,144 f64 steps written one at a time in one span — a 64-column block, so a 512-byte timestamp row — carrying a daily-plus-annual profile with AR(1) noise rather than anything conveniently compressible:

The 512-byte chunking wrote a file larger than the raw data: at that size deflate's per-chunk overhead exceeds what it saves. Write time bottoms out around 16–32 KiB and file size keeps falling past it. Sweeps are flat at every chunk size — a sweep visits every chunk regardless, and the rows a chunk brings along are the next ones it wants, so the reader's main pattern is untouched.

Scattered single-timestamp reads are the one thing taller chunks cost in principle, since they decompress rows nobody asked for. In practice the measurement varied by ±0.4 s run to run and could not separate 4 KiB from 32 KiB; a consumer whose access pattern is genuinely scattered rather than swept should benchmark it rather than trust this note.

The floor reaches past the narrow blocks it is aimed at. A 1,000-column f64 growth pool is an 8,000-byte row and takes five rows per chunk; a year of hourly f64 in a 1,915-column span block is 15 KiB and takes three. On that wide shape the trade measured as about 15% more write time for about 3% less on disk.

Chunking is a write-time policy: it is not recorded in data_format_version, HDF5 readers do not care, and files written under either rule stay readable.

• Rows are timesteps, columns are series. Column i holds one complete series.
• Hash companion dataset. Each packed dataset has a sibling {dataset}_h dataset of u8, shaped (cols, 64). Row i holds the lowercase hex SHA-256 (64 characters) of column i as raw bytes, or 64 zero bytes if the column is free. This is the on-disk index: on open, the backend scans every …_h, decodes the non-zero rows, and rebuilds its hash → (dataset, column) map. (The backend also recovers each dataset's cols from its column dimension length, so per-dataset widths round-trip.)
• Spill. When a family's current dataset is full — a batch exceeds the column cap, or incremental writes fill a default-width dataset — the next write creates a spill dataset …__1, then …__2, and so on.

Standalone mode

Used for the dense forecast arrays (Deterministic, Probabilistic, Scenarios), and for a NonSequentialTimeSeries or PersistentTimeSeries whose time axis nothing else shares. Each array is its own typed, multi-dimensional variable named arr_{hex_hash} in the time_series/single group. There is no column packing and no companion hash variable — the variable name carries the hash.

• A lone explicit-time-axis series (NonSequentialTimeSeries or PersistentTimeSeries) is shaped [length, *element_shape] and chunked as a single whole-array chunk. Its explicit, strictly-increasing timestamps are not in the array: they are a shared time axis, stored once in the timestamps group below. Packing is only a win once a cohort is several columns wide — a packed dataset spreads one array over length chunks — so a series alone on its time axis stays standalone. Whether a given array is packed or standalone is a write-time choice with no effect on reads (they resolve by content hash and handle either), and one cohort can hold columns of both.
• Dense forecasts are shaped [H, count, *element_shape] (Deterministic), [num_percentiles, H, count, *element_shape] (Probabilistic), or [num_scenarios, H, count, *element_shape] (Scenarios), where count is the number of forecast windows. They are chunked in bounded blocks along the count (window) axis — full on every other axis, cols windows wide, where cols is the largest count keeping one chunk within the same 1 MiB budget the packed datasets use. Reading a single window therefore decompresses one block rather than the whole array, and the ForecastReader aligns its in-memory cache to the same block width so sweeping the window timeline decompresses each block exactly once. The chunk width is not recorded anywhere: it is a write-time storage choice that reads transparently regardless of the width a store was written with, so it does not affect the data-format version.

Timestamp vectors

The explicit time axis of a NonSequentialTimeSeries lives in time_series/timestamps/, one dataset per distinct vector, named tsv_{hex_hash} after the vector's content hash — the same hash the association row carries in timestamps_hash and the nsts_… pool name carries as {tshash}. Each is a 1-D i64 dataset of unix milliseconds, one element per timestamp, and carries the file's compression policy once it is big enough to be chunked (short vectors go in the object header, unfiltered, exactly as small standalone arrays do).

Milliseconds are the store's precision floor for every instant it records, so this is exact for anything a write accepts; a finer timestamp is refused rather than truncated. Ordinary numbers in an ordinary dataset: h5py reads a time axis with no help from this library.

Storing it here rather than in the catalog is what makes the timestamps data. Irregular series in a power-systems model overwhelmingly share one axis — event times, an outage schedule, a market timeline — so a thousand components sampled at the same instants hold one copy between them; but a store may hold many distinct axes, each of them long, and a .sqlite file is the wrong place for bulk numeric arrays. It also means the vectors travel with the artifact: a document round trip that carries locators rather than values still finds the timestamps where it finds the arrays. (Before 0.19.0 they were a delta-varint blob in a timestamp_sets catalog table.)

Vectors are shared, so removing one series never deletes one; see Deletion and compaction. verify_integrity reads each referenced vector back and rehashes it, exactly as it does an array — the name records the hash, so a dataset perturbed behind it is caught rather than silently served.

Compression

Compression is chosen at store creation and applies to every data variable, packed and standalone alike (the …_h hash variables are strings and are not compressed). The default is DEFLATE (zlib) level 3 with the byte-shuffle filter; the level (0–9) and shuffle can be changed, or compression turned off entirely. The choice is persisted in the compression global attribute and restored when the store is reopened, so later appends reuse the same filter:

An absent or unparseable attribute falls back to the default (deflate:3:shuffle), which is what such a file was written with. Compression is a storage detail only: arrays decode transparently regardless of the filter, so stores written with different settings stay mutually readable and data_format_version is unaffected by the choice.

Deletion and compaction

• Packed: deletion zero-fills both the column's hash row and the column's data, so no stale values are readable through a reused slot. The slot becomes reusable by the next compatible write. The dataset does not shrink.
• Standalone: deletion unlinks the HDF5 dataset. The object is unreachable immediately and stays gone across a reopen, but the space it occupied is not returned to the filesystem until a compaction (HDF5 cannot reclaim space in place). Re-adding the same content therefore rewrites the dataset rather than re-indexing it.
• Feature sets and timestamp vectors: because both are shared, deleting an association never deletes either; removing the last association that referenced one leaves it unreachable. compact() deletes them — the feature set as a catalog row, the timestamp vector as an unlinked HDF5 dataset — and reports the counts as feature_sets_reclaimed and timestamp_sets_reclaimed. Clearing a store is the exception: it orphans every one of both by construction, so it drops them outright rather than leaving a cleared store's worth of them for a compaction.

compact() on an on-disk store rewrites the .h5 file, because that is the only way HDF5 gives the freed space back:

1. The catalog is swept of unreachable feature sets, and the file of unreachable timestamp vectors; the catalog is then read for the live set — the catalog, not the file, is what makes an array or a time axis live.
2. Every live array is written into a fresh file at <store>.h5.repack, sibling to the original so the two share a filesystem. Layouts are planned from scratch: packed pools are created at exactly their cohort width rather than the growth-sized width an incremental write reserves.
3. The store's HDF5 handle is closed, the temp file is renamed over the original, and the store reopens on the result. A crash before the rename leaves the original untouched plus a stray .repack file, which the next compaction deletes.

What the new file therefore lacks: freed packed slots, unreferenced datasets (an interrupted bulk add's leftovers, unreferenced timestamp vectors, or tombstones left by a store written before deletion unlinked), and the slack in over-wide packed pools. The .sqlite half is not touched — arrays are content-addressed, so a different physical layout is invisible to it — and data_format_version is unchanged, because the rewrite emits the same format.

Compaction assumes the process running it is the file's only user. On Unix another process holding the file open keeps reading the pre-compaction inode; on Windows its lock makes the rename fail, and the error surfaces with the compacting store still open on the original file.

SQLite Schema

The catalog database is created with PRAGMA foreign_keys = ON and the following DDL (idempotent — CREATE TABLE IF NOT EXISTS).

time_series_associations

One row per association between an owner and a stored array.

The two content-address hashes are the last two columns. Column order is not load-bearing — every statement names its columns — so the layout is chosen for readability.

timestamps_hash is set only on NonSequentialTimeSeries rows and is NULL on every other type. It is a locator, not a value: the vector itself lives in the HDF5 file (see Timestamp vectors), and the same hash is the cohort key the nsts_… pools are named by.

time_reference

One TEXT column holds all four spellings: utc, zoneless, a fixed offset (-07:00), or an IANA zone name (America/Denver). That is unambiguous rather than merely hoped for, because the core refuses a zone name that reads as an offset or as either literal — the utc literal is lowercase precisely so the IANA zone UTC stays a distinct value.

NULL means unspecified, never utc. For query bounds it groups with the three zoned spellings (an instant bound is accepted, a wall-clock bound refused), but it is not a claim the timestamps were written as UTC.

Deliberately not indexed, and that does not change now that ListFilter::zoneless exists. The idx_component_field partial-index pattern is wrong here twice over: WHERE time_reference IS NOT NULL would exclude exactly the NULL rows that filter has to return, and the column is low-cardinality — a handful of distinct values across a whole store — so it is not selective enough to earn an index. In practice it is combined with owner_id or name, which are indexed.

The column does not reach storage. array_hash takes no timestamp input at all, the packed dataset names carry no timestamp, and initial_timestamp stays RFC 3339 UTC whatever the reference says — a -07:00 series stores 2024-01-01T07:00:00Z and the label -07:00, and the offset is applied on the way out. Two series with equal values therefore pool into the same dataset and share a data_hash regardless of their references, which is correct: they are the same numbers.

feature_sets

The expanded feature map, one row per key. The typed columns are populated according to value_kind.

Feature sets are content-addressed, exactly as arrays are: the table is keyed by the SHA-256 of the feature map, and one set is stored once and shared by every association whose features_hash matches. The association row already carries that hash, so no join column is needed. Two associations with the same features therefore reference the same rows here — including a DeterministicSingleTimeSeries and the SingleTimeSeries it was derived from, which is why transform_single_time_series writes no feature rows at all.

An empty feature map stores no rows.

There is deliberately no foreign key to time_series_associations and no cascade: rows here are shared, so deleting one association must not delete a set another association still uses. Removing the last association that referenced a set instead leaves it unreachable — the same deletion semantics as the HDF5 side's unreachable standalone variables. Store::compact sweeps unreachable sets and reports the count as feature_sets_reclaimed; clearing a store drops them all outright.

supplemental_attribute_associations

Which supplemental attributes are attached to which components. Columns match infrasys' table of the same name, whose logic this replaces. See Associations Between Entities for the data model.

CREATE TABLE supplemental_attribute_associations (
    id             INTEGER PRIMARY KEY AUTOINCREMENT,
    component_id   INTEGER NOT NULL,
    component_type TEXT    NOT NULL,
    attribute_id   INTEGER NOT NULL,
    attribute_type TEXT    NOT NULL
);

CREATE UNIQUE INDEX uq_sa_assoc
    ON supplemental_attribute_associations(component_id, attribute_id);
CREATE INDEX idx_sa_assoc_attribute
    ON supplemental_attribute_associations(attribute_id, component_id, component_type);

uq_sa_assoc makes the (component_id, attribute_id) pair the row's identity — the type columns are denormalized labels for filtering, so the same pair under different type names is a duplicate and surfaces as DuplicateAssociation. That index also serves lookups keyed on the component; idx_sa_assoc_attribute serves the reverse direction ("which components carry this attribute").

One attribute may be attached to many components; one component may carry many attributes. Only the exact pair is constrained.

parent_child_associations

Directed edges between components — a generator (parent) connected to a bus (child), say. Both endpoints are always components, which is why there is no category column: a supplemental attribute cannot appear here by construction.

CREATE TABLE parent_child_associations (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    parent_id   INTEGER NOT NULL,
    parent_type TEXT    NOT NULL,
    child_id    INTEGER NOT NULL,
    child_type  TEXT    NOT NULL
);

CREATE UNIQUE INDEX uq_parent_child
    ON parent_child_associations(parent_id, child_id);
CREATE INDEX idx_parent_child_child
    ON parent_child_associations(child_id, parent_id, parent_type);

Identity is the ordered (parent_id, child_id) pair, so the reversed pair is a different edge. There is no relationship-kind column: two components may be related at most once. Recording a second kind of edge between the same pair would need such a column added and uq_parent_child widened — which is a format change, unlike the tables themselves.

Properties shared by both association tables

Neither table has a foreign key — to time_series_associations or anywhere else — and neither cascades. The endpoints live in the consumer's object graph, not in this store, so the store never observes a component or attribute being deleted and a cascade could never fire. Removing a time series therefore leaves both tables untouched, and vice versa; consumers issue both calls when they want both effects.

Both are also independent of each other: the same integer may name a row in each without collision.

Compatibility

Both tables were originally added without bumping data_format_version, since the gate is exact equality with no upgrade path and bumping for a purely additive table would have made every existing store unreadable in exchange for nothing.

That is no longer how they land. 0.18.0 gave both an AUTOINCREMENT id, which is part of a table's declaration and so cannot reach an existing table through CREATE TABLE IF NOT EXISTS; the version check rejects an older catalog before this DDL runs. Only a store at the current version can contain them, and it always does.

One consequence of the additive era survives, and is still required: opening read-only cannot run DDL at all, so reads of a missing table return empty results (0, false, no rows) rather than failing. That tolerance is not vestigial — it is what lets a read-only open work on a store whose catalog was never opened for writing.

store_attributes

Free-form key/value provenance about the artifact as a whole — who built it, from what source system, under which of the consumer's own schema versions. The store never interprets a value, in the same spirit as a row's application_data, and nothing here participates in any identity, hash, or query plan. See Store attributes for the model.

CREATE TABLE store_attributes (
    key   TEXT PRIMARY KEY NOT NULL,
    value TEXT NOT NULL
);

key is the primary key, so a set is an upsert rather than an append: an artifact records one creator, not a history of them. Values are TEXT; a caller wanting structure stores JSON, which is what application_data already asks of them. Keys beginning with infrastore. are reserved and refused on the write path — by the store, not by a CHECK, so a future build can stamp its own facts through a deliberate internal write without a format change.

The table is additive, like the two association tables: an existing store gains it on its first writable open, so it needed no CATALOG_SCHEMA_REVISION bump. A read-only open of a catalog written before it existed cannot run that DDL, so reads degrade to the empty answer rather than erroring. The table is not carried by the OpenAPI export or import, which have no place for it.

schema_version

CREATE TABLE schema_version (version INTEGER NOT NULL);

A single-column table holding the catalog's own schema revision — CATALOG_SCHEMA_REVISION, which is 2 as of this build. It holds at most one row: the DDL creates the table, and the migration ladder writes the revision, replacing whatever was there. A fresh catalog is stamped with the current revision directly rather than seeded at 1 and walked up the ladder.

An absent table, or a present one holding no row, reads as revision 1 — the pre-ladder shape, which is where the ladder starts. This tracks the SQLite schema alone and is distinct from the HDF5 data_format_version attribute, which governs the artifact as a whole and is the value open validates; the two move independently.

catalog_identity

CREATE TABLE catalog_identity (generation TEXT NOT NULL);

Holds zero or one row pairing this catalog with one HDF5 file, whose catalog_generation root attribute carries the same value. Store::open compares them and rejects a mismatch with MismatchedArtifact, which is what turns an interrupted persist_to — two renames that cannot be made atomic together — into a loud error instead of a store that quietly disagrees with itself. It also catches one half being copied without the other.

Added additively, so it lands on an existing store's first writable open without a data_format_version change. A store predating it has no row here and no root attribute, and reads as unstamped rather than as a mismatch. Because a read-only open cannot run the DDL, every read of this table tolerates the table being absent.

Indexes

CREATE UNIQUE INDEX uq_ts_assoc ON time_series_associations
    (owner_id, owner_category, time_series_type, name, resolution, interval, features_hash);
CREATE UNIQUE INDEX uq_ts_assoc_coalesced ON time_series_associations
    (owner_id, owner_category, time_series_type, name,
     COALESCE(resolution, ''), COALESCE(interval, ''), features_hash);

CREATE INDEX idx_hash       ON time_series_associations(data_hash);
CREATE INDEX idx_owner      ON time_series_associations(owner_id, owner_category);
CREATE INDEX idx_resolution ON time_series_associations(resolution);

-- Secondary indexes for the filter / discovery surface.
CREATE INDEX idx_ts_type        ON time_series_associations(time_series_type);
CREATE INDEX idx_name           ON time_series_associations(name);
CREATE INDEX idx_owner_type     ON time_series_associations(owner_type);
CREATE INDEX idx_category_owner ON time_series_associations(owner_category, owner_id);
CREATE INDEX idx_interval       ON time_series_associations(interval);
CREATE INDEX idx_component_field ON time_series_associations(component_field)
    WHERE component_field IS NOT NULL;

idx_component_field is the only partial index, because component_field is the only optional column anything filters on: a store that never sets it would otherwise pay index maintenance on every insert to record one NULL per row and buy nothing, so the WHERE clause makes that case cost zero entries. SQLite still uses it for the predicate the filter issues — component_field = ? cannot be true of a NULL whatever the parameter binds to — but it can never serve an IS NULL query, which is the same reason ListFilter::component_field cannot select the rows that left the field unset.

An index over a column that already exists is additive: a store gains any it lacks on its first writable open, at a one-time build cost proportional to catalog size, with no data_format_version change. idx_component_field is the exception, because it names a column introduced by a bump — it applies only to a catalog already carrying component_field, and an older store never reaches the DDL, being rejected by the version check first.

Together the two unique indexes enforce identity uniqueness; a violation surfaces as DuplicateTimeSeries. Both owner_id and owner_category are part of the key, so a component and a supplemental attribute that share an owner_id are independent owners. interval is part of the key, so two forecasts of one variable at the same resolution but different intervals are distinct series. SQLite treats NULL values as distinct in a UNIQUE index, so uq_ts_assoc does not constrain rows with a NULL resolution or interval (e.g. NonSequentialTimeSeries, or any static series, which carry no interval). uq_ts_assoc_coalesced covers that case by folding NULL to the empty-string sentinel via COALESCE before enforcing uniqueness (the empty string is never a valid ISO-8601 period).

These two were named uq_assoc and uq_assoc_coalesced before the association tables above existed and made a bare "assoc" ambiguous. The DDL drops the old names before creating the new ones, so a store written by an earlier build is renamed in place on its first writable open rather than accumulating two equivalent index pairs. Index names are not part of the on-disk contract, so this carries no data_format_version change.

time_series_readable (view)

CREATE VIEW time_series_readable AS
SELECT id, owner_id, owner_type,
       CASE owner_category WHEN 0 THEN 'Component'
                           WHEN 1 THEN 'SupplementalAttribute'
                           ELSE 'unknown(' || owner_category || ')' END AS owner_category,
       CASE time_series_type WHEN 0 THEN 'SingleTimeSeries'
                             WHEN 1 THEN 'NonSequentialTimeSeries'
                             WHEN 2 THEN 'Deterministic'
                             WHEN 3 THEN 'DeterministicSingleTimeSeries'
                             WHEN 4 THEN 'Probabilistic'
                             WHEN 5 THEN 'Scenarios'
                             WHEN 6 THEN 'PersistentTimeSeries'
                             ELSE 'unknown(' || time_series_type || ')' END AS time_series_type,
       name,
       initial_timestamp, resolution, length, horizon, interval, count,
       units, quantity_kind, unit_system, time_reference, component_field,
       element_type, element_shape, application_data,
       lower(hex(data_hash))       AS data_hash,
       lower(hex(features_hash))   AS features_hash,
       lower(hex(timestamps_hash)) AS timestamps_hash
FROM time_series_associations;

This view decodes both discriminants and every content hash, so it is the convenient thing to query by hand. Nothing in the library reads it.

A projection of the association table with both hashes hex-encoded, for humans opening the catalog in sqlite3 — see Reading the catalog by hand. Nothing in the library reads it. It costs no storage, is created by the same idempotent DDL as the tables above, and so lands on an existing store's first writable open without a data_format_version change.

Field Encoding Notes

• Timestamps are RFC 3339 strings in UTC, holding a whole number of milliseconds. The format could carry finer digits and a store written before the precision rule may, so a reader must parse the full string; but no current write path produces one — see timestamp precision.
• Periods (resolution, horizon, interval) are canonical ISO-8601 duration strings in SQLite (PT1H, P1M, P1Y), and the packed dataset name's {res} field uses the same encoding. Calendar periods (Month/Quarter/Year) are stored distinctly from fixed spans.
• Hashes are raw 32-byte BLOBs in SQLite, lowercase hex in HDF5 (the …_h dataset for packed arrays, the arr_ dataset name for standalone arrays). BLOB rather than hex TEXT in SQLite because the two hash columns sit in the association table, in idx_hash, and in both unique indexes — hex would cost roughly 32% more catalog space — and because a BLOB literal (X'…') compares case-insensitively while a hex TEXT column would not. The time_series_readable view supplies the readable form.
• element_shape is the per-step shape only (the trailing axes); the time length is a separate column.

Discriminant encoding

owner_category and time_series_type are stored as small INTEGER codes, not names:

Both columns sit in the two wide unique indexes, and time_series_type additionally in idx_ts_type while owner_category sits in idx_owner and idx_category_owner. A 1-byte code instead of a 9–29 byte string shrinks those indexes by roughly 35% and the whole catalog by roughly 29% on a 400k-row store, and makes type-scoped scans about 1.2x faster warm (1.4–1.5x under page cache pressure). Point lookups by key are unaffected — those are dominated by the 32-byte features_hash.

Two code assignments are load-bearing and cannot be reordered without a version bump:

• Deterministic (2) and DeterministicSingleTimeSeries (3) are adjacent. A request for Deterministic matches both (see the data model), so adjacency lets that widen to time_series_type BETWEEN 2 AND 3 — one index seek instead of a two-value IN.
• The static types (0–1) and forecast types (2–5) are contiguous blocks, so the static and forecast summary queries each scope with a single BETWEEN.

The C ABI uses the same integers for its ts_type arguments, so there is one mapping rather than two. The time_series_readable view decodes both columns for hand inspection.

Inspecting a Store by Hand

The quickest route is the CLI, which resolves the hash-to-location step for you:

infrastore --store system.h5 store-info          # both file paths, format version, compression
infrastore --store system.h5 arrays              # every distinct array, its location and sharers
infrastore --store system.h5 info --name load    # one series: data_hash, hdf5_dataset, hdf5_column

To go straight at the files:

h5ls -r system.h5                        # groups, datasets, dtypes, shapes
h5dump -A -n system.h5                   # root attributes and the object list
sqlite3 system.h5.sqlite '.schema'
# Query the view, not the table, to see decoded discriminants.
sqlite3 system.h5.sqlite \
  'SELECT name, time_series_type, element_type, element_shape, length FROM time_series_readable;'
# Both association tables are present in every current-format store.
sqlite3 system.h5.sqlite 'SELECT * FROM supplemental_attribute_associations;'
sqlite3 system.h5.sqlite 'SELECT * FROM parent_child_associations;'

Reading the catalog by hand

sqlite3 renders a BLOB as raw bytes in its default list mode and in .mode box / .mode json, which corrupts the terminal (and, in box mode, the table borders). Use the time_series_readable view, which is the association table with both hashes hex-encoded:

sqlite3 system.h5.sqlite 'SELECT name, data_hash FROM time_series_readable;'

The view spells hashes in lowercase, matching hash_hex in the core, every binding, and the CLI, so a value copied from it compares equal to one printed anywhere else. Against the base table, use hex(data_hash) or .mode quote; hex() returns uppercase.

The view is created by the same idempotent DDL as the tables, on a store's first writable open, so an older store gains it without a format bump. Nothing in the library reads it.

Mapping an association to its bytes

Read the association's data_hash. For a standalone array, read the dataset named arr_<hex_hash> directly. For a packed array, hex-encode the hash and find the matching row in the relevant sts_…_h dataset; that row index is the column index into the sts_… dataset. Note that a packed pool which fills up spills into {name}__1, {name}__2, so the dataset holding a given array is not derivable from its metadata — scan the _h datasets, or let infrastore info / infrastore arrays report the resolved dataset and column.

A NonSequentialTimeSeries row's time axis needs no scan: hex-encode its timestamps_hash and read time_series/timestamps/tsv_<hex_hash>, an i64 vector of unix milliseconds in the row's own order.

Parquet Layout

What infrastore export -f parquet writes and infrastore add --parquet reads: a normalized, partitioned layout. Per partition, two files sharing a stem — one holding every distinct array once, one holding the catalog rows that name them.

Two things it is not. Not one file per series: a store with thousands of series would become thousands of files, which defeats every reader worth exporting for. And not one table with the catalog row beside every value: the store is content-addressed, so a thousand components sharing one profile hold one array, and a denormalized table would write that profile a thousand times. Parquet's compression does not find repeats across pages, so the file really would be a thousand times larger.

This is a file interchange format, not the store's own. The on-disk store is HDF5 plus SQLite, and nothing here is load-bearing for it. Python's to_arrow() and from_arrow() are a different thing again: per-series, in-memory conveniences. The relationship is one sentence — a series file's columns are to_arrow()'s schema-metadata keys turned into columns, and the values file is its two columns keyed by the array — and neither side depends on the other.

The two files

parquet/
  SingleTimeSeries.f64.utc.values.parquet     one row per value, each array once
  SingleTimeSeries.f64.utc.series.parquet     one row per series

They join on the array key, and both are sorted by it. That is the entire relationship: a values row belongs to whichever series rows carry its key, and a series row reads whichever values group carries its key.

The array key

(data_hash, time_axis)

data_hash alone will not do. It covers the array's bytes and not the axis those bytes sit on, and the store pools irregular series by time axis precisely because two series with identical values on different timelines are one stored array. So the key is the pair.

time_axis is a string spelling whatever decides where a value sits in time, per type:

R24/2024-01-01T00:00:00Z/PT1H
9f0c4e...                                        (an irregular axis)
R24/2024-01-01T00:00:00Z/PT1H/PT2H/PT1H          (a forecast)

The instant is spelled in UTC whatever the partition's time_reference, because the reference is a partition key and repeating it would say nothing. A forecast needs its horizon as well as its interval: the horizon's own step decides how many target times a window has, so two forecasts sharing an array, an anchor and an interval but not a horizon are different tables.

The axis is read off the values being exported, not off the catalog row. The two agree for a whole-series export; where they disagree — export --time-range writes a slice whose anchor and length are its own — the values are what the file holds, and data_hash beside them is computed from the same slice.

Partitioning

Three things cannot vary within one Parquet file without nullable or ill-typed columns: the set of key columns (a forecast has an issue_time, a static series does not), the Arrow type of value, and the zone of timestamp. So an export partitions its selection by the triple

(time_series_type, value type, time_reference)

and writes one file pair per distinct triple. The payoff is that every column in both files is required: there are no nullable columns anywhere in this format, and a reader never has to ask whether a null means "absent" or "unknown".

The value type is the element type with its per-step shape, except that the four composite kinds (linear_function, quadratic_function, piecewise_linear, piecewise_step) partition by kind alone. Their stored width varies per series — a piecewise_linear row is [n, x₁, y₁, …] zero-padded to the widest timestep in that series — so keying by width would scatter one kind across a file per width. Instead every composite row in a partition is re-padded to the widest series in it, which the layout allows: the leading count n keeps each row self-describing whatever the padding. The cost is a data_hash caveat, below.

An unspecified time_reference is a partition of its own. A series that declared no spelling must not pool with one that declared UTC, even though both write a UTC-zoned column.

File names

<type>.<value-slug>.<reference-slug> plus .values.parquet or .series.parquet:

SingleTimeSeries.f64.utc.values.parquet
SingleTimeSeries.f64_2x3.America_Denver.series.parquet
Deterministic.tuple3_f64.offset_minus07_00.values.parquet
NonSequentialTimeSeries.piecewise_linear.zoneless.series.parquet
PersistentTimeSeries.f64.unspecified.values.parquet

Slugs avoid every character Windows forbids (<>:"/\|?*), which matters because a zone name carries / and a fixed offset carries :; a leading - is avoided too, since it reads as a flag wherever the name is passed to a command. Trailing dots go (Windows strips them, so a. and a would be one file there), and a fragment that would be a device name (CON, NUL, LPT9, …) gets a prefix.

The name is a convenience, not the truth. The slug function is one-way — the zones a/b and a_b both flatten to a_b — so nothing parses a partition back out of a filename, and two partitions that would collide get a numeric suffix in the keys' own sort order, so a re-run of the same export produces the same names. The footer carries the exact key.

Values file columns

Nothing about who owns the array is here, which is exactly what stops a shared profile being written once per component.

Series file columns