Introduction

infrastore is a Rust library for managing time-series data in power-systems and energy simulations. It separates persistence into two concerns: numerical arrays are stored in NetCDF4, while the metadata that associates each array with an owning component lives in SQLite. Identical arrays are stored once and shared through content addressing.

The library ships native Rust, Python (via PyO3), and Julia (via a C ABI) interfaces, plus the infrastore command-line tool and a read-only gRPC server with a Rust client.

flowchart TB
    subgraph clients["Language Interfaces"]
        RUST["Rust<br/>(native crate)"]
        PY["Python<br/>(PyO3 wheel)"]
        JL["Julia<br/>(C ABI)"]
        CLI["infrastore<br/>(CLI)"]
    end

    subgraph core["infrastore-core"]
        STORE["Store"]
        STORE --> NC[("NetCDF4<br/>arrays")]
        STORE --> SQL[("SQLite<br/>metadata")]
    end

    subgraph remote["Remote Access"]
        SRV["gRPC server<br/>(read-only)"]
        RC["Rust client"]
    end

    RUST --> STORE
    PY --> STORE
    JL --> STORE
    CLI --> STORE
    SRV --> STORE
    RC -->|"gRPC"| SRV

    style RUST fill:#4a9eff,color:#fff
    style PY fill:#17a2b8,color:#fff
    style JL fill:#9558b2,color:#fff
    style CLI fill:#fd7e14,color:#fff
    style STORE fill:#28a745,color:#fff
    style NC fill:#28a745,color:#fff
    style SQL fill:#28a745,color:#fff
    style SRV fill:#ffc107,color:#000
    style RC fill:#ffc107,color:#000

Key Features

• One array, stored once — Arrays are addressed by a SHA-256 content hash, so identical series shared across components are written to disk a single time (content addressing)
• NetCDF4 for arrays, SQLite for metadata — Numerical data lands in a compact, chunked NetCDF4 file; queryable associations live in a catalog SQLite database (storage model)
• Feature-tagged associations — Each association carries an arbitrary map of typed features (int / float / bool / str) so multiple variants of a series can coexist under one owner
• Typed, N-dimensional arrays — Store f64, f32, i64, i32, u64, or bool values, with an optional per-step element shape (e.g. the coefficient tuple of a cost curve)
• Three language bindings — Use it from Rust, Python, or Julia with the same on-disk format
• A infrastore command-line tool — Load time series from CSV, and list, read, and inspect a store straight from a terminal, with table / json / csv output (CLI how-to)
• Read-only gRPC service — Serve a store over the network for remote readers, with optional API-key authentication
• Designed for power-systems data — The data model maps onto InfrastructureSystems.jl and infrasys owners, categories, and time-series concepts

Who Should Read This

Next Steps

• Setting up? Start with Installation.
• Want the 60-second tour? Read the Quick Start for Python or Julia. Rust users can go straight to the Rust Developer Guide.
• Want to understand how it works? Read the Architecture.
• Need exact bytes on disk? See the On-Disk File Format.

Installation

infrastore is a Rust workspace. Building any of its interfaces requires a Rust toolchain plus the HDF5, NetCDF, and Protobuf system libraries. The Python and Julia bindings additionally need a Python interpreter (3.10+) or Julia (1.10+).

System Libraries

macOS (Homebrew)

brew install hdf5 netcdf protobuf maturin

hdf5 is a transitive dependency of netcdf, but the hdf5-metno-sys build script does not always locate it on its own. If cargo build fails with Unable to locate HDF5 root directory and/or headers, point it at the Homebrew install explicitly:

export HDF5_DIR="$(brew --prefix hdf5)"

Add that line to your shell profile to make it permanent.

Linux (Debian / Ubuntu)

sudo apt-get install libhdf5-dev libnetcdf-dev protobuf-compiler
# If the build script can't find HDF5:
export HDF5_DIR=/usr/lib/x86_64-linux-gnu/hdf5/serial

Rust Toolchain

The workspace targets edition 2024 and declares an MSRV of Rust 1.94 — that is the oldest toolchain it is guaranteed to build on. (rust-version in the root Cargo.toml is the authority; the repo pins no rust-toolchain file, and CI builds on stable.)

rustup update stable

Build the Workspace

git clone https://github.com/NatLabRockies/time-series-store
cd infrastore
cargo build --workspace --all-features
cargo test --workspace --all-features
cargo clippy --workspace --all-targets --all-features -- -D warnings

The workspace Cargo config (.cargo/config.toml) sets macOS linker flags so cargo build --workspace can link the PyO3 cdylib without maturin. On Linux those flags are inert.

Crates in the Workspace

Next Steps

• Build a store and round-trip a series in the Python or Julia Quick Start.
• Set up a language binding: Python · Julia.
• Stand up the gRPC server.

Quick Start (Python)

This walkthrough creates an in-memory store, adds a SingleTimeSeries, and reads it back — the shortest path to a working round-trip. It assumes the infrastore wheel is installed in the active environment; if import infrastore fails, see Integrate with Python.

A Minimal Round-Trip

from datetime import datetime, timedelta, timezone

import numpy as np
from infrastore import OwnerCategory, SingleTimeSeries, Store

# `in_memory=True` means no filesystem I/O. Pass `path=` instead to write a
# NetCDF file plus its SQLite catalog.
store = Store.create(in_memory=True)

# The name lives on the series object, not on `add_time_series`.
ts = SingleTimeSeries(
    datetime(2024, 1, 1, tzinfo=timezone.utc),  # initial timestamp (timezone-aware)
    timedelta(hours=1),                         # resolution
    np.arange(24, dtype=np.float64) + 100,      # 24 hourly values
    "load",                                     # name
)

# The owner is identified by an integer id, an owner type, and a category.
# Features and units are optional.
key = store.add_time_series(
    owner_id=42,
    owner_type="Generator",
    owner_category=OwnerCategory.Component,
    time_series=ts,
    features={"model_year": 2030},
    units="MW",
)

got = store.get_time_series(key)
print(f"read {got.length} values @ {got.resolution} from {got.initial_timestamp}")
# read 24 values @ PT1H from 2024-01-01 00:00:00+00:00
assert np.array_equal(np.asarray(got.data), np.asarray(ts.data))

What Just Happened

1. Store.create(in_memory=True) built a store backed by an in-memory array backend and an in-memory SQLite metadata database.
2. add_time_series hashed the array, wrote it to the backend (deduplicating on the hash), and recorded a metadata association keyed by (owner_id, owner_category, type, name, resolution, interval, features). It returned a TimeSeriesKey that can re-find the series.
3. get_time_series(key) looked up the association, read the array back by its content hash, and reconstructed a SingleTimeSeries.

The array is any NumPy array whose dtype is float64, float32, int64, int32, uint64, or bool — whatever you pass round-trips unchanged. Shapes beyond (length,) attach a per-step element shape, such as the coefficient tuple of a cost curve.

Slice and List

Pass a (start, end) tuple of datetimes to read a window instead of the whole series (end is exclusive):

window = store.get_time_series(
    key,
    time_range=(
        datetime(2024, 1, 1, 6, tzinfo=timezone.utc),
        datetime(2024, 1, 1, 12, tzinfo=timezone.utc),
    ),
)
print(window.length)   # 6

list_time_series returns metadata dicts, filtered by any combination of arguments:

for m in store.list_time_series(owner_id=42):
    print(m["name"], m["resolution"], m["units"], m["features"])
# load PT1H MW {'model_year': 2030}

Writing to Disk

Swap the constructor to persist:

store = Store.create(path="system.nc")
# ... add_time_series ...
store.flush()   # sync buffered NetCDF writes to disk

This produces two files that travel together:

• system.nc — the NetCDF4 file holding the arrays.
• system.nc.sqlite — the catalog holding the metadata associations.

Reopen them later with Store.open("system.nc", read_only=True).

Next Steps

• Work through the Python Developer Guide for forecasts, bulk reads, associations, and error handling.
• Understand the Data Model: owners, keys, and features.
• Browse the full Python API reference.

Quick Start (Julia)

This walkthrough creates an in-memory store, adds a SingleTimeSeries, and reads it back — the shortest path to a working round-trip. It assumes InfraStore.jl can find the native library; if the first store call errors, see Integrate with Julia.

A Minimal Round-Trip

using Dates, InfraStore

# `in_memory=true` means no filesystem I/O. Pass `path=` with `in_memory=false`
# to write a NetCDF file plus its SQLite catalog.
store = Store(in_memory=true)

# The name lives on the series struct, not on `add_time_series!`.
ts = SingleTimeSeries(
    DateTime(2024, 1, 1),    # initial timestamp
    Hour(1),                 # resolution
    collect(100.0:123.0),    # 24 hourly values
    "load",                  # name
)

# The owner is identified by an integer id, an owner type, and a category.
# Features and units are optional.
key = add_time_series!(
    store,
    42,             # owner_id
    "Generator",    # owner_type
    Component,      # owner_category
    ts;
    features = Dict("model_year" => 2030),
    units = "MW",
)

got = get_time_series(store, key)
println("read $(length(got)) values @ $(got.resolution) from $(got.initial_timestamp)")
# read 24 values @ 3600000 milliseconds from 2024-01-01T00:00:00
@assert got.data == ts.data

What Just Happened

1. Store(in_memory=true) built a store backed by an in-memory array backend and an in-memory SQLite metadata database.
2. add_time_series! hashed the array, wrote it to the backend (deduplicating on the hash), and recorded a metadata association keyed by (owner_id, owner_category, type, name, resolution, interval, features). It returned a TimeSeriesKey holding an opaque handle into the store.
3. get_time_series(store, key) looked up the association, read the array back by its content hash, and reconstructed a SingleTimeSeries. Note that resolution comes back as a Millisecond.

features is serialized to JSON, so its values must be JSON scalars (Int, Float64, Bool, String). The data is any AbstractArray of Float64, Float32, Int64, Int32, UInt64, or Bool; dimensions beyond the first attach a per-step element shape, such as the coefficient tuple of a cost curve.

Look It Up Without the Key

A series can also be addressed by its attributes, which is convenient when a caller keeps its own identifiers:

got = get_time_series(SingleTimeSeries, store, 42, Component, "load"; resolution = Hour(1))

for m in list_time_series(store; owner_id = 42)
    println(m["name"], " ", m["resolution"], " ", m["units"])
end
# load PT1H MW

get_time_series, has_time_series, and remove_time_series! all accept either a TimeSeriesKey or (owner_id, owner_category, name; resolution, features) attributes.

Writing to Disk

Swap the constructor to persist. The do-block form closes the store on exit, including on throw:

Store(in_memory=false, path="system.nc") do store
    add_time_series!(store, 42, "Generator", Component, ts; units = "MW")
    flush!(store)   # sync buffered NetCDF writes to disk
end

This produces two files that travel together:

• system.nc — the NetCDF4 file holding the arrays.
• system.nc.sqlite — the catalog holding the metadata associations.

Reopen them later with open_store("system.nc"; read_only=true), which has a do-block form too:

open_store("system.nc"; read_only=true) do store
    keys = get_time_series_keys(store, 42, Component)
    series = get_time_series(SingleTimeSeries, store, keys[1])
end

Next Steps

• Work through the Julia Developer Guide for forecasts, readers, associations, and error handling.
• Understand the Data Model: owners, keys, and features.
• Browse the full Julia API reference.

Explanation

This section explains how infrastore is put together and why. It is understanding-oriented: read it to build a mental model, not to accomplish a specific task. For step-by-step recipes see the How-To Guides; for exhaustive listings see the Reference.

• Architecture — The crates, the two-file storage split, and how the language bindings sit on top of a single core.
• Design Choices — What infrastore optimizes for and why, written for developers of parent packages like IS.jl and infrasys.
• Data Model — Owners, time-series types, keys, feature maps, and the associations between catalog entities.
• Storage Model — Why arrays go to NetCDF4 and metadata goes to SQLite, and how the two stay consistent.
• Content Addressing — How arrays are hashed, deduplicated, and verified.
• Language Bindings — How the Python, Julia, and gRPC interfaces wrap the Rust core.

Architecture

infrastore is a Rust workspace with one core library and a ring of interface crates around it. Every interface — native Rust, Python, Julia, the infrastore CLI, and the gRPC server — ultimately drives the same Store type in infrastore-core, and every interface reads and writes the same on-disk format.

Crate Layout

flowchart TB
    subgraph ifaces["Interface crates"]
        PY["infrastore-py<br/>PyO3 wheel"]
        FFI["infrastore-ffi<br/>C ABI cdylib"]
        SRV["infrastore-server<br/>gRPC server + Rust client"]
        CLI["infrastore-cli<br/>infrastore binary"]
    end

    PYMOD["infrastore<br/>(Python module)"]
    JL["InfraStore.jl<br/>(Julia package)"]
    PROTO["infrastore-proto<br/>protobuf + tonic"]

    subgraph core["infrastore-core"]
        STORE["Store"]
        META["MetadataStore<br/>(SQLite)"]
        BACK["StorageBackend<br/>(trait)"]
        NC["NetCdfBackend"]
        MEM["MemoryBackend"]
        STORE --> META
        STORE --> BACK
        BACK --> NC
        BACK --> MEM
    end

    PY --> STORE
    FFI --> STORE
    SRV --> STORE
    CLI --> STORE
    PYMOD -->|"import"| PY
    JL -->|"ccall"| FFI
    SRV --> PROTO

    style STORE fill:#28a745,color:#fff
    style META fill:#28a745,color:#fff
    style BACK fill:#1e7e34,color:#fff
    style NC fill:#1e7e34,color:#fff
    style MEM fill:#1e7e34,color:#fff
    style PY fill:#17a2b8,color:#fff
    style PYMOD fill:#17a2b8,color:#fff
    style FFI fill:#9558b2,color:#fff
    style JL fill:#9558b2,color:#fff
    style SRV fill:#ffc107,color:#000
    style PROTO fill:#ffc107,color:#000
    style CLI fill:#fd7e14,color:#fff

The Core: Store

Store is a thin orchestration layer that composes two collaborators:

• A StorageBackend that holds the numerical arrays, addressed by content hash.
• A MetadataStore (SQLite) that holds the associations between owners and arrays.

flowchart LR
    CALL["add_time_series(...)"] --> HASH["array_hash()"]
    HASH --> PUT["backend.put_array(hash, data)"]
    HASH --> INS["MetadataStore::insert(association)"]
    PUT --> NC[("NetCDF4")]
    INS --> SQL[("SQLite")]

    style CALL fill:#4a9eff,color:#fff
    style HASH fill:#6f42c1,color:#fff
    style PUT fill:#28a745,color:#fff
    style INS fill:#28a745,color:#fff
    style NC fill:#1e7e34,color:#fff
    style SQL fill:#1e7e34,color:#fff

The backend is chosen behind the StorageBackend trait. v0 ships two implementations:

• MemoryBackend — arrays in a hash map; selected when in_memory = true. No filesystem I/O.
• NetCdfBackend — arrays in a NetCDF4 file; selected when a path is given.

Because the seam is a trait, the metadata layer, the hashing, and every binding are identical no matter where the arrays live. Tests run against the memory backend; production uses NetCDF.

Why Two Files

Numerical arrays and their descriptive metadata have different access patterns. Arrays are large, append-mostly, and read by content; metadata is small, frequently queried, and benefits from indexes and transactions. infrastore puts each where it is strongest:

• Arrays → NetCDF4. Chunked, compressed, columnar storage that HDF5 tooling already understands.
• Metadata → SQLite. A queryable, transactional catalog at <path>.nc.sqlite.

The Storage Model page covers the trade-offs and the consistency protocol that keeps the two files in agreement.

Read Paths: Local and Remote

Writes always require local filesystem access — they go straight through a Store. Reads can happen two ways:

flowchart LR
    subgraph local["Local process"]
        APP["Your code"] --> STORE["Store"]
    end
    subgraph network["Over the network"]
        APP2["Reader"] --> RC["RemoteClient"]
        RC -->|"gRPC"| GS["gRPC server"]
        GS --> STORE2["Store (read-only)"]
    end

    style STORE fill:#28a745,color:#fff
    style STORE2 fill:#28a745,color:#fff
    style GS fill:#ffc107,color:#000
    style RC fill:#ffc107,color:#000

The gRPC server exposes a read-only subset of the API (list, get, keys, resolutions, counts, existence checks, integrity). It never writes. See Language Bindings and the gRPC Server guide.

Concurrency

Within a process, NetCdfBackend guards its NetCDF handle with a Mutex, so the storage backend itself is Send + Sync. The Store as a whole, however, is Send but not Sync: its MetadataStore wraps a single rusqlite::Connection, which is internally a RefCell and therefore cannot be shared between threads. In practice this means a Store can be moved to another thread, but sharing one across threads requires external synchronization — the gRPC server holds its store as an Arc<Mutex<Store>>, and the PyO3 binding marks the class unsendable so a Python Store stays on the thread that created it.

MetadataStore uses transactions for atomic multi-row writes. The library does not coordinate multiple processes writing the same file concurrently — a single writer owns the files at a time.

Design Choices

infrastore is a foundation library. End users rarely call it directly — they reach it through a parent package such as InfrastructureSystems.jl (IS.jl) or infrasys, which embed infrastore to persist the time-series data behind their component models. This page records the decisions that shape the API and the on-disk format, and the reasoning behind them, so that developers of those parent packages understand what infrastore optimizes for — and, just as importantly, what it deliberately does not.

Data Orientation: Optimize for Reading Every Component at One Timestamp

The decision. In the NetCDF file, SingleTimeSeries arrays that share a (dtype, element_shape, length, resolution) are packed as columns of one dataset: columns are series, rows are timesteps, and the HDF5 chunking is (1, cols, *element_shape) so that a single chunk holds one timestamp across every column. We optimize for reading all components' values at a given timestamp, and accept that reading one component's entire array is comparatively slow.

Why. The workload that matters is simulation. A production-cost or power-flow model steps through time and, at each step, needs the value of every generator, load, and branch for that one timestamp — a slice across series, not down one. With this layout that slice is a single chunk read; the ForecastReader / StaticReader columnar surface is built directly on it. The inverse access — pulling one component's full history — has to touch every chunk band and is slow by design. That trade is deliberate: the simulation read path is the hot one, and it is the one parent packages hand to their users.

What this means for parent-package developers.

• Lay out bulk writes so that series sharing a shape land in the same dataset — that is what fills whole chunks in one pass and keeps the timestamp-slice read fast. See add_time_series_bulk and bulk_add.
• Do not build a user-facing feature whose common path is "read this one component's entire array" and expect it to be cheap. It works, but it is the slow direction. If a downstream workload genuinely needs that orientation, that is a signal to raise with infrastore, not to work around with many single-series reads.
• The orientation is a property of the packed NetCDF layout only. NonSequentialTimeSeries and the dense forecast types are stored as standalone per-array variables and do not participate in it.

Values are immutable. There is no API — in any binding, by design — to edit a single value, slice, row, or column of an array already in the store. A stored array is added or deleted as a whole; changing data means writing a new array (content-addressed, so unchanged neighbors are not rewritten) and deleting the old one. This falls out of the two priorities above. A chunk holds one timestamp across many series, so editing one value would force a read-modify-write of a whole chunk band — the slow direction turned into the write path. And because arrays are content-addressed, an array's identity is its bytes: mutating it in place would invalidate the hash that every association row and every dedup reference depends on. Parent packages that expose "update this value" to their users must implement it as replace-the-array, not edit-in-place.

Forecast Storage: Chunked So One Window Is Cheap

Dense forecasts (Deterministic, Probabilistic, Scenarios) do not use the packed, cross-component layout above — each forecast is stored as its own standalone array, [H, count, *E] for a Deterministic, where H is the horizon length and count the number of forecast windows. But reading one window at a time is a first-class access pattern (a simulation stepping the forecast timeline, one issue time per step), so the array is chunked in bounded blocks along the count axis rather than as one whole-array chunk. A window read then decompresses one block instead of the entire year of windows.

Two consequences a parent-package developer should know:

• A window sweep is cheap; naive per-window reads are not, unless you go through the reader. Because a chunk is the decompression unit, reading a single window still pulls its whole block. The ForecastReader sizes its in-memory cache to that same block width, so stepping the window timeline decompresses each block exactly once. Reach for ForecastReader (or read the whole array once and index it) rather than issuing an independent whole-array read per window — the latter re-decompresses overlapping data.
• Cross-component savings come from dedup, not packing. Where static series share storage by packing many components into one dataset, forecasts share it by content addressing: identical forecast arrays are stored once, and ForecastReader reads each unique array a single time and fans it out to every component that references it.

The block width is a write-time storage choice only — it reads transparently regardless of the width a store was written with, so it does not change the on-disk format version.

Split Arrays From Metadata

Numerical arrays live in NetCDF4 and metadata associations live in a companion SQLite catalog, because the two have opposite size, access, and mutation profiles and each format is strongest at one of them. The full rationale, the consistency ordering that keeps the two files in step, and the compaction behavior are covered in the Storage Model.

Content-Address and Deduplicate Arrays

Arrays are keyed by the SHA-256 of their contents, so two associations with identical data share one stored array and writes are idempotent on hash. This is what lets many components reference the same profile without duplicating storage, and it is why deletes are reference-counted. See Content Addressing.

Keep the Multi-Language Surface Consistent

infrastore-core is the single source of truth; the Rust, Python (PyO3), Julia (C ABI), CLI, and gRPC interfaces are thin wrappers over the same Store. A capability is not considered done until it behaves the same across the bindings that support it, and unsupported operations return an explicit error rather than silently changing semantics. This keeps a parent package free to move between bindings — for example, Julia via the C ABI and Python via the wheel — without the data model shifting underneath it. See Language Bindings.

Data Model

The data model mirrors the time-series concepts originally developed in InfrastructureSystems.jl: a component (or supplemental attribute) owns one or more named time series, and each time series may exist in several variants distinguished by features.

Owners

Every time series belongs to an owner, identified by three fields:

owner_id is a signed 64-bit integer identifier. The owner identity is the pair (owner_id, owner_category): both participate in the association's uniqueness constraint, while owner_type is descriptive. Component and supplemental-attribute integer-id streams are independent, so the same owner_id can name a component and a supplemental attribute at once — the category disambiguates them, keeping the two owners' series distinct. Owner-scoped operations therefore take the category alongside the id (see Keys).

Time-Series Types

The data model defines six time-series types, all present in the TimeSeriesType enum and the metadata schema. Both static series types are implemented across every interface. The four forecast types support reading values across the Rust core, the C ABI, Python, Julia, and gRPC. Dense forecasts are written through the generic add_time_series across the Rust core, Python, and Julia (the C ABI keeps the per-type infrastore_store_add_forecast / infrastore_store_add_probabilistic transport functions), and DeterministicSingleTimeSeries is derived from stored SingleTimeSeries via transform_single_time_series (gRPC stays read-only):

All six types can be read from every interface: the Rust core, the C ABI, Python, Julia, the infrastore CLI, and the gRPC server. The write paths in the table are available in the Rust core, the C ABI, Python, Julia, and the CLI — never over gRPC, whose service is read-only. And no interface adds a DeterministicSingleTimeSeries directly: it only ever comes into existence by transforming a stored SingleTimeSeries.

DeterministicSingleTimeSeries is a storage-level view, and reads always return a Deterministic. This is by design in every binding: TimeSeriesData has no DeterministicSingleTimeSeries variant, and a read synthesizes the windowed Deterministic from the underlying static array without copying it. The DeterministicSingleTimeSeries tag stays visible in catalog surfaces — keys, metadata rows, counts, summaries — because callers need it to address, copy, or remove the association (and RequestedType::AbstractDeterministic exists so readers can match either concrete type without caring which is stored).

Reading forecast values is wired across the Rust core, the C ABI, Python, Julia, and gRPC. Writing dense forecasts goes through the generic add_time_series (a Deterministic, Probabilistic, or Scenarios object) in the Rust core, Python, and Julia, with the C ABI exposing the per-type infrastore_store_add_forecast / infrastore_store_add_probabilistic transport; a DeterministicSingleTimeSeries is produced by transform_single_time_series. The read-only gRPC server serves forecast reads but does not accept writes. See Forecasts below.

NonSequentialTimeSeries

A NonSequentialTimeSeries pairs each value with an explicit UTC timestamp. Timestamps must be strictly increasing and their count must match the data length. Its values are stored as a standalone NetCDF array; timestamps are stored with the association metadata.

SingleTimeSeries

A SingleTimeSeries is an initial_timestamp, a resolution (a period), and an array of values:

value
  ^
  |        *
  |     *     *
  |  *           *
  +--+--+--+--+--+--> time
   t0 t0+r  ...   t0+(n-1)r

The timestamps are implied — sample i is at initial_timestamp + i * resolution — so only the values are stored.

Periods

resolution, and the forecast horizon/interval, are calendar-aware periods, not plain fixed spans. A period is one of two kinds:

• fixed — a fixed nanosecond span (Hour, Minute, Day, Week), backed by a duration;
• calendar — a count of calendar months (Month = 1, Quarter = 3, Year = 12), where initial_timestamp + i * resolution is computed by calendar arithmetic (so a monthly grid lands on the same day-of-month each step rather than every N milliseconds).

A fixed period is never equal to a calendar one, even when their spans coincide for a given month. Periods are encoded as ISO-8601 duration strings (PT1H, P1M, P1Y) on disk and across every binding (the Python/gRPC surfaces accept a timedelta/duration for fixed periods and an ISO-8601 string for either kind, and return the ISO-8601 string).

Typed, N-dimensional arrays

Every series' values are a TypedArray: an element dtype (f64, f32, i64, i32, u64, or bool) and a shape [length, k1, k2, …]. The first axis is time; the trailing axes are a fixed per-step element shape, so a step can hold a scalar (empty element shape) or a small tuple — for example the 3 coefficients of a quadratic cost curve (element shape [3]). The optional ext payload travels with the metadata so a binding can reconstruct its domain object on read; the store itself never interprets it.

Forecasts

The four forecast types store their values as a content-addressed TypedArray in its native shape (the dense types as standalone NetCDF variables; a DeterministicSingleTimeSeries reuses its backing SingleTimeSeries array), while the windowing parameters live in metadata. A forecast association records horizon (the span each window covers), interval (the spacing between successive window start times), count (the number of windows), and — for Probabilistic — a percentiles vector.

The store does not interpret the layout — the caller owns the array shape (the Rust core takes a native-shape TypedArray inside a Deterministic / Probabilistic / Scenarios object; the C ABI takes a row-major byte buffer with explicit dims, and the Julia wrapper accepts a native array and serializes it row-major), and a DeterministicSingleTimeSeries deduplicates against the static series it forecasts. A DeterministicSingleTimeSeries is not added directly — it is derived from every stored SingleTimeSeries by transform_single_time_series (Rust core, C ABI, Python, Julia), sharing the underlying array. Because a DeterministicSingleTimeSeries is a synthetic view of a SingleTimeSeries, it is mutually exclusive with a real Deterministic for the same family (owner, name, resolution, features, regardless of interval): adding a Deterministic when a DeterministicSingleTimeSeries view exists — or deriving one when a Deterministic exists — raises InvalidParameter. Forecast values read back through the high-level path — get_time_series returns a forecast object in the Rust core, Python, and over gRPC, and Julia exposes get_time_series(Deterministic, …) / get_time_series(Probabilistic, …) / get_time_series(Scenarios, …) — while the low-level metadata + array path remains available for raw access. See the Rust API and C ABI.

Features

Two series can share an owner and a name yet differ — for example a load profile for model year 2030 versus 2050. Features disambiguate them. A feature map is a set of typed key/value pairs:

features = {"model_year": 2030, "scenario": "high", "calibrated": True}

Feature values are one of four kinds: int, float, bool, or str. Internally the map is sorted by key (a BTreeMap), which gives a stable order for hashing and for the uniqueness constraint.

Keys

A TimeSeriesKey is the logical handle that re-finds a series. It is exactly the tuple that must be unique:

TimeSeriesKey = (owner_id, owner_category, time_series_type, name, resolution, interval, features)

add_time_series returns a key; get_time_series, has_time_series, and remove_time_series take one. Two series with the same key cannot coexist — attempting to add a duplicate raises DuplicateTimeSeries. Change any element of the tuple (a different name, a different model_year feature, a different resolution, a different forecast interval, or a different owner_category) and you have a distinct series. interval is NULL for the static types (which never carry one); for forecasts it lets two series of one variable at the same resolution but different intervals (e.g. a day-ahead and a real-time forecast) coexist as distinct series. Because owner_category is part of the key, a component and a supplemental attribute that share a numeric owner_id keep entirely separate sets of series.

flowchart LR
    OWNER["owner_id=42, category=Component, type=Generator"]
    OWNER --> K1["name=load<br/>year=2030"]
    OWNER --> K2["name=load<br/>year=2050"]
    OWNER --> K3["name=max_active_power"]
    K1 --> A1[("array A")]
    K2 --> A2[("array B")]
    K3 --> A1

    style OWNER fill:#4a9eff,color:#fff
    style K1 fill:#17a2b8,color:#fff
    style K2 fill:#17a2b8,color:#fff
    style K3 fill:#17a2b8,color:#fff
    style A1 fill:#28a745,color:#fff
    style A2 fill:#28a745,color:#fff

Note that two different keys (K1 and K3 above) can point at the same underlying array. The key is a metadata concept; the array is shared by content addressing.

Optional Descriptors

Each association can also carry:

• units — a free-form, end-user-facing label such as "MW". No dimensional analysis is performed.
• ext — an opaque, package-owned extension payload stored verbatim (typically JSON, e.g. {"function_type":"QuadraticFunctionData"}) that a binding writes and reads to reconstruct a domain object. The store never parses or interprets it, and end users are not expected to set it.

These are recorded in metadata and returned on read, but they do not affect identity or storage.

Associations Between Entities

Beyond owning time series, catalog entities can be related to each other. The catalog records two such relationships, in two separate tables, because they are not the same kind of thing: attaching an attribute to a component and wiring one component to another have different identities and different query patterns.

Supplemental attributes attached to components

Identity is the (component_id, attribute_id) pair. The type names are denormalized labels, not part of identity: re-attaching the same pair under different type names is a duplicate and is rejected. One attribute may be attached to many components, and one component may carry many attributes; only the exact pair is constrained.

Parent/child edges between components

Both endpoints are always components, so unlike time-series owners there is no category to disambiguate. Identity is the ordered (parent_id, child_id) pair — the reversed pair is a different edge. There is no relationship-kind column, so a given pair may be related at most once.

Properties shared by both

Two consequences of the deliberate absence of foreign keys and cascades:

• Associations and time series are independent in both directions. Removing a component's time series does not remove its attribute attachments or its edges, and removing either does not touch any series. A consumer that wants both effects makes both calls.
• The store never observes a deletion it did not perform. Components and attributes live in the consumer's object graph, so a cascade could never fire; consumers call the matching remove_* with the appropriate filter instead.

Filtering takes lists of concrete type names, rendered into SQL IN (…). Expanding an abstract type into its subtypes stays in the calling language, where the type hierarchy lives.

Terminology: rows of the time_series_associations table — the owner-to-time-series records described above — are also called "associations" throughout this documentation and the code. They are unrelated to the entity-to-entity tables described in this section.

Both are available in the Rust core, the C ABI, Julia, and Python; neither is exposed over gRPC or the infrastore CLI. The supplemental-attribute surface is the wider of the two (it carries counts and a grouped summary) because each of its operations is driven by an existing consumer; the parent/child surface is deliberately narrower for now.

Storage Model

A persistent store is two files that travel together:

system.nc          # NetCDF4 — the numerical arrays
system.nc.sqlite   # SQLite  — the metadata associations

The catalog path is derived by appending .sqlite to the NetCDF file name. This page explains the split and how the two halves stay consistent. For the exact bytes, dataset names, and table columns, see the On-Disk File Format reference.

Why Split Arrays From Metadata

Arrays and metadata pull in opposite directions:

Forcing both into one format would compromise one of them. Instead, each lives where it is strongest, and the Store layer coordinates them.

The Array Side: NetCDF4

Arrays live under time_series/single/ in the NetCDF file, in one of two storage modes.

Stored arrays are immutable: a value array is added or deleted as a whole and is never edited in place — there is no API to mutate a row, slice, or column of an array already in the store. Changing data means writing a new array (content-addressed and deduplicated on write) and, if desired, deleting the old one.

Packed mode holds SingleTimeSeries (and the backing array of a DeterministicSingleTimeSeries). Arrays that share a (dtype, element_shape, length, resolution) are packed together as columns of one dataset named sts_{dtype}_{shape}_{length}_{res}, with shape (length, cols, *element_shape). The column count cols is sized to the batch that created the dataset (capped so one chunk stays within a byte budget); an incremental, one-at-a-time write path uses a default width of 1,000:

flowchart TB
    subgraph ds["dataset sts_f64_s_8760_PT1H  shape (8760, cols)"]
        direction LR
        C0["col 0<br/>series A"]
        C1["col 1<br/>series B"]
        C2["col 2<br/>(free)"]
        CN["col cols-1<br/>(free)"]
    end
    H["companion sts_f64_s_8760_PT1H_h<br/>cols hash strings"]
    C0 -.hash.-> H
    C1 -.hash.-> H

    style C0 fill:#28a745,color:#fff
    style C1 fill:#28a745,color:#fff
    style C2 fill:#6c757d,color:#fff
    style CN fill:#6c757d,color:#fff
    style H fill:#6f42c1,color:#fff

• Columns are series, rows are timesteps. Chunking is (1, cols, *element_shape), so one HDF5 chunk holds a single timestamp across every column. The layout favors bulk writes (a batch fills whole chunks in one pass) and reads across series by timestamp (one timestamp is one chunk). The reverse directions are the slow ones, by design: reading a single series in full touches every chunk band, and adding one series at a time rewrites a chunk band per timestep.
• A companion string variable holds the hashes. For each packed dataset there is a sibling {dataset}_h; slot i holds the SHA-256 hex of column i, or an empty string if the slot is free. This is the on-disk index the backend rebuilds on open.
• Datasets spill when full into …__1, …__2, and so on — when a batch exceeds the per-dataset column cap, or when incremental writes fill a default-width (1,000-column) dataset.
• Compression is configurable at store creation and applies to every data variable (packed and standalone). The default is DEFLATE (zlib) level 3 with the byte-shuffle filter; you may change the level (0–9), disable shuffle, or turn compression off entirely. The choice is recorded in a compression global attribute and restored when the store is reopened for appends, so later writes reuse the same filter. Compression is a storage detail only — arrays decode transparently regardless of the filter, so stores written with different settings stay mutually readable and the data-format version is unaffected. In-memory stores ignore the setting.

Standalone mode holds NonSequentialTimeSeries and the dense forecast arrays (Deterministic, Probabilistic, Scenarios). Each is its own typed, multi-dimensional variable arr_{hex_hash} — no column packing and no companion hash (the variable name carries the hash). Irregular series are shaped [length, *element_shape] and chunked whole; dense forecasts are shaped [H, count, *element_shape] (with extra leading axes for Probabilistic / Scenarios) and chunked in bounded blocks along the count (window) axis, so reading one forecast window decompresses a single block rather than the whole array. The ForecastReader caches a block at a time to match.

The file-format reference gives the precise naming and dimension scheme. Nothing on the array side distinguishes a forecast from a static series of the same physical shape — the type, timestamps, and windowing parameters all live in metadata.

The Metadata Side: SQLite

The catalog holds five tables. The first three describe time series; the last two record relationships between catalog entities and have nothing to do with time series at all.

• time_series_associations — one row per (owner_id, owner_category, time_series_type, name, resolution, interval, features) association, including the data_hash that links it to a packed column or standalone variable, the array typing (dtype, element_shape), the opaque package-owned ext payload, plus temporal fields, forecast parameters (horizon, interval, count, percentiles), units, and the features_hash.
• feature_sets — the expanded key/value pairs of a feature map, one row per key, typed by a value_kind discriminator. The table is content-addressed, exactly as arrays are: its primary key is (features_hash, key) — the same hash the association row already carries, so no join column is needed. A feature set is therefore stored once and shared by every association whose hash matches, not copied per association; an empty map stores no rows at all. Because the rows are shared, there is deliberately no foreign key to time_series_associations and no ON DELETE CASCADE (see Compaction below).
• schema_version — a single version column holding the catalog schema version (currently 1).
• supplemental_attribute_associations — which supplemental attributes are attached to which components, as (component_id, component_type, attribute_id, attribute_type). Identity is the (component_id, attribute_id) pair.
• parent_child_associations — directed edges between components, as (parent_id, parent_type, child_id, child_type). Identity is the ordered (parent_id, child_id) pair.

The last two are described in Associations Between Entities. They carry no foreign keys and no cascade, and they are independent of time_series_associations in both directions: removing a time series never touches them, and removing an association never touches a series. They were also added without a data_format_version bump, so a store written before they existed simply gains them on its first writable open — which is why every read of them tolerates the table being absent.

A unique index over (owner_id, owner_category, time_series_type, name, resolution, interval, features_hash) enforces the key uniqueness invariant at the database level. owner_category is part of the key, so a component and a supplemental attribute that share an owner_id are independent; interval is part of the key too, so forecasts of one variable that differ only by interval are distinct. Because SQLite treats NULL as distinct in a UNIQUE index, a second index folds NULL resolution and interval to a sentinel so series without them (e.g. NonSequentialTimeSeries, or any static series, which carry no interval) are still constrained. Indexes on data_hash, (owner_id, owner_category), and resolution keep lookups fast.

Keeping the Two Files Consistent

Because a write touches both files, Store follows a careful ordering so a failure cannot leave a dangling reference:

sequenceDiagram
    participant C as add_time_series
    participant B as NetCDF backend
    participant M as SQLite
    C->>B: put_array(hash, data)
    Note over C,B: idempotent on hash — staged for rollback
    C->>M: BEGIN
    C->>M: INSERT association
    alt insert succeeds
        C->>M: COMMIT
        C-->>C: return TimeSeriesKey
    else insert fails (duplicate etc.)
        C->>M: ROLLBACK
        C->>B: remove_array(staged hashes)
        C-->>C: return Err
    end

• Array first, metadata second. put_array is idempotent — calling it with an already-present hash is a no-op — so staging the array before committing metadata is safe.
• Metadata commit is the point of no return. If the SQLite insert fails (most commonly a DuplicateTimeSeries constraint violation), the transaction rolls back and any array column staged in this call is removed, returning the store to its prior state.
• Bulk writes are all-or-nothing. add_time_series_bulk (and the buffered bulk_add session) group packed series by shape and stage each group as one batch-sized block — filling whole chunks — then insert every association in one transaction; any error rolls the whole batch back and removes the staged arrays.

On delete, the order reverses and is reference-counted: the association rows are removed inside a transaction, then an array column is only zeroed/freed if no remaining association references that hash. This is what lets two keys share one array safely. Feature sets are shared too, but they are not reference-counted — deleting an association never deletes its feature set; the set is left unreachable for a later compact() to sweep.

Persistence and Copying

The NetCDF backend buffers writes. Call flush() (which issues nc_sync) before copying the files for backup or archival; afterward both system.nc and system.nc.sqlite can be copied as a pair without closing the handle. The two files must always be kept together — neither is usable alone.

Compaction

compact() reclaims space in both halves of the artifact, and the two halves behave differently.

On the array side, compaction reports rather than reclaims. Deleting a series frees its column slot but does not shrink the NetCDF dataset. The freed slot is transparently reused by the next compatible write (first_free), and a deleted standalone variable lingers as dead space. compact() reports how many slots are reclaimable (slots_reclaimed); v0 does not physically shrink datasets or drop dead variables, because netcdf-c cannot resize a dimension in place — that is a follow-up.

On the catalog side, compaction physically deletes. Because feature sets are shared, deleting an association cannot cascade into them: removing the last association that referenced a set leaves it unreachable, mirroring the NetCDF side's unreachable standalone variables. compact() sweeps those rows and reports the count as feature_sets_reclaimed — the one thing compaction actually removes. (Clearing the whole store is the exception that needs no sweep: it orphans every set by construction, so it drops them all outright.)

See compact.

Content Addressing

Every array is identified by the SHA-256 hash of its contents, not by a name or an ID. Two series with byte-identical values therefore resolve to the same hash and are stored exactly once. This is the mechanism that lets many keys share one underlying array.

The Array Hash

array_hash produces a deterministic 32-byte digest from a TypedArray. The hashed byte stream is, in order:

1. A dtype tag: the dtype name (f64, f32, i64, i32, u64, bool) followed by a NUL byte.
2. The shape: the rank as a little-endian u64, then each dimension as a little-endian u64.
3. The elements in row-major order. Float dtypes canonicalize NaN to a single quiet-NaN bit pattern before hashing; integer and bool bytes are hashed verbatim.

flowchart LR
    T["dtype tag<br/>e.g. f64\\0"] --> S["rank + dims<br/>(u64 LE each)"]
    S --> E["elements<br/>(typed LE, row-major)"]
    E --> H["SHA-256"]
    H --> D["32-byte hash"]

    style T fill:#6f42c1,color:#fff
    style S fill:#6f42c1,color:#fff
    style E fill:#6f42c1,color:#fff
    style H fill:#4a9eff,color:#fff
    style D fill:#28a745,color:#fff

Identity is therefore (dtype, shape, content):

• dtype and shape are part of the identity. Two arrays with the same numbers but a different dtype, or a different shape, never collide. Reshaping or retyping changes the hash.
• NaN is canonicalized (float dtypes only). Any NaN, regardless of its payload bits, hashes as a single canonical quiet-NaN pattern, so semantically equal float arrays never collide on NaN representation.

The Features Hash

Feature maps are hashed by features_hash with the same discipline: a domain tag, the entry count, then, in the BTreeMap's sorted-by-key order, a length-prefixed key plus a kind tag and the value bytes for each entry. Because the map is always sorted, insertion order does not affect the hash:

{"model_year": 2030, "scenario": 1}   # hashes identically to
{"scenario": 1, "model_year": 2030}

The features hash does double duty in the catalog:

• It identifies the association. It is stored in the features_hash column and is part of the metadata uniqueness index — it is how the database distinguishes two otherwise-identical associations that differ only in their features.
• It is the key of the feature set. The feature_sets table is keyed by (features_hash, key), so a feature map is content-addressed exactly as an array is: stored once and shared by every association whose hash matches. The association row already carries that hash, so no join column, foreign key, or association id is needed. An empty map stores no rows at all.

Sharing is not a rare case, it is the common one. Thousands of components typically carry the same feature set (often the empty one, or a single scenario tag), and it collapses to one copy. The sharpest example is a DeterministicSingleTimeSeries: it is a view over the SingleTimeSeries it was derived from and has exactly the same features, so transform_single_time_series writes zero feature rows — it reuses the set the source series already stored, just as it reuses the source's array.

Deduplication on Write

Both hashes deduplicate, by the same mechanism, on the same write.

The array. Store hashes the array and asks the backend whether that hash is already present:

• Present → the existing array is reused; no new array bytes are written. Only a new metadata association row is inserted.
• Absent → the array is written. A packed array (SingleTimeSeries) goes into the first free column of a compatible sts_{dtype}_{shape}_{length}_{res} dataset, recording its hash in the companion hash variable; a standalone array (NonSequentialTimeSeries, dense forecasts) becomes a new arr_{hash} variable.

The feature set. The association insert (MetadataStore::insert_batched) writes the feature set under its hash with an INSERT OR IGNORE, so a set some other association already stored is a no-op — equal hash implies equal set, so an ignored conflict cannot hide a different set behind the same hash. Within one batch a FeatureSetCache remembers the sets already written, so a bulk add issues one write per distinct set rather than a no-op statement per row per feature. This is what keeps a transform flat in feature count instead of linear in it.

So storage cost scales with the number of distinct arrays and distinct feature sets, while metadata cost scales with the number of associations. A profile shared by a thousand generators costs one array, one feature set, and a thousand small rows.

Deletion is Reference-Counted

Because arrays are shared, deleting an association cannot blindly delete its array. On remove_time_series (and clear_time_series), Store:

1. Deletes the matching association rows inside a SQLite transaction, collecting their data_hashes.
2. For each freed hash, counts how many associations still reference it.
3. Only frees the NetCDF column for hashes whose reference count has dropped to zero.

This keeps shared arrays alive until the last referencing key is gone.

Feature sets are the deliberate exception

Feature sets are shared by the same mechanism but are not reference-counted, and the symmetry stops there. Counting references on every delete would make deletion pay for a scan the array side already pays for, to reclaim a handful of tiny rows; the schema instead accepts unreachable rows and sweeps them in bulk. Concretely:

• No foreign key, no ON DELETE CASCADE. A cascade would be actively wrong: rows are shared, so deleting one association must not delete a set another association still uses.
• Deleting the last user leaves the set unreachable, rather than deleting it — the same end-state as the NetCDF side's dead standalone variables, which also linger.
• Store::compact sweeps them. It deletes every feature set no association references any more and reports the row count as feature_sets_reclaimed. This is the one thing compaction physically removes.
• Clearing the whole store drops them all outright, since a cleared store orphans every set by construction and may never see a compaction.

The practical consequence: an unreachable feature set is never read (every lookup goes through an association's features_hash), so it costs bytes, not correctness — and a re-added association with the same features silently adopts the row that was still sitting there.

Discovering Shared Series

Sharing is observable on read, not just an internal write optimization. Every metadata row carries its array's data_hash, so a caller can group series by that hash to learn which ones resolve to the same stored array — without reading any array bytes. Two cases land in the same group: arrays that were deduplicated because their content was identical, and a SingleTimeSeries together with any DeterministicSingleTimeSeries derived from it (the DST shares the backing array).

In the Julia binding, list_array_groups returns the list_keys rows plus a data_hash field (a 64-character hex string); group by it to find the shared sets. count_array_references reports, for one hash, how many SingleTimeSeries and DeterministicSingleTimeSeries associations point at it.

groups = Dict{String, Vector{NamedTuple}}()
for row in list_array_groups(store)
    push!(get!(groups, row.data_hash, eltype(values(groups))[]), row)
end
shared = filter(((_, rows),) -> length(rows) > 1, groups)   # arrays referenced by >1 series

This read-side view is what lets a downstream caller collapse work across owners that share data. The ForecastReader builds on the same grouping internally: forecasts that share an array (and read plan) are read from disk once per timestamp no matter how many components reference them — see Window-read deduplication. (This is exactly how InfrastructureSystems.jl backs its own get_shared_time_series and forecast reader.)

Stability is a Contract

These hashes are part of the on-disk format. Any change to a hashing domain above that perturbs a stored hash is a format-breaking change and must bump DATA_FORMAT_VERSION. Treat the hashing rules above as fixed, not as an implementation detail.

The golden_hash_pin integration test guards the array domain by pinning the exact SHA-256 of one fixed input (an f64 array of [0, 1, 2, 3]). That is a tripwire, not a proof: it covers a single dtype, shape, and value set, and it does not pin features_hash at all. A change to the hashing rules can therefore break the format without breaking that test — the reasoning above, not the test, is the contract.

Integrity Verification

Because the stored hash and the stored data are independent on disk, they can be cross-checked. verify_integrity walks every indexed column, recomputes array_hash from the stored values, and reports any mismatch between the recorded hash and the recomputed one — detecting silent corruption. See verify_integrity.

What it does not cover

The check is scoped to the array half of the store. It reads what the NetCDF side knows about and rehashes it; it never opens the SQLite catalog. A clean report is therefore a statement about the arrays, not about the store as a whole. Three things fall outside it:

• a data_hash in the catalog that names no stored array — a truncated or corrupted catalog, or a catalog paired with the wrong .nc file. Every read of the affected key fails, and verify_integrity reports nothing.
• a catalog row whose dtype, element_shape, or length misdescribes the array it points at.
• a missing catalog entirely. The two artifacts are one logical store, but nothing enforces that: opening read-write with the .sqlite half deleted silently recreates it empty, and the resulting store — zero time series, every array still present and now unreachable — verifies clean.

This is a deliberate scope rather than an oversight: the array side is where silent bit-level corruption happens, and it is the half no other call examines. The catalog has its own purpose-built checks — check_static_consistency for per-resolution grid agreement, and compact for the unreachable arrays and feature sets a delete leaves behind (an expected state, documented in the file format, not corruption). SQLite also enforces a good deal itself: the NOT NULL and CHECK constraints, and the two unique indexes that guarantee identity uniqueness.

If you need end-to-end assurance that a copied or restored store is intact, the operational rule in the file format is the one that matters: move, copy, and delete the .nc and its .sqlite together.

Language Bindings

Every interface wraps the same Store. Understanding how each binding bridges to the core explains why the APIs look the way they do, how errors propagate, and what each layer owns.

flowchart TB
    PYAPP["Python code"] --> PYO3["PyO3 classes<br/>(infrastore_py)"]
    JLAPP["Julia code"] --> JLPKG["InfraStore.jl"]
    JLPKG -->|"ccall"| CABI["C ABI<br/>(infrastore_ffi)"]
    RUSTAPP["Rust client code"] --> RC["RemoteClient"]
    RC -->|"gRPC / HTTP2"| GS["gRPC server"]

    PYO3 --> STORE["Store"]
    CABI --> STORE
    GS --> STORE

    style STORE fill:#28a745,color:#fff
    style PYO3 fill:#17a2b8,color:#fff
    style CABI fill:#9558b2,color:#fff
    style JLPKG fill:#9558b2,color:#fff
    style GS fill:#ffc107,color:#000
    style RC fill:#ffc107,color:#000

Python (PyO3)

infrastore-py uses PyO3 to expose Store as native Python classes in a module importable as infrastore. The binding:

• Converts Python datetime/timedelta to chrono types and NumPy arrays (any shape) to TypedArrays at the boundary, supporting the full dtype set (f64, f32, i64, i32, u64, bool).
• Translates the typed TimeSeriesError variants into a Python exception hierarchy rooted at TimeSeriesError (NotFoundError, DuplicateTimeSeriesError, InvalidParameterError, IntegrityError, ReadOnlyStoreError).
• Builds an abi3-py310 wheel, so one wheel works across CPython 3.10+ without recompiling.

The metadata side is owned entirely by Rust; Python never touches SQLite directly. See the Python guide and Python API reference.

Julia (C ABI)

Julia does not call Rust directly. Instead, infrastore-ffi compiles a C-compatible cdylib with an opaque-handle API, and InfraStore.jl ccalls into it.

flowchart LR
    JL["InfraStore.jl<br/>structs hold Ptr{Cvoid}"] -->|"ccall infrastore_store_*"| LIB["libinfrastore_ffi"]
    LIB --> STORE["Store"]
    LIB -.->|"infrastore_last_error_message"| JL

    style JL fill:#9558b2,color:#fff
    style LIB fill:#6f42c1,color:#fff
    style STORE fill:#28a745,color:#fff

The conventions that shape the Julia API:

• Opaque handles. InfraStore and InfraStoreKey are pointers; the Julia structs wrap them and register finalizers (close!, _finalize_key) that call the matching ts_*_free function.
• Status codes plus thread-local error messages. Every C function returns an int32_t code. On a non-zero code, Julia calls infrastore_last_error_message to retrieve the detail string and raises the matching Julia exception type.
• Out-parameters and caller-owned buffers. Arrays come back through an out-pointer plus a length and a dtype code; Julia copies them into a Vector{T} for the requested element type and frees the Rust buffer with the deallocator matching the buffer's element type — infrastore_buffer_free_f64, infrastore_buffer_free_u8, infrastore_buffer_free_i64, or infrastore_buffer_free_u64 (shape/dims buffers).
• Features cross as JSON. Julia serializes the feature dict to a JSON string, which the FFI layer parses into a Features map.
• Forecasts are wrapped. InfraStore.jl exposes Deterministic / Probabilistic / Scenarios structs passed to the generic add_time_series!, type-dispatched get_time_series(Type, …) getters, and transform_single_time_series!, so all four forecast types are usable from Julia.
• Bulk reads use a result handle. bulk_read reads many full SingleTimeSeries at once: the FFI fetches them in one decompress-once pass per dataset into a InfraStoreBulkReadHandle (infrastore_store_bulk_read_single), and Julia reads each element out, then frees the handle. Python's store.bulk_read exposes the same operation directly. Managed bulk writes already take the fast block-write path through the existing batch / add_time_series_bulk APIs.

InfraStore.jl loads the cdylib from the path in the INFRASTORE_LIB environment variable. See the Julia guide, the C ABI reference, and the Julia API reference.

InfrastructureSystems.jl Integration

The model was shaped to drop into InfrastructureSystems.jl: owners are identified by integer component identifiers (i64), owner categories map to Component / SupplementalAttribute, and features accept string values so InfrastructureSystems.jl's feature dictionaries round-trip unchanged. The FFI exposes attribute-based metadata accessors (infrastore_store_get_metadata, infrastore_store_has_by_attrs, infrastore_store_remove_by_attrs) and a hash-based array fetch (infrastore_store_get_array_by_hash) so an InfrastructureSystems.jl-side store can keep its own key objects and reach the array layer directly.

gRPC Server and Client

infrastore-server wraps a Store in a tonic gRPC service generated from infrastore-proto. It exposes a read-only slice of the API and adds optional API-key auth. The matching async RemoteClient mirrors the read methods and maps gRPC Status codes back to TimeSeriesError::ConnectionError, so remote calls surface the same error type as local ones.

Writes are deliberately not exposed over gRPC — they require local filesystem access. The server is for fan-out reads of an existing store. See the gRPC Server guide and the gRPC API reference.

CLI (infrastore)

infrastore-cli builds the infrastore binary, a thin wrapper over the core Store for use from a terminal. Unlike the gRPC server it is not read-only: it opens the on-disk .nc + .nc.sqlite pair directly and supports both reads and writes. Its shape:

• CSV in, store out. Numeric values come from a CSV; the metadata that does not fit a flat grid (owner, name, type, dtype, resolution, timestamps, units, features) is described in a descriptor JSON. All six dtypes and all five writable types are supported, forecasts included.
• **A global -f/--format selects table (default), json, or csv.
• Store access is isolated. All store opening lives behind one module, so a future remote/gRPC mode can be added without touching the command handlers; today there is no remote mode.

See the Use the infrastore CLI how-to and the CLI reference.

What Every Binding Shares

A file written by Python reads identically from Julia, Rust, or the server, because none of the bindings reimplement storage — they all funnel through the one core.

Feature Coverage Varies by Binding

The bindings funnel through one core, and the surface is now broadly consistent. Both static series types are available everywhere (read+write, except the read-only gRPC server), and forecasts read back across every interface. The remaining asymmetry is that the read-only gRPC server does not accept any writes:

The only gap is by design: writes (including forecasts added through add_time_series) require local filesystem access, so the read-only gRPC server serves forecast reads but not writes.

Developer Guides

These guides are written for developers building on infrastore from a specific language. Each walks through the full workflow — create or open a store, add series, query, read, and persist — with the idioms of that language, then points at the matching reference for exact signatures.

• Rust — Embed infrastore-core directly.
• Python — Use the infrastore PyO3 wheel.
• Julia — Use the InfraStore.jl package over the C ABI.
• gRPC Server & Client — Serve a store for remote readers.
• Benchmarks — Measure bulk-add and simulation-loop read performance.

If you just want a task-sized recipe (install, wire up a binding), see the How-To Guides. For the concepts underneath all of these, read the Explanation section — especially Data Model and Content Addressing, which apply equally to every binding.

Rust Developer Guide

This guide covers using infrastore-core from Rust. For exact signatures see the Rust API reference.

For a runnable end-to-end round-trip, examples/basic_rust.rs creates an in-memory store, adds a SingleTimeSeries, and reads it back:

cargo run --manifest-path crates/infrastore-core/Cargo.toml --example basic

Add the Dependency

The crate is part of this workspace. From another crate in the workspace:

[dependencies]
infrastore-core = { path = "../infrastore-core" }
chrono = "0.4"

You will use chrono::Duration/DateTime<Utc> for time and the crate's TypedArray for values, since those are the types the API speaks.

Open or Create a Store

#![allow(unused)]
fn main() {
use std::path::Path;
use infrastore_core::{create_store, open_store};

// In-memory (tests, scratch work): no filesystem I/O.
let mut store = create_store(None, true)?;

// On disk: writes system.nc and system.nc.sqlite.
let mut store = create_store(Some(Path::new("system.nc")), false)?;

// Reopen later, read-only.
let store = open_store(Path::new("system.nc"), /* read_only */ true)?;
}

Add a Series

#![allow(unused)]
fn main() {
use chrono::{Duration, TimeZone, Utc};
use infrastore_core::{
    Features, FeatureValue, OwnerCategory, SingleTimeSeries, TimeSeriesData, TypedArray,
};

let initial = Utc.with_ymd_and_hms(2024, 1, 1, 0, 0, 0).unwrap();
let values: Vec<f64> = (0..24).map(|i| 100.0 + i as f64).collect();
let data = TypedArray::from_f64(vec![24], &values);   // shape [length]; or [length, k1, ...]
// The name is part of the series object, not an argument to `add_time_series`.
let ts = SingleTimeSeries::new(initial, Duration::hours(1), data, "load");

let mut features = Features::new();
features.insert("model_year".into(), FeatureValue::Int(2030));

let key = store.add_time_series(
    42,                                     // owner_id
    "Generator",                            // owner_type
    OwnerCategory::Component,
    TimeSeriesData::SingleTimeSeries(ts),
    features,
    Some("MW".into()),                      // units
)?;
}

add_time_series returns a TimeSeriesKey — keep it to re-find the series, or rebuild it from its fields later. Adding a series whose key already exists returns TimeSeriesError::DuplicateTimeSeries.

Bulk inserts

For many series at once, add_time_series_bulk takes a Vec<AddRequest> and commits the whole batch atomically — any error rolls back every array and association in the call:

#![allow(unused)]
fn main() {
use infrastore_core::AddRequest;

let keys = store.add_time_series_bulk(vec![
    AddRequest {
        owner_id: 42,
        owner_type: "Generator".into(),
        owner_category: OwnerCategory::Component,
        data: TimeSeriesData::SingleTimeSeries(ts_a),   // the series carries its own name
        features: Features::new(),
        units: Some("MW".into()),
        ext: None,                             // opaque package-owned payload (e.g. JSON)
    },
    // ...
])?;
}

A bulk insert is also the fast write path: packed SingleTimeSeries are grouped by shape and written into batch-sized datasets so the timestamp-major HDF5 chunks are filled whole, rather than a slow column-at-a-time write. When you'd rather stream requests instead of building one big Vec, use a buffered session — it accumulates in memory and writes the same way on commit, discarding the buffer if dropped without committing:

#![allow(unused)]
fn main() {
let mut bulk = store.bulk_add();
for (owner_id, ts) in many_series {
    // `add` builds the AddRequest from its parts (ext = None);
    // `push` takes a prebuilt AddRequest when you need to set ext.
    bulk.add(owner_id, "Generator", OwnerCategory::Component,
        TimeSeriesData::SingleTimeSeries(ts), Features::new(), Some("MW".into()));
}
println!("staging {} series", bulk.len());   // also: bulk.is_empty()
let keys = bulk.commit()?;                    // consumes the session; keys in push order
}

BulkAdd buffers requests in memory and does no validation or I/O until commit, which is all-or-nothing. Dropping the session without committing writes nothing.

Adding series one at a time with add_time_series instead packs them incrementally into shared default-width datasets; that stays space-efficient but writes each column with a read-modify-write, so prefer a bulk insert or session when loading in volume.

Read a Series

Every lookup method (get_time_series, get_metadata, has_time_series, remove_time_series, bulk_read) takes a &KeyIdentity — the identifying tuple inside a key. Call TimeSeriesKey::identity() to get it:

#![allow(unused)]
fn main() {
let data = store.get_time_series(key.identity(), None)?;
let single = data.as_single().expect("this key names a SingleTimeSeries");
println!("{} values starting {}", single.length, single.initial_timestamp);
}

Slice on the time axis by passing a range; end is exclusive and the returned series' initial_timestamp/length reflect the slice:

#![allow(unused)]
fn main() {
let start = Utc.with_ymd_and_hms(2024, 1, 1, 6, 0, 0).unwrap();
let end   = Utc.with_ymd_and_hms(2024, 1, 1, 12, 0, 0).unwrap();
let window = store.get_time_series(key.identity(), Some((start, end)))?;
}

To read many whole series at once — say, loading everything for an interactive plot — bulk_read takes a slice of keys and returns a TimeSeriesData per key in order. It reads packed SingleTimeSeries in one decompress-once pass per dataset, which is much cheaper than a get_time_series per key (a single full-series read otherwise touches every chunk under the timestamp-major layout):

#![allow(unused)]
fn main() {
let ids: Vec<_> = keys.iter().map(|k| k.identity()).collect();
let series = store.bulk_read(&ids)?;
}

Query Metadata

list_time_series takes a ListFilter builder; every clause is ANDed, and the features clause is a subset match:

#![allow(unused)]
fn main() {
use infrastore_core::{ListFilter, OwnerCategory, TimeSeriesType};

let metas = store.list_time_series(
    ListFilter::new()
        .owner_id(42)
        .owner_category(OwnerCategory::Component)
        .time_series_type(TimeSeriesType::SingleTimeSeries)
        .name("load"),
)?;
for m in &metas {
    println!("{} {:?} units={:?}", m.name, m.resolution, m.units);
}

// All keys for one owner, an existence check, distinct resolutions, counts.
// The owner is the (owner_id, owner_category) pair.
let keys = store.get_time_series_keys(42, OwnerCategory::Component)?;
let present = store.has_time_series(key.identity())?;
let resolutions = store.get_resolutions(Some(TimeSeriesType::SingleTimeSeries))?;
let counts = store.get_time_series_counts()?;
}

list_keys(filter) is the key-centric counterpart of list_time_series, and list_keys_with_hash(filter) pairs each key with the 32-byte content hash of the array it resolves to, so you can group series that share stored data:

#![allow(unused)]
fn main() {
for (key, hash) in store.list_keys_with_hash(ListFilter::new().owner_id(42))? {
    println!("{} -> {}", key.identity().name, infrastore_core::hash::hash_hex(&hash));
}
}

The low-level read path

To read values without reconstructing a full SingleTimeSeries — for example when bridging to another store that holds its own keys — resolve metadata and fetch the array by hash:

#![allow(unused)]
fn main() {
let meta = store.get_metadata(key.identity())?;
let array = store.get_array_by_hash(&meta.data_hash)?;
}

Forecasts

Dense forecasts are written through the generic add_time_series by wrapping a Deterministic, Probabilistic, or Scenarios object in TimeSeriesData. Each forecast object holds a TypedArray in its native shape (the data model lists the conventional shapes per type); the store content-addresses it and records the windowing parameters:

#![allow(unused)]
fn main() {
use infrastore_core::{Deterministic, TimeSeriesData, TimeSeriesError, TypedArray};

// A Deterministic forecast: a [H, count, *E] array (here scalar steps, so [H, count]).
let (horizon_count, count) = (24, 7);
let values: Vec<f64> = vec![0.0; horizon_count * count];   // row-major, shape (horizon_count, count)
let data = TypedArray::from_f64(vec![horizon_count, count], &values);

// `new` returns Result<_, String>; map it if your function returns TimeSeriesError.
let forecast = Deterministic::new(
    initial, Duration::hours(1),      // initial_timestamp, resolution
    Duration::hours(24),              // horizon
    Duration::hours(24),              // interval
    count,
    data,
    "load_forecast",                  // name
).map_err(TimeSeriesError::InvalidParameter)?;

let key = store.add_time_series(
    42,                               // owner_id (i64)
    "Generator",
    OwnerCategory::Component,
    TimeSeriesData::Deterministic(forecast),
    Features::new(),
    Some("MW".into()),                // units
)?;
}

Probabilistic::new additionally takes the percentiles vector, and Scenarios::new takes a scenario_count; wrap them in TimeSeriesData::Probabilistic / TimeSeriesData::Scenarios the same way.

A DeterministicSingleTimeSeries is not added directly. Call transform_single_time_series to derive one from every stored SingleTimeSeries (it shares the backing array and derives count from the series length); it returns the number of series transformed. The trailing owner_category and resolution arguments are optional filters that restrict which series are transformed:

#![allow(unused)]
fn main() {
let n = store.transform_single_time_series(
    Duration::hours(24),                 // horizon
    Duration::hours(24),                 // interval
    Some(OwnerCategory::Component),      // owner_category filter (None = every category)
    Some(Duration::hours(1).into()),     // resolution filter (Option<Period>; None = every one)
)?;
}

get_time_series reconstructs forecasts too, returning a TimeSeriesData::Deterministic, Probabilistic, or Scenarios variant (a DeterministicSingleTimeSeries is synthesized into a Deterministic). Match on the variant or use the as_deterministic / as_probabilistic / as_scenarios accessors:

#![allow(unused)]
fn main() {
if let Some(d) = store.get_time_series(key.identity(), None)?.as_deterministic() {
    // d.data is the TypedArray; d.horizon, d.interval, d.count carry the forecast parameters
}
}

The low-level path is still available when you only need the raw array: get_metadata exposes horizon, interval, count, and percentiles, and get_array_by_hash returns the TypedArray in its stored shape (to_f64_vec returns a Result<_, String>, so map the error if your function returns TimeSeriesError):

#![allow(unused)]
fn main() {
let meta = store.get_metadata(key.identity())?;
let arr = store.get_array_by_hash(&meta.data_hash)?;   // arr.shape == [horizon_count, count]
let values = arr.to_f64_vec().map_err(TimeSeriesError::InvalidParameter)?;
}

When a forecast is addressed by its attributes rather than by a key you already hold, use resolve_forecast_key: it resolves (owner_id, owner_category, name, resolution, interval, features) plus a RequestedType to the single matching key, erroring with NotFound if nothing matches and InvalidParameter if the request is ambiguous. RequestedType::AbstractDeterministic matches a stored Deterministic or a DeterministicSingleTimeSeries, so callers need not know which one is stored:

#![allow(unused)]
fn main() {
use infrastore_core::RequestedType;

let key = store.resolve_forecast_key(
    42,
    OwnerCategory::Component,
    "load_forecast",
    None,                                  // resolution: Option<Period> (None = any)
    None,                                  // interval: Option<Period> (None = any)
    Features::new(),
    RequestedType::AbstractDeterministic,
)?;
}

Copy, Remove, and Maintain

copy_time_series re-points an existing association at another owner, optionally renaming it. It writes a metadata row only — arrays are content-addressed, so no data is duplicated — and it preserves the source's time_series_type (a DeterministicSingleTimeSeries stays one instead of being materialized into a dense Deterministic, which is what a read-then-write copy would produce):

#![allow(unused)]
fn main() {
let copied = store.copy_time_series(
    key.identity(),
    99,                    // dst_owner_id
    "Generator",           // dst_owner_type
    Some("load_copy"),     // new_name; None keeps the source name
)?;
}

#![allow(unused)]
fn main() {
store.remove_time_series(key.identity())?;   // one series
// The owner is the (owner_id, owner_category) pair.
store.clear_time_series(Some((42, OwnerCategory::Component)))?;  // all series for an owner
store.clear_time_series(None)?;        // everything

let report = store.compact()?;         // reports reusable slots
let integrity = store.verify_integrity()?;  // arrays only; catalog not checked
assert!(integrity.errors.is_empty());
}

Removal is reference-counted: a shared array survives until its last referencing key is gone. count_array_references(&hash) returns the (SingleTimeSeries, DeterministicSingleTimeSeries) association counts on one array, which is how you tell whether removing a SingleTimeSeries would orphan a forecast derived from it.

Associations

Two catalog tables record relationships between entities the store does not otherwise model, wholly independently of time series: which supplemental attributes are attached to which components, and directed parent/child edges between components. Removing a time series never touches either, and vice versa — see Associations Between Entities.

Attachments are keyed on the (component_id, attribute_id) pair. The type names ride along for filtering and are not part of identity, so re-attaching the same pair under different type names is a duplicate and fails with DuplicateAssociation:

#![allow(unused)]
fn main() {
use infrastore_core::{SupplementalAttributeAssociation, SupplementalAttributeFilter};

store.add_supplemental_attribute_association(SupplementalAttributeAssociation {
    component_id: 42,
    component_type: "Generator".into(),
    attribute_id: 100,
    attribute_type: "GeographicInfo".into(),
})?;

// Bulk add is one all-or-nothing transaction.
store.add_supplemental_attribute_associations(vec![SupplementalAttributeAssociation {
    component_id: 43,
    component_type: "Generator".into(),
    attribute_id: 100,
    attribute_type: "GeographicInfo".into(),
}])?;
}

Filters are all-optional and ANDed; the default matches everything, which is what makes a bulk export/import round trip. Queries run in both directions:

#![allow(unused)]
fn main() {
// The attributes on one component...
let attrs =
    store.list_supplemental_attribute_ids(&SupplementalAttributeFilter::new().component_id(42))?;
assert_eq!(attrs, vec![100]);

// ...and the components carrying one attribute.
let owners =
    store.list_components_with_attributes(&SupplementalAttributeFilter::new().attribute_id(100))?;
assert_eq!(owners, vec![42, 43]);

// `*_types` filters take CONCRETE type names, rendered as SQL `IN (…)`. Expanding an
// abstract type into its subtypes is the caller's job — the store has no type hierarchy.
// An empty list is a deliberate "none of these" and matches nothing.
let geo = SupplementalAttributeFilter::new().attribute_types(["GeographicInfo"]);
assert_eq!(store.count_supplemental_attributes(&geo)?, 1);   // distinct attributes
assert_eq!(store.count_components_with_attributes(&geo)?, 2); // distinct components

for row in store.supplemental_attribute_summary()? {
    println!("{} on {}: {}", row.attribute_type, row.component_type, row.count);
}

// Removal returns a count. Matching nothing is `Ok(0)`, not an error: assert on the
// count yourself if you expected a hit.
let removed = store
    .remove_supplemental_attribute_associations(&SupplementalAttributeFilter::new().component_id(43))?;
assert_eq!(removed, 1);
}

Parent/child edges work the same way, except that identity is the ordered pair — the reverse of an edge is a different edge — and both endpoints are always components, so there is no category:

#![allow(unused)]
fn main() {
use infrastore_core::{ParentChildAssociation, ParentChildFilter};

store.add_parent_child_association(ParentChildAssociation {
    parent_id: 42,
    parent_type: "Generator".into(),
    child_id: 7,
    child_type: "Bus".into(),
})?;

assert_eq!(store.list_children(&ParentChildFilter::new().parent_id(42))?, vec![7]);
assert_eq!(store.list_parents(&ParentChildFilter::new().child_id(7))?, vec![42]);

// Renumbering a component rewrites both ends of every edge in one statement, so an
// edge that names it twice is counted once.
let updated = store.replace_parent_child_component_id(42, 99)?;
assert_eq!(updated, 1);
}

Neither table is reachable over gRPC or the infrastore CLI.

Persist to Disk

The NetCDF backend buffers writes. Call flush before copying the files for backup:

#![allow(unused)]
fn main() {
store.flush()?;   // nc_sync; afterwards system.nc + system.nc.sqlite can be copied as a pair
}

persist_to writes the whole store to a new path — both halves of the artifact, path and <path>.sqlite, overwriting anything already there. It works for an on-disk store (it flushes and copies the pair) and for an in-memory store, which is how you materialize a scratch store built with create_store(None, true):

#![allow(unused)]
fn main() {
let mut store = create_store(None, true)?;   // in-memory
// ... add series ...
store.persist_to(Path::new("system.nc"))?;   // writes system.nc + system.nc.sqlite
}

Always keep the .nc and .nc.sqlite files together — neither is usable alone.

Error Handling

Every fallible method returns Result<T, TimeSeriesError>. Match on the variant to react:

#![allow(unused)]
fn main() {
use infrastore_core::TimeSeriesError;

match store.get_time_series(key.identity(), None) {
    Ok(data) => { /* ... */ }
    Err(TimeSeriesError::NotFound) => { /* missing */ }
    Err(TimeSeriesError::ReadOnlyStore) => unreachable!("this is a read"),
    Err(e) => return Err(e.into()),
}
}

Threading

Store is Send but not Sync: the SQLite catalog holds a rusqlite::Connection, which is not shareable across threads. A store can therefore be moved into another thread, but it cannot be shared by reference — not even for concurrent reads.

To use one store from several threads, wrap it in external synchronization and serialize every access, reads included: Arc<Mutex<Store>> (this is exactly what the gRPC server does). The library does not coordinate multiple processes writing the same files either.

Python Developer Guide

This guide covers building on the infrastore PyO3 module. For exact signatures and return shapes, see the Python API reference. To install the wheel into your environment, see Integrate with Python.

Import

from datetime import datetime, timedelta, timezone
import numpy as np
from infrastore import Store, SingleTimeSeries, OwnerCategory, TimeSeriesType

The module exposes Store; the static series classes SingleTimeSeries and NonSequentialTimeSeries; the forecast classes Deterministic, Probabilistic, and Scenarios; TimeSeriesKey; the TimeSeriesType and OwnerCategory enums; the init_tracing function; and an exception hierarchy rooted at TimeSeriesError.

Open or Create a Store

# In-memory: no filesystem I/O.
store = Store.create(in_memory=True)

# On disk: writes system.nc and system.nc.sqlite.
store = Store.create(path="system.nc")

# Reopen read-only.
store = Store.open("system.nc", read_only=True)

Build a Series

SingleTimeSeries takes a timezone-aware datetime, a resolution (a timedelta or an ISO 8601 duration string such as "PT1H" — the string form is required for calendar periods like "P1M"), and a NumPy array:

ts = SingleTimeSeries(
    datetime(2024, 1, 1, tzinfo=timezone.utc),
    timedelta(hours=1),
    np.arange(24, dtype=np.float64) + 100,
    "load",  # name (required)
)

Use timezone-aware datetimes (UTC is stored). The binding is dtype-generic — it accepts and returns NumPy arrays of float64, float32, int64, int32, uint64, or bool, and whatever dtype you pass round-trips unchanged. The array may be multi-dimensional: shape (length,) for scalar steps, or (length, k1, …) to attach a per-step element shape (such as cost-curve coefficients). The required name is an association attribute carried on the object — the same array can be added under different names. Use NonSequentialTimeSeries(timestamps, data, name) for explicitly timestamped series.

Add a Series

key = store.add_time_series(
    owner_id=42,
    owner_type="Generator",
    owner_category=OwnerCategory.Component,
    time_series=ts,   # name comes from ts
    features={"model_year": 2030, "scenario": "high"},
    units="MW",
)

features is a plain dict whose values are int, float, bool, or str. Adding a series whose key already exists raises DuplicateTimeSeriesError. The returned key exposes owner_id, owner_category, time_series_type, name, resolution, interval, and features as read-only properties (resolution and interval are ISO 8601 duration strings or None).

Read a Series

got = store.get_time_series(key)
assert np.array_equal(np.asarray(got.data), np.asarray(ts.data))
print(got.length, got.initial_timestamp, got.resolution)

Slice on the time axis with a (start, end) tuple of datetimes (end exclusive):

window = store.get_time_series(
    key,
    time_range=(
        datetime(2024, 1, 1, 6, tzinfo=timezone.utc),
        datetime(2024, 1, 1, 12, tzinfo=timezone.utc),
    ),
)

To read many whole series at once — e.g. loading everything for a plot — bulk_read takes a list of keys and returns the typed series objects in the same order. Packed SingleTimeSeries are read in one decompress-once pass per dataset, which is much faster than a get_time_series per key:

series = store.bulk_read(keys)   # keys: list[TimeSeriesKey]

Per-Timestamp Reads (Simulation Loop)

get_time_series hands back a whole series or forecast. Simulations instead walk the timeline and, at each timestamp, want the value of every series at that instant. For that, build a reader once and drive it in a loop — it pins one resolution and reuses its output buffers, so the loop allocates almost nothing. StaticReader serves SingleTimeSeries; ForecastReader serves forecasts. (Full signatures: Python API reference.)

Static series

Series are grouped by (dtype, element_shape); each group's group_values is one dense (num_columns, *element_shape) array whose columns line up with that group's keys. All matched series must share one grid (initial_timestamp + length), validated at build.

reader = store.build_static_reader(timedelta(hours=1))
grid = reader.grid()               # {"initial_timestamp", "resolution", "length"}
groups = reader.groups()           # each: {"dtype", "element_shape", "keys"}
for ts in reader.timestamps():
    store.static_read(reader, ts)
    for i, g in enumerate(groups):
        vals = reader.group_values(i)   # (num_columns, *element_shape); column j ↔ g["keys"][j]

Forecasts

entry_values(i) returns the window backing entries()[i], shaped (horizon, *element_shape) for Deterministic/DeterministicSingleTimeSeries, (num_percentiles, horizon, *element_shape) for Probabilistic, and (scenario_count, horizon, *element_shape) for Scenarios. A Deterministic reader is abstract — it also includes any DeterministicSingleTimeSeries (read into identical windows).

reader = store.build_forecast_reader(TimeSeriesType.Deterministic, timedelta(hours=1))
tl = reader.timeline()             # {"initial_timestamp", "resolution", "interval", "count", ...}
entries = reader.entries()         # list[TimeSeriesKey], parallel to entry_values
for ts in reader.timestamps():
    store.forecast_read(reader, ts)
    for i, key in enumerate(entries):
        window = reader.entry_values(i)   # window for key's owner

Shared forecasts are read once

Forecasts that share a backing array (deduplicated identical data, or several DeterministicSingleTimeSeries over one SingleTimeSeries) collapse to a single window slot. forecast_read reads each slot from the .nc file once per timestamp, so a forecast shared by 10 components costs one read, not ten. reader.num_slots() is the physical read count, and reader.entry_slot(i) says which slot an entry uses — group by slot to materialize each unique window only once on the Python side too:

store.forecast_read(reader, ts)
windows: dict[int, np.ndarray] = {}
for i, key in enumerate(entries):
    window = windows.setdefault(reader.entry_slot(i), reader.entry_values(i))

Query Metadata

list_time_series returns a list of plain dicts, filtered by any combination of arguments (the features argument is a subset match):

for m in store.list_time_series(
    owner_id=42,
    owner_category=OwnerCategory.Component,
    time_series_type=TimeSeriesType.SingleTimeSeries,
):
    print(m["name"], m["resolution"], m["units"], m["features"])

# The owner is the (owner_id, owner_category) pair.
keys = store.get_time_series_keys(42, OwnerCategory.Component)
exists = store.has_time_series(key)
resolutions = store.get_resolutions()          # list[str] (ISO 8601 durations)
counts = store.get_time_series_counts()        # dict

Remove and Maintain

store.remove_time_series(key)
# The owner is the (owner_id, owner_category) pair.
n = store.clear_time_series(42, OwnerCategory.Component)   # all series for one owner; returns count
store.clear_time_series()                                  # remove everything

# Reassign every series from one owner to another; returns the number moved.
moved = store.replace_owner(42, 43, OwnerCategory.Component)

report = store.compact()            # {"slots_reclaimed": ..., "datasets_dropped": ...,
                                   #  "feature_sets_reclaimed": ...}
integrity = store.verify_integrity()   # {"ok": True, "errors": []} when arrays are intact
                                   # (arrays only — the SQLite catalog is not checked)

Associations

Two catalog tables record relationships between entities the store does not otherwise model, wholly independently of time series: which supplemental attributes are attached to which components, and directed parent/child edges between components. Removing a time series never touches either, and vice versa — see Associations Between Entities.

Filter arguments are keyword-only, all optional, and ANDed; passing none matches everything.

from infrastore import (
    SupplementalAttributeAssociation,
    ParentChildAssociation,
    DuplicateAssociationError,
)

store.add_supplemental_attribute_association(
    SupplementalAttributeAssociation(42, "Generator", 100, "GeographicInfo")
)

# Bulk add is one all-or-nothing transaction.
store.add_supplemental_attribute_associations([
    SupplementalAttributeAssociation(43, "Generator", 100, "GeographicInfo"),
    SupplementalAttributeAssociation(43, "Generator", 101, "Outage"),
])

# Queries run in both directions, returning distinct ids in ascending order.
assert store.list_supplemental_attribute_ids(component_id=43) == [100, 101]
assert store.list_components_with_attributes(attribute_id=100) == [42, 43]
assert store.has_supplemental_attribute_association(component_id=42, attribute_id=100)

# `*_types` filters take CONCRETE type names. Expanding an abstract type into its
# subtypes is the caller's job — the store has no type hierarchy. An empty list is a
# deliberate "none of these" and matches nothing.
assert store.list_supplemental_attribute_ids(
    component_id=43, attribute_types=["Outage"]
) == [101]

assert store.count_supplemental_attributes() == 2        # distinct attributes
assert store.count_components_with_attributes() == 2     # distinct components
store.supplemental_attribute_counts_by_type()
# [('GeographicInfo', 2), ('Outage', 1)]
store.supplemental_attribute_summary()
# [{'component_type': 'Generator', 'attribute_type': 'GeographicInfo', 'count': 2}, ...]

Identity is the (component_id, attribute_id) pair. The type names ride along for filtering and are not part of it, so re-attaching the same pair under different type names is still a duplicate:

try:
    store.add_supplemental_attribute_association(
        SupplementalAttributeAssociation(42, "Load", 100, "Outage")
    )
except DuplicateAssociationError as e:
    print(e)   # attribute 100 is already attached to component 42

# Removal returns a count. Matching nothing returns 0 rather than raising, so assert on
# the count yourself if you expected a hit.
assert store.remove_supplemental_attribute_associations(component_id=43) == 2

Parent/child edges work the same way, except that identity is the ordered pair — the reverse of an edge is a different edge — and both endpoints are always components:

store.add_parent_child_association(ParentChildAssociation(42, "Generator", 7, "Bus"))
store.add_parent_child_associations([ParentChildAssociation(43, "Generator", 7, "Bus")])

assert store.list_children(parent_id=42) == [7]
assert store.list_parents(child_id=7) == [42, 43]
assert store.count_parent_child_associations() == 2

# Renumbering a component rewrites both ends of every edge.
assert store.replace_parent_child_component_id(42, 99) == 1
assert store.list_parents(child_id=7) == [43, 99]

Neither table is reachable over gRPC or the infrastore CLI.

Persist to Disk

store.flush()   # sync buffered writes; afterwards system.nc + system.nc.sqlite can be copied

Keep the two files together — the .nc and .nc.sqlite pair is a single logical store.

Error Handling

The store's own exceptions inherit from TimeSeriesError, so you can catch broadly or narrowly:

from infrastore import NotFoundError, DuplicateTimeSeriesError, TimeSeriesError

try:
    store.add_time_series(...)
except DuplicateTimeSeriesError:
    ...                       # key already exists
except TimeSeriesError as e:
    ...                       # anything else from the store

Argument parsing is the exception to that rule: a period argument (resolution, horizon, interval) that is a malformed ISO 8601 duration string raises a plain ValueError, and one that is neither a timedelta nor a str raises a plain TypeError. Neither is a TimeSeriesError, so except TimeSeriesError will not catch them.

One gotcha: because Python's bool is a subclass of int, the binding deliberately checks bool first, so True/False feature values are stored as booleans (not as 1/0 integers).

A Complete Round-Trip

from datetime import datetime, timedelta, timezone
import numpy as np
from infrastore import Store, SingleTimeSeries, OwnerCategory

store = Store.create(in_memory=True)
ts = SingleTimeSeries(
    datetime(2024, 1, 1, tzinfo=timezone.utc),
    timedelta(hours=1),
    np.arange(24, dtype=np.float64) + 100,
    "load",
)
key = store.add_time_series(
    owner_id=42, owner_type="Generator",
    owner_category=OwnerCategory.Component,
    time_series=ts,
    features={"model_year": 2030}, units="MW",
)
got = store.get_time_series(key)
assert got.name == "load"
assert np.array_equal(np.asarray(got.data), np.asarray(ts.data))

Diagnostics and tracing

The store emits structured tracing spans for every significant operation. To see them, initialize a subscriber before your first store call.

Via environment variable — set RUST_LOG before starting Python. The module auto-initializes a subscriber on import when this variable is set:

RUST_LOG=debug python myscript.py
# or, to limit output to the store core only:
RUST_LOG=infrastore_core=debug python myscript.py

Programmatically — call init_tracing with a filter directive string:

from infrastore import init_tracing

init_tracing("infrastore_core=debug")

store = Store.create(in_memory=True)
store.add_time_series(...)   # spans appear on stderr

init_tracing is a no-op if a subscriber is already registered (including the automatic one from RUST_LOG). The filter syntax is the same as RUST_LOG: comma-separated target=level pairs, or a bare level such as "debug" to match everything. Useful targets:

Julia Developer Guide

This guide covers building on InfraStore.jl, the Julia package that wraps the C ABI. For exact signatures see the Julia API reference; to set up the package and the native library, see Integrate with Julia.

Load the Package

InfraStore.jl resolves the native library from the INFRASTORE_LIB environment variable (or the InfraStore_jll package when installed). Build the cdylib and point at it before using InfraStore:

cargo build -p infrastore-ffi --release
export INFRASTORE_LIB=$PWD/target/release/libinfrastore_ffi.dylib  # .so on Linux

using Dates, InfraStore

Exported names include Store, SingleTimeSeries, NonSequentialTimeSeries, the forecast structs (Deterministic, Probabilistic, Scenarios), OwnerCategory (Component, SupplementalAttribute), the add_time_series! / get_time_series / get_metadata family, and transform_single_time_series!. The store type is named Store.

Open or Create a Store

# In-memory.
store = Store(in_memory=true)

# On disk: writes system.nc and system.nc.sqlite.
store = Store(in_memory=false, path="system.nc")

# Reopen read-only.
store = open_store("system.nc"; read_only=true)

The store is finalized automatically, but you can release it eagerly with close!(store).

Add a Series

# `name` ("load") is a required field on the struct.
ts = SingleTimeSeries(DateTime(2024, 1, 1), Hour(1), collect(100.0:123.0), "load")

key = add_time_series!(
    store,
    42,
    "Generator",
    Component,
    ts;                                   # name comes from ts
    features = Dict("model_year" => 2030),
    units = "MW",
)

Notes:

• owner_id is an integer (Int64) — the component identifier, e.g. 42.
• resolution is a Period such as Hour(1) or Minute(5).
• features is a Dict serialized to JSON, so values must be JSON scalars (Int, Float64, Bool, String). String features are supported and round-trip unchanged.
• Adding a duplicate key throws DuplicateTimeSeriesError.

add_time_series! returns a TimeSeriesKey holding an opaque handle into the store.

Read a Series

got = get_time_series(store, key)
@assert got.data == ts.data
println(got.initial_timestamp, " ", got.resolution)   # resolution comes back as Millisecond

To read many whole series at once — e.g. loading everything for a plot — bulk_read takes a vector of keys and returns the SingleTimeSeries in the same order, reading each packed dataset's column span once instead of re-reading every chunk per series:

series = bulk_read(store, keys)   # keys :: Vector{TimeSeriesKey}, all SingleTimeSeries

Attribute-Based Lookups

Beyond key handles, InfraStore.jl can resolve a series directly from its attributes — convenient when a caller keeps its own identifiers (as an InfrastructureSystems.jl-side store does):

meta = get_metadata(
    store,
    42,
    Component,        # owner_category; the owner is the (owner_id, owner_category) pair
    "load";
    resolution = Hour(1),
    features = Dict("model_year" => 2030),
)
# meta :: (initial_timestamp::DateTime, resolution::Millisecond, length::Int,
#          data_hash::Vector{UInt8}, dtype, ext)

values = get_array_by_hash(store, meta.data_hash)     # Vector{Float64}; pass ::Type{T} for other dtypes

# get_time_series itself resolves by attributes too (pass the type as the first argument):
got = get_time_series(SingleTimeSeries, store, 42, Component, "load"; resolution = Hour(1))

present = has_time_series(store, 42, Component, "load"; resolution = Hour(1))
remove_time_series!(store, 42, Component, "load"; resolution = Hour(1))

get_time_series, has_time_series, and remove_time_series! all accept either a TimeSeriesKey or (owner_id, owner_category, name; resolution, features) attributes — the conventions are interchangeable for every time series type, static or forecast.

Forecasts

InfraStore.jl exposes Deterministic, Probabilistic, and Scenarios structs that wrap a native AbstractArray in the type's logical shape (the wrapper derives the dtype and dims and serializes the buffer row-major). Construct one and add it through the generic add_time_series!:

data = zeros(Float64, 24, 7)   # (horizon_count, count)
fc = Deterministic(DateTime(2024, 1, 1), Hour(1), Hour(24), Hour(24), 7, data, "load_fc")
key = add_time_series!(
    store,
    42,
    "Generator",
    Component,
    fc;                         # name comes from fc
    units = "MW",
)

got = get_time_series(Deterministic, store, 42, Component, "load_fc"; resolution = Hour(1))
values = got.data   # Float64 matrix, shape (24, 7)

# Same forecast, read by the key returned from add_time_series! — forecasts and
# static series both support the key-based and attribute-based conventions.
got_by_key = get_time_series(Deterministic, store, key)

Probabilistic(initial_timestamp, resolution, horizon, interval, count, percentiles, data, name) carries the percentile vector, and Scenarios(initial_timestamp, resolution, horizon, interval, count, data, name) takes scenario_count from data's leading axis. Every forecast constructor also accepts a ext= keyword. Read the corresponding type back with get_time_series(Probabilistic, …) / get_time_series(Scenarios, …).

If two forecasts of one owner/name/type differ only by interval (say day-ahead and intra-day), pass interval= to pin the one you want; without it the read is ambiguous and throws InvalidParameterError:

got = get_time_series(Deterministic, store, 42, Component, "load_fc";
                      resolution = Hour(1), interval = Hour(6))

A DeterministicSingleTimeSeries is not added directly — derive one from the stored SingleTimeSeries with transform_single_time_series!, which returns the number transformed. It optionally restricts the transform to one owner_category and/or one resolution:

n = transform_single_time_series!(store, Hour(24), Hour(24))
n = transform_single_time_series!(store, Hour(24), Hour(24);
                                  owner_category = Component, resolution = Hour(1))

Requesting the concrete Deterministic type does not match a transformed DeterministicSingleTimeSeries — that read throws NotFoundError. To read whichever of the two is stored under an identity, request the family type AbstractDeterministic (it returns a Deterministic, since a DST has no materialized struct):

fc = get_time_series(AbstractDeterministic, store, 42, Component, "load")

A genuine miss still throws NotFoundError, and an identity that holds both concrete types is ambiguous and errors — request a concrete type then.

transform_single_time_series! returns no keys, so to read a derived forecast by key, enumerate the owner's keys with get_time_series_keys(store, owner_id, owner_category) (the owner is the (owner_id, owner_category) pair) and use key_info, whose time_series_type is the actual Julia type — pass it straight to get_time_series (a DeterministicSingleTimeSeries reads back as a Deterministic):

for k in get_time_series_keys(store, 42, Component)
    info = key_info(k)
    series = get_time_series(info.time_series_type, store, k)
end

has_typed, remove_typed!, and copy_time_series! address a series by its ts_type integer code (and take the same resolution / interval / features keywords). copy_time_series! re-points a stored series at another owner without duplicating data — it writes one association row against the same content-addressed array, preserving the stored type (a DST stays a DST):

copy_time_series!(store, 42, Component, "load", 0, 43, "Generator")   # ts_type 0 = SingleTimeSeries

The low-level get_metadata + get_array_by_hash path is still available for raw access. See the Julia API reference.

Per-Timestamp Reads (Simulation Loop)

get_time_series hands back a whole series or forecast. Simulations instead walk the timeline and, at each timestamp, want the value of every series at that instant. For that, build a reader once and drive it in a loop — it pins one resolution and reuses its output buffers, so the loop allocates almost nothing. StaticReader serves SingleTimeSeries; ForecastReader serves forecasts. (Full signatures: Julia API reference.)

Static series

reader = build_static_reader(store; resolution = Hour(1))
grid = static_grid(reader)                 # (initial_timestamp, resolution, length)
for k in 0:(grid.length - 1)
    static_read!(reader, grid.initial_timestamp + grid.resolution * k)
    for (gi, g) in enumerate(static_groups(reader))
        vals = static_values(reader, gi)   # (num_columns, element_dims...); column j ↔ g.keys[j]
    end
end

Series are grouped by (dtype, element_shape); each group's static_values is one dense array whose columns line up with the group's keys. All matched series must share one grid (initial_timestamp + length), validated at build.

Forecasts

reader = build_forecast_reader(store, Deterministic; resolution = Hour(1))
tl = forecast_timeline(reader)             # (initial_timestamp, resolution, interval, count)
for k in 0:(tl.count - 1)
    forecast_read!(reader, tl.initial_timestamp + tl.interval * k)
    for (i, e) in enumerate(forecast_entries(reader))
        window = forecast_values(reader, i)   # shape e.window_shape, for e.key
    end
end

A Deterministic reader is abstract — it also includes any DeterministicSingleTimeSeries (read into identical windows).

Shared forecasts are read once

Forecasts that share a backing array (deduplicated identical data, or several DeterministicSingleTimeSeries over one SingleTimeSeries) collapse to a single window slot. forecast_read! reads each slot from the .nc file once per timestamp, so a forecast shared by 10 components costs one read, not ten. forecast_num_slots(reader) is the physical read count, and each ForecastEntry.slot says which slot an entry uses — group by slot to materialize each unique window only once:

forecast_read!(reader, t)
windows = Dict{Int, Any}()
for (i, e) in enumerate(forecast_entries(reader))
    w = get!(() -> forecast_values(reader, i), windows, e.slot)
    # apply w to e.key's owner
end

Store-Wide Operations

counts = get_counts(store)        # (components_with_time_series, static_time_series, forecasts)
nerr   = verify_integrity(store)  # 0 == arrays intact (catalog not checked)
compact!(store)

Associations

Two catalog tables record relationships between entities the store does not otherwise model, wholly independently of time series: which supplemental attributes are attached to which components, and directed parent/child edges between components. Removing a time series never touches either, and vice versa — see Associations Between Entities.

Filter keywords are all optional and ANDed; passing none matches everything.

add_supplemental_attribute_association!(
    store, SupplementalAttributeAssociation(42, "Generator", 100, "GeographicInfo"))

# Bulk add is one all-or-nothing transaction.
add_supplemental_attribute_associations!(store, [
    SupplementalAttributeAssociation(43, "Generator", 100, "GeographicInfo"),
    SupplementalAttributeAssociation(43, "Generator", 101, "Outage"),
])

# Queries run in both directions, returning distinct ids in ascending order.
list_supplemental_attribute_ids(store; component_id=43)      # [100, 101]
list_components_with_attributes(store; attribute_id=100)     # [42, 43]
has_supplemental_attribute_association(store; component_id=42, attribute_id=100)  # true

# `*_types` filters take CONCRETE type names, so expand an abstract type yourself —
# `get_all_subtype_names` in InfrastructureSystems.jl is the usual source. An empty
# vector is a deliberate "none of these" and matches nothing.
list_supplemental_attribute_ids(store; component_id=43, attribute_types=["Outage"])  # [101]

count_supplemental_attributes(store)         # 2, distinct attributes
count_components_with_attributes(store)      # 2, distinct components
supplemental_attribute_counts_by_type(store)
# [(type = "GeographicInfo", count = 2), (type = "Outage", count = 1)]
supplemental_attribute_summary(store)
# [(component_type = "Generator", attribute_type = "GeographicInfo", count = 2), ...]

Identity is the (component_id, attribute_id) pair. The type names ride along for filtering and are not part of it, so re-attaching the same pair under different type names is still a duplicate:

try
    add_supplemental_attribute_association!(
        store, SupplementalAttributeAssociation(42, "Load", 100, "Outage"))
catch e
    e isa InfraStore.DuplicateAssociationError || rethrow()
    @info e.msg   # attribute 100 is already attached to component 42
end

# Removal returns a count. Matching nothing returns 0 rather than throwing, so assert on
# the count yourself if you expected a hit.
remove_supplemental_attribute_associations!(store; component_id=43)   # 2

Parent/child edges work the same way, except that identity is the ordered pair — the reverse of an edge is a different edge — and both endpoints are always components:

add_parent_child_association!(store, ParentChildAssociation(42, "Generator", 7, "Bus"))
add_parent_child_associations!(store, [ParentChildAssociation(43, "Generator", 7, "Bus")])

list_children(store; parent_id=42)      # [7]
list_parents(store; child_id=7)         # [42, 43]
count_parent_child_associations(store)  # 2

# Renumbering a component rewrites both ends of every edge.
replace_parent_child_component_id!(store, 42, 99)   # 1
list_parents(store; child_id=7)                     # [43, 99]

Neither table is reachable over gRPC or the infrastore CLI.

Persist to Disk

flush!(store)   # sync NetCDF + SQLite; afterwards system.nc + system.nc.sqlite can be copied

Keep the .nc and .nc.sqlite files together.

Error Handling

Errors subtype InfraStore.TimeSeriesException. The exception types are not exported, so reference them module-qualified. Catch broadly or narrowly:

try
    add_time_series!(store, 42, "Generator", Component, ts)
catch e
    if e isa InfraStore.DuplicateTimeSeriesError
        @warn "already present"
    else
        rethrow()
    end
end

The available types are InfraStore.NotFoundError, InfraStore.DuplicateTimeSeriesError, InfraStore.InvalidParameterError, InfraStore.IntegrityError, InfraStore.ReadOnlyStoreError, InfraStore.IncompatibleFormatError (the on-disk store was written by an incompatible data format version), and InfraStore.GenericError (which carries the raw FFI status code).

InfrastructureSystems.jl Integration Notes

The model is designed to back an InfrastructureSystems.jl time-series store:

• Owners are integer component identifiers (Int64), matching InfrastructureSystems.jl component/attribute IDs.
• OwnerCategory distinguishes Component from SupplementalAttribute and is part of the owner identity: the owner is the (owner_id, owner_category) pair, so a component and a supplemental attribute may share a numeric id and stay distinct. Owner-scoped calls take the category alongside the id.
• The attribute-based accessors (get_metadata, has_time_series, remove_time_series!) plus get_array_by_hash let an InfrastructureSystems.jl-side store keep its own key objects and reach the array layer without holding a TimeSeriesKey.
• For the simulation read pattern — iterate every component's value at each timestamp, reading a forecast shared across components only once — use the readers (Per-Timestamp Reads). The ForecastEntry.slot / forecast_num_slots surface lets the wrapping store dedup its own per-component work, mirroring the store's one-read-per-shared-array behavior.

See Language Bindings for how this maps onto the FFI.

Diagnostics and tracing

The store emits structured tracing spans for every significant operation. To see them, initialize a subscriber before your first store call.

Via environment variable — set RUST_LOG before loading the package. The module's __init__ hook calls init_logging("") automatically, which reads RUST_LOG if set:

# shell
export RUST_LOG=infrastore_core=debug
julia --project=. myscript.jl

Programmatically — call init_logging with a filter directive string:

using InfraStore

init_logging("infrastore_core=debug")

store = Store(in_memory=true)
add_time_series!(store, ...)   # spans appear on stderr

init_logging is a no-op if a subscriber is already registered (including the automatic one from RUST_LOG). The filter syntax is the same as RUST_LOG: comma-separated target=level pairs, or a bare level such as "debug" to match everything. Useful targets:

gRPC Server & Client Guide

The gRPC server exposes a store for remote, read-only access. Writes always require local filesystem access, so the service offers only list/get/keys/resolutions/counts/exists/verify. This guide covers running the server and talking to it from Rust. For the wire contract see the gRPC API reference; for the config file see Server Configuration.

When to Use It

Use the server to fan out reads of an existing store to many clients or across the network — for example, serving a published dataset to analysis jobs. A single writer produces the .nc + .nc.sqlite pair locally; the server then reads it and answers queries. It never modifies the files.

Run the Server

1. Produce a store with any binding and flush() it to disk.

2. Write a config (start from examples/server.toml):

   [server]
   host = "0.0.0.0"
   port = 50051
   
   [data]
   files = ["./system.nc"]   # the .nc.sqlite catalog must sit beside it
   
   [authentication]
   method = "none"
   

3. Launch:

   cargo run -p infrastore-server -- --config my_server.toml
   # or, from a release build:
   ./target/release/infrastore-server --config my_server.toml
   

On startup the server validates the auth section, opens the first [data].files entry read-only, and serves the CatalogStore service on host:port. Set RUST_LOG=debug for verbose logs.

The Rust Client

RemoteClient mirrors the read methods of Store and returns the same core types. gRPC status codes are mapped back onto TimeSeriesError, so remote and local calls surface the same error taxonomy.

use infrastore_core::OwnerCategory;
use infrastore_server::client::RemoteClient;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = RemoteClient::connect("http://127.0.0.1:50051".into()).await?;

    let counts = client.get_counts().await?;
    println!("{} static series", counts.static_time_series);

    // The owner is the (owner_id, owner_category) pair.
    let keys = client.get_time_series_keys(42, OwnerCategory::Component).await?;
    if let Some(key) = keys.first() {
        let data = client.get_time_series(key, None).await?;
        println!("read {} values", data.as_single().unwrap().length);
    }
    Ok(())
}

Available methods: connect, from_channel, list_time_series, get_time_series, get_time_series_keys, get_resolutions, get_counts, get_forecast_parameters, has_time_series, verify_integrity.

Authentication

To require an API key, configure the server:

[authentication]
method = "api_key"
keys = ["replace-me-with-a-secret-1", "replace-me-with-a-secret-2"]

method = "api_key" with an empty keys list is rejected at startup. Clients must then send the key in the x-api-key metadata header; a missing or wrong key is rejected with Unauthenticated before the RPC runs. The comparison against the configured keys does not early-exit — every key of the same length as the supplied one is checked — so which key matched is not leaked by timing. The supplied key's length is not blinded; keys of a different length are rejected without a byte-wise compare, on the assumption that length is not secret.

RemoteClient::connect does not attach auth metadata, so against an authenticated server use the generated client with an interceptor that injects the header:

#![allow(unused)]
fn main() {
use infrastore_proto::pb::catalog_store_client::CatalogStoreClient;
use infrastore_proto::pb::CountsReq;
use tonic::metadata::MetadataValue;
use tonic::transport::Channel;

let channel = Channel::from_shared("http://127.0.0.1:50051")?.connect().await?;
let key: MetadataValue<_> = "replace-me-with-a-secret-1".parse()?;
let mut client = CatalogStoreClient::with_interceptor(channel, move |mut req: tonic::Request<()>| {
    req.metadata_mut().insert("x-api-key", key.clone());
    Ok(req)
});

let counts = client.get_counts(CountsReq {}).await?.into_inner();
}

Clients in Other Languages

The proto file at proto/infrastore/v1/store.proto is a standard proto3 definition. Generate a client for any gRPC-supported language from it, sending the x-api-key metadata header when the server requires authentication.

Benchmarks

infrastore-bench is a standalone binary (infrastore-bench) for measuring two critical performance dimensions: bulk ingestion and simulation-loop reads.

Build

# Debug build (fast to compile, slower to run)
cargo build -p infrastore-bench

# Release build (recommended for any real measurement)
cargo build --release -p infrastore-bench

The binary is placed at target/release/infrastore-bench.

Subcommands

Common flags

The read and all subcommands also accept --steps T (default: --length) to control how many simulation timesteps are benchmarked.

Add benchmark

# 10 000 components, 1 week hourly, in-memory
infrastore-bench add --count 10000 --length 168 --in-memory

# 100 000 components, same length, on-disk (tests real I/O + OS page cache)
infrastore-bench add --count 100000 --length 168

Reports for both SingleTimeSeries and Deterministic:

• Build requests — time to construct AddRequest objects in memory (array allocation only; excluded from the add throughput measurement).
• add_time_series_bulk — wall time, items/s, and data MB/s.

SingleTimeSeries arrays are column-packed into NetCDF datasets of up to 1 000 columns each and wrapped in a single SQLite transaction; the add benchmark stresses both. Deterministic arrays are standalone NetCDF variables; their add cost is dominated by per-variable write overhead and the same single-transaction SQLite commit.

Read benchmark

# 1 000 components, 168 timesteps, in-memory (pure CPU / metadata cost)
infrastore-bench read --count 1000 --length 168 --in-memory

# Same, but on-disk — store is written, dropped, then reopened read-only
infrastore-bench read --count 1000 --length 168

# Benchmark only the first 24 simulation steps of a 168-step series
infrastore-bench read --count 10000 --length 168 --steps 24

The read benchmark simulates the access pattern of an energy-simulation step loop:

for t in 0..T:
    for each component key:
        store.get_time_series(key, time_range=(t₀ + t·Δt, t₀ + (t+1)·Δt))

For on-disk stores the binary flushes, drops, and reopens the store read-only before the timed loop. This rebuilds the NetCDF in-memory index and starts the HDF5 chunk cache cold, reflecting real simulation startup conditions. The OS page cache may still be warm.

Reports per-step min / median / p95 / max and total component-reads/s.

Deterministic note. The current get_time_series implementation reads the full [H × count] array from storage and then slices to the requested window in memory. For large count (e.g. 168 windows) each single-step read transfers H × count × 8 bytes, which is 24 × 168 × 8 = 32 KB per component. The benchmark output flags this explicitly so the overhead is visible.

Interpreting results

The two metrics to watch:

• add throughput drops sharply when the number of items exceeds a SQLite transaction or NetCDF dataset threshold. If adding 100 000 items is notably slower per-item than adding 10 000, the bottleneck is likely the SQLite commit or NetCDF file growth, not array construction.

• Per-step time for the read benchmark scales linearly with --count. If the cost per step is much higher for on-disk than in-memory, the bottleneck is HDF5 chunk reads or SQLite metadata queries rather than Rust overhead. The packed SingleTimeSeries layout chunks (1, cols) — one timestamp across every column — so reading one timestamp across many series is a single chunk read, while reading one full series touches every chunk band (the slow direction, expected for exploration/plotting rather than hot loops).

Tracing spans for deeper diagnosis

When the benchmark numbers show a problem but don't tell you which layer is slow, enable the built-in tracing spans with --log-level:

# Show all debug-level spans from the store core only (least noise)
infrastore-bench --log-level infrastore_core=debug add --count 100 --in-memory

# Show everything — useful when the bottleneck might be in infrastore-bench itself
infrastore-bench --log-level debug add --count 100

RUST_LOG is also accepted as a fallback when --log-level is not provided.

The key spans emitted by infrastore-core:

Spans nest: a single add_time_series_bulk call groups packed series by shape and emits one put_packed_block span per group (filling whole chunks), plus a put_array → put_standalone span per standalone item; a single add_time_series call instead emits one put_array → put_packed span. This makes it straightforward to see whether time is spent in metadata insertion, NetCDF I/O, or the debug_span overhead itself.

The infrastore CLI supports the same --log-level flag for diagnosing a live store.

How-To Guides

Task-oriented recipes. Each page solves one problem in the fewest steps and links out to the guides and reference for depth.

• Install the Native Library — Build the cdylib and system dependencies.
• Integrate with Python — Get the infrastore wheel into your project.
• Integrate with Julia — Wire InfraStore.jl to the native library.
• Run the gRPC Server — Serve a store for remote readers.
• Use the infrastore CLI — Load and inspect a store from the command line.

Install the Native Library

This recipe builds the components other languages depend on. For the full prerequisites and workspace build, see Installation.

1. Install System Libraries

# macOS
brew install hdf5 netcdf protobuf

# Debian / Ubuntu
sudo apt-get install libhdf5-dev libnetcdf-dev protobuf-compiler

If the build fails with Unable to locate HDF5 root directory and/or headers, set HDF5_DIR:

export HDF5_DIR="$(brew --prefix hdf5)"          # macOS
export HDF5_DIR=/usr/lib/x86_64-linux-gnu/hdf5/serial  # Linux

2. Build What You Need

The C ABI cdylib (used by Julia and any C consumer):

cargo build -p infrastore-ffi --release
# -> target/release/libinfrastore_ffi.{dylib,so,dll}
# Regenerates the C header at crates/infrastore-ffi/include/infrastore.h

The whole workspace (core, server, proto, ffi):

cargo build --workspace
cargo test --workspace

The Python wheel is built separately with maturin — see Integrate with Python.

3. Point Consumers at the Library

The Julia binding and other C consumers locate the cdylib via an environment variable:

export INFRASTORE_LIB=$PWD/target/release/libinfrastore_ffi.dylib  # .so on Linux

Add it to your shell profile to make it permanent.

Next

• Integrate with Python
• Integrate with Julia

Integrate with Python

Get the infrastore module into a Python environment. For API usage once it imports, see the Python Developer Guide.

Prerequisites

• Python 3.10 or newer.
• The system libraries (HDF5, NetCDF, Protobuf).

Build and Install the Wheel (development)

The binding is built with maturin. maturin develop compiles the extension and installs it into the active virtual environment:

cd crates/infrastore-py
python3 -m venv .venv && source .venv/bin/activate
pip install maturin pytest numpy
maturin develop

Verify the install:

python -c "import infrastore; print(infrastore.__version__)"
pytest ../../python/tests

Build a Distributable Wheel

To produce a wheel you can install elsewhere:

cd crates/infrastore-py
maturin build --release
# -> target/wheels/infrastore-<version>-cp310-abi3-<platform>.whl
pip install ../../target/wheels/infrastore-*.whl

The wheel is built against the abi3-py310 stable ABI, so a single wheel works on CPython 3.10 and every newer 3.x without recompiling.

Smoke Test

from datetime import datetime, timedelta, timezone
import numpy as np
from infrastore import Store, SingleTimeSeries, OwnerCategory

store = Store.create(in_memory=True)
ts = SingleTimeSeries(
    datetime(2024, 1, 1, tzinfo=timezone.utc),
    timedelta(hours=1),
    np.arange(24, dtype=np.float64) + 100,
    "load",
)
key = store.add_time_series(
    owner_id=42, owner_type="Generator",
    owner_category=OwnerCategory.Component,
    time_series=ts,
    features={"model_year": 2030}, units="MW",
)
assert np.array_equal(np.asarray(store.get_time_series(key).data), np.asarray(ts.data))
print("ok")

Troubleshooting

• ImportError for the extension — Ensure you ran maturin develop in the active venv, or that pip install-ed the wheel into the interpreter you are running.
• HDF5 not found during build — Set HDF5_DIR (see Install).
• InvalidParameterError on add — In Python, pass a NumPy array (any shape) whose dtype is one of float64, float32, int64, int32, uint64, or bool; any other dtype (e.g. complex128 or a string dtype) raises. Feature values must be int/float/bool/str. Timestamps for a NonSequentialTimeSeries must be strictly increasing.

Next

• Python Developer Guide
• Python API reference

Integrate with Julia

Wire InfraStore.jl to the native library. For API usage once it loads, see the Julia Developer Guide.

Prerequisites

• Julia 1.10 or newer.
• The system libraries (HDF5, NetCDF, Protobuf).

1. Build the Native Library

InfraStore.jl calls into the C ABI cdylib, so build it first:

cargo build -p infrastore-ffi --release

2. Point Julia at the Library

InfraStore.jl resolves the cdylib at first use, in this order:

1. The INFRASTORE_LIB environment variable — the development override, pointing at a build from step 1.
2. The InfraStore_jll binary package, if it is installed in the active environment.

For a development build, export the variable (add it to your shell profile to make it permanent):

export INFRASTORE_LIB=$PWD/target/release/libinfrastore_ffi.dylib  # .so on Linux

using InfraStore always works; the resolution happens on the first call that reaches the native library. If neither source yields a path, that call errors (see Troubleshooting). With InfraStore_jll installed you can skip the export entirely — set INFRASTORE_LIB only when you want your local build to win over the JLL.

3. Instantiate and Test the Package

julia --project=julia/InfraStore.jl -e 'using Pkg; Pkg.instantiate()'
julia --project=julia/InfraStore.jl julia/InfraStore.jl/test/runtests.jl

4. Use It From Your Project

Develop the package into your own environment, then activate it with the library variable set:

using Pkg
Pkg.develop(path="/path/to/infrastore/julia/InfraStore.jl")

Smoke Test

using Dates, InfraStore

store = Store(in_memory=true)
ts = SingleTimeSeries(DateTime(2024, 1, 1), Hour(1), collect(100.0:123.0), "load")
key = add_time_series!(
    store,
    42,
    "Generator",
    Component,
    ts;
    features=Dict("model_year" => 2030),
    units="MW",
)
got = get_time_series(store, key)
@assert got.data == ts.data
@assert got.name == "load"
println("ok")

Troubleshooting

• Could not locate libinfrastore_ffi. Set the INFRASTORE_LIB environment variable to a built cdylib, or install InfraStore_jll. — Neither resolution path produced a library. Export the variable (step 2) before the first store call, in the same shell that launched Julia, or add InfraStore_jll to the environment.
• could not load library — Check the path exists and has the right extension for your OS (.dylib on macOS, .so on Linux, .dll on Windows), and that you built with --release if your variable points at target/release.
• InvalidParameterError on add — owner_id must be an integer (e.g. 42, an Int64), and features values must be JSON scalars.

Next

• Julia Developer Guide
• Julia API reference
• C ABI reference

Run the gRPC Server

Serve an existing store read-only over gRPC. For background and the Rust client, see the gRPC Server guide; for every config key, see Server Configuration.

1. Produce a Store on Disk

Create the .nc + .nc.sqlite pair with any binding, then flush() it. For example, in Rust:

#![allow(unused)]
fn main() {
let mut store = create_store(Some(Path::new("system.nc")), false)?;
// ... add_time_series ...
store.flush()?;
}

2. Write a Config

# my_server.toml
[server]
host = "0.0.0.0"
port = 50051

[data]
files = ["./system.nc"]   # the ./system.nc.sqlite catalog must sit beside it

[authentication]
method = "none"

3. Launch

cargo run -p infrastore-server -- --config my_server.toml
# or a release build:
./target/release/infrastore-server --config my_server.toml

Add RUST_LOG=debug for verbose logging.

Require an API Key

[authentication]
method = "api_key"
keys = ["replace-me-with-a-secret"]

Clients must then send the key in the x-api-key metadata header; a missing or wrong key is rejected with Unauthenticated. An empty keys list with method = "api_key" is rejected at startup.

Quick Connectivity Check

From Rust:

#![allow(unused)]
fn main() {
use infrastore_server::client::RemoteClient;

let client = RemoteClient::connect("http://127.0.0.1:50051".into()).await?;
println!("{:?}", client.get_counts().await?);
}

Or with grpcurl using the proto file:

grpcurl -plaintext -proto proto/infrastore/v1/store.proto \
  127.0.0.1:50051 infrastore.v1.CatalogStore/GetCounts

(Add -H 'x-api-key: replace-me-with-a-secret' when authentication is enabled.)

Notes

• The server is read-only; it never writes the files. Run one writer locally, then serve the result.
• v0 serves the first [data].files entry; multi-file serving is reserved for later.

Use the infrastore CLI

infrastore loads time series from CSV files and inspects a store, talking directly to the on-disk .nc + .nc.sqlite pair (no gRPC server required). For the full command and descriptor reference, see CLI Reference.

1. Build the Binary

cargo build -p infrastore-cli      # debug build at target/debug/infrastore
# or a release build:
cargo build -p infrastore-cli --release

The examples below assume infrastore is on your PATH (or use ./target/debug/infrastore).

2. Describe the Data

Numeric values live in a CSV; everything that does not fit a flat grid (owner, name, type, dtype, resolution, initial timestamp, units, features) lives in a descriptor JSON. Print a starting point for any type with template:

infrastore template single > load.json       # print an example descriptor to edit

Edit it to point at your data and metadata:

{
  "owner_id": 42,
  "owner_type": "Generator",
  "owner_category": "component",
  "name": "load",
  "type": "single",
  "dtype": "f64",
  "units": "MW",
  "csv": "load.csv",
  "has_header": true,
  "initial_timestamp": "2024-01-01T00:00:00Z",
  "resolution": "1h",
  "features": {
    "model_year": 2030
  }
}

# load.csv
value
100.0
101.5
103.0
104.2
102.8
101.0

The descriptor rejects unknown keys, so a typo (resolutionn) is a hard error rather than a silently ignored setting.

3. Add It to a Store

infrastore --store demo.nc add --descriptor load.json

The store (demo.nc and its demo.nc.sqlite catalog) is created on first add. A descriptor may also be a JSON array of objects to add many series in one transaction. --csv overrides the descriptor's csv path, but only for a single-series descriptor: with an array of two or more objects it errors (--csv cannot be used with an array descriptor).

4. Read It Back

infrastore follows an output convention: a global -f/--format with table (default), json, and csv. Only the read commands (list, get, info) honor it; add, remove, transform, and template accept the flag but ignore it and print plain text.

infrastore --store demo.nc list                                       # what's in the store
infrastore --store demo.nc list --name-glob 'load_*'                  # name pattern (SQLite GLOB)
infrastore --store demo.nc get  --owner-id 42 --name load             # pretty table
infrastore --store demo.nc -f csv  get  --owner-id 42 --name load     # round-trippable CSV
infrastore --store demo.nc -f json info --owner-id 42 --name load     # metadata + stats
infrastore --store demo.nc -f csv  export --dir out/                  # one file per series

export is the bulk read-direction inverse of add: every series the selector matches is written to its own CSV or JSON file under --dir (or to stdout when exactly one matches). Setting INFRASTORE_STORE in the environment stands in for --store, and destructive commands (remove, clear, replace-owner, rename, copy) accept --dry-run to preview their effect.

info reports metadata plus stats over the values: min/max/mean for numeric dtypes, or true_count/false_count when dtype is bool, and always num_elements.

get/info/remove select a single series with --owner-id, --owner-category, --name, --type, --resolution, and repeated --feature key=value (--feature is the only repeatable one); if more than one series matches, infrastore lists the candidates so you can narrow the query. The owner is the (owner_id, owner_category) pair, so a component and a supplemental attribute may share a numeric id — add --owner-category (component / supplemental_attribute) to disambiguate. Large series truncate in table output — pass --limit N or --full.

--time-range START..END on get takes two timestamps (RFC3339 or epoch-ms), not a duration:

infrastore --store demo.nc get --owner-id 42 --name load \
  --time-range 2024-01-01T01:00:00Z..2024-01-01T03:00:00Z

Beware that inputs and outputs are spelled differently. You type the short, lowercase forms (--type single, --owner-category component), but list/get/info print the canonical CamelCase names (SingleTimeSeries, Component). Both spellings are accepted as input, so -f json list output can be fed back into a selector unchanged; just don't expect the rendered value to string-match what you typed.

5. Forecasts

All five writable types work (single, non_sequential, deterministic, probabilistic, scenarios). infrastore template deterministic prints a descriptor to edit, but it is plain JSON and says nothing about the data layout, so here is the rule:

Forecast CSVs are a flat, row-major stream of values with no structure of their own. The count must equal the product of the type's shape:

H = horizon / resolution — with the template's "horizon": "24h", "resolution": "1h", and "count": 7, a scalar deterministic needs exactly 24 * 7 = 168 values (plus the header row that has_header: true skips). Use -f json to read the flat values back at full fidelity. get -f csv on a forecast emits timestamped analysis rows instead — one row per (window, step) with issue_time/target_time columns and one value column per percentile or scenario — so it is not re-ingestible by add (static series' get -f csv still round-trips).

DeterministicSingleTimeSeries is not added from CSV — store a SingleTimeSeries, then derive it. transform takes no selector: it rewrites every SingleTimeSeries in the store. --horizon must fit inside each one (horizon / resolution steps must not exceed its length), so with the 6-row hourly load above, a 24-hour horizon fails and a 3-hour one works:

infrastore --store demo.nc transform --horizon 3h --interval 1h

The derived series keeps the source's owner, name, and resolution, so load now matches two entries and a bare selector becomes ambiguous. Disambiguate with --type:

infrastore --store demo.nc get --owner-id 42 --name load --type single
infrastore --store demo.nc get --owner-id 42 --name load --type deterministic_single

Notes

• The CLI writes locally; there is no remote/gRPC mode yet (store access is isolated so one can be added later).
• Output is colored (green table headers) only when stdout is a terminal; it is plain when piped/redirected or when NO_COLOR is set, so -f json/-f csv stay clean for other tools.
• --log-level (or RUST_LOG) controls logging; the default is quiet (warn).
• The .nc and .nc.sqlite files are one artifact — move, copy, and delete them together.

Reference

Information-oriented listings: exact signatures, schemas, and on-disk layouts. Reach for these pages when you already know what you want and need the precise shape of it. For narrative guidance, see the Developer Guides; for concepts, the Explanation section.

• On-Disk File Format — The NetCDF4 layout and SQLite schema, byte for byte.
• Rust API — infrastore-core public types and Store methods.
• Python API — The infrastore module.
• Julia API — The InfraStore.jl package.
• C ABI — The infrastore_ffi cdylib functions.
• gRPC API — The infrastore.v1 service.
• Server Configuration — The server TOML file.

On-Disk File Format

A persisted store is a pair of files that must be kept together:

<name>.nc          # NetCDF4 — numerical arrays
<name>.nc.sqlite   # SQLite  — metadata associations

The SQLite catalog path is the NetCDF path with .sqlite appended to the file name. This page is the authoritative description of both. For the rationale behind the split, see the Storage Model.

Format Version

The NetCDF root carries two global attributes:

data_format_version = "0.11.0"
compression         = "deflate:3:shuffle"

data_format_version is the semver of the on-disk format (DATA_FORMAT_VERSION). It is bumped when the NetCDF layout, the SQLite schema, or the hashing domain changes in a backward-incompatible way. A purely additive SQLite table does not qualify: it costs old readers nothing and old stores pick it up from the idempotent DDL on their first writable open. The associations table landed that way.

compression records the filter policy the store was created with, so that appends made after reopening reuse the same filter. It is not part of the compatibility contract — see Compression below.

Opening a store whose recorded version differs from the version this build reads fails with IncompatibleFormat, naming both versions. Every bump is backward-incompatible by definition, so the check is exact equality and there is no in-place upgrade path: regenerate the store with the matching build.

(0.11.0 renamed the metadata column logical_type to ext — an opaque, package-owned extension payload (typically JSON) the store stores verbatim and never interprets; 0.10.0 replaced the per-association features table with the content-addressed feature_sets table below, so a feature map is stored once and shared by every association that uses it — dropping the association_id foreign key and its ON DELETE CASCADE; 0.9.0 changed the packed-dataset chunking to timestamp-major (1, cols, *element_shape) and made the column count cols per-dataset (sized to the writing batch) instead of a fixed 1,000, optimizing reads across series by timestamp and bulk writes; 0.8.0 added the forecast interval to the association uniqueness key — so two forecasts of one variable that differ only by interval are now distinct series — widening both unique indexes (the NULL-folding index now COALESCEs interval as well as resolution); 0.7.0 made resolution/horizon/interval calendar-aware periods: they are now encoded as ISO-8601 duration strings (e.g. PT1H, P1M, P1Y) rather than integer milliseconds, in both the packed dataset names and the SQLite columns, so irregular periods (Month/Quarter/Year) can be represented distinctly from fixed spans; 0.6.0 added owner_category to the association uniqueness key (so the owner identity is the pair (owner_id, owner_category)), widening the unique indexes and idx_owner; 0.5.0 changed the owner identifier to a signed 64-bit integer (owner_id); 0.4.0 is the baseline the Rust port of InfrastructureSystems.jl shipped with — the version that introduced DATA_FORMAT_VERSION itself; 0.3.0 switched the time unit from nanoseconds to milliseconds, renaming the SQLite *_ns columns to *_ms and encoding the packed dataset name's {res} field in milliseconds instead of whole seconds; 0.2.0 introduced typed, multi-dimensional arrays and the two-mode NetCDF layout below; 0.1.0 stored only 1-D f64.)

Arrays Are Typed and N-Dimensional

Every stored array is a TypedArray: an element dtype, a shape [length, k1, k2, …] whose first axis is time and whose trailing axes are a fixed per-step element shape, and the raw row-major, little-endian element bytes. The supported dtypes and their stable integer codes (shared with the bindings and the C ABI):

A scalar-per-step series has an empty element shape; a per-step tuple (e.g. the 3 coefficients of a quadratic cost curve) has element shape [3].

NetCDF Layout

Arrays live under a two-level group hierarchy, in one of two storage modes:

<name>.nc
├── attribute  data_format_version = "0.11.0"
├── attribute  compression         = "deflate:3:shuffle"
└── group      time_series/
    └── group  single/
        ├── var  sts_{dtype}_{shape}_{length}_{res}      packed dataset  (length, cols, *element_shape)
        ├── var  sts_{dtype}_{shape}_{length}_{res}_h    str  (cols,)     # per-column hex hashes
        ├── var  sts_{dtype}_{shape}_{length}_{res}__1   packed spill dataset
        ├── var  arr_{hex_hash}                          standalone array  [length, *element_shape]
        └── ...

Every dimension is private to the variable that owns it and is named after that variable, so ncdump -h shows one set of dimensions per dataset:

• A packed dataset {dataset} is dimensioned ({dataset}_t, {dataset}_c, {dataset}_e0, …): {dataset}_t = length (the time axis), {dataset}_c = cols (the column axis), and one {dataset}_e{i} per trailing axis, sized element_shape[i]. A scalar-per-step dataset has no _e{i} dimensions.
• Its hash companion {dataset}_h is dimensioned on {dataset}_c alone — one hash slot per column.
• A standalone array arr_{hex_hash} is dimensioned arr_{hex_hash}_d{i}, one per axis of its full shape [length, *element_shape].

{dataset}_c is load-bearing, not decorative: on open the backend recovers each packed dataset's column count from the length of its second dimension, which is how per-dataset widths round-trip.

Packed mode

Used for SingleTimeSeries and the underlying array of a DeterministicSingleTimeSeries. Many arrays that share a (dtype, element_shape, length, resolution) are column-packed into one dataset:

The dataset shape is (length, cols, *element_shape) and chunking is (1, cols, *element_shape), so one HDF5 chunk holds a single timestamp across every column — making a read across series by timestamp one chunk, and a buffered bulk write fill whole chunks. cols is chosen per dataset: a managed bulk write sizes it to the batch, while an incremental one-at-a-time write path uses a default width (DEFAULT_COLS_PER_DATASET = 1000). In both cases cols is capped so one chunk stays within a byte budget (MAX_CHUNK_BYTES = 1 MiB); a batch wider than the cap spills across datasets.

• Rows are timesteps, columns are series. Column i holds one complete series.
• Hash companion variable. Each packed dataset has a sibling string variable {dataset}_h of shape (cols,). Slot i holds the lowercase hex SHA-256 (64 chars) of column i, or an empty string if the column is free. This is the on-disk index: on open, the backend scans every …_h, decodes the non-empty hashes, and rebuilds its hash → (dataset, column) map. (The backend also recovers each dataset's cols from its column dimension length, so per-dataset widths round-trip.)
• Spill. When a family's current dataset is full — a batch exceeds the column cap, or incremental writes fill a default-width dataset — the next write creates a spill dataset …__1, then …__2, and so on.

Standalone mode

Used for NonSequentialTimeSeries and the dense forecast arrays (Deterministic, Probabilistic, Scenarios). Each array is its own typed, multi-dimensional variable named arr_{hex_hash} in the time_series/single group. There is no column packing and no companion hash variable — the variable name carries the hash.

• NonSequentialTimeSeries is shaped [length, *element_shape] and chunked as a single whole-array chunk. It stores its explicit, strictly-increasing timestamps in the association's timestamps_json metadata field, not in the array.
• Dense forecasts are shaped [H, count, *element_shape] (Deterministic), [num_percentiles, H, count, *element_shape] (Probabilistic), or [num_scenarios, H, count, *element_shape] (Scenarios), where count is the number of forecast windows. They are chunked in bounded blocks along the count (window) axis — full on every other axis, cols windows wide, where cols is the largest count keeping one chunk within the same 1 MiB budget the packed datasets use. Reading a single window therefore decompresses one block rather than the whole array, and the ForecastReader aligns its in-memory cache to the same block width so sweeping the window timeline decompresses each block exactly once. The chunk width is not recorded anywhere: it is a write-time storage choice that reads transparently regardless of the width a store was written with, so it does not affect the data-format version.

Compression

Compression is chosen at store creation and applies to every data variable, packed and standalone alike (the …_h hash variables are strings and are not compressed). The default is DEFLATE (zlib) level 3 with the byte-shuffle filter; the level (0–9) and shuffle can be changed, or compression turned off entirely. The choice is persisted in the compression global attribute and restored when the store is reopened, so later appends reuse the same filter:

An absent or unparseable attribute falls back to the default (deflate:3:shuffle), which is what such a file was written with. Compression is a storage detail only: arrays decode transparently regardless of the filter, so stores written with different settings stay mutually readable and data_format_version is unaffected by the choice.

Deletion and compaction

• Packed: deletion writes an empty string to the column's hash slot and zero-fills the column's data, so no stale values are readable through a reused slot. The slot becomes reusable by the next compatible write. The dataset does not shrink.
• Standalone: deletion drops the array from the in-memory index; the NetCDF variable lingers as dead space (NetCDF cannot delete variables in place).
• compact() reports reclaimable slots but does not physically resize datasets or remove dead standalone variables in this release (netcdf-c cannot resize a dimension in place).
• Feature sets: because they are shared, deleting an association never deletes its feature set; removing the last association that referenced one leaves it unreachable. compact() deletes unreachable sets and reports the row count as feature_sets_reclaimed. This is the one thing compaction physically removes.

SQLite Schema

The catalog database is created with PRAGMA foreign_keys = ON and the following DDL (idempotent — CREATE TABLE IF NOT EXISTS).

time_series_associations

One row per association between an owner and a stored array.

The two content-address hashes are the last two columns. Column order is not load-bearing — every statement names its columns — so the layout is chosen for readability.

feature_sets

The expanded feature map, one row per key. The typed columns are populated according to value_kind.

Feature sets are content-addressed, exactly as arrays are: the table is keyed by the SHA-256 of the feature map, and one set is stored once and shared by every association whose features_hash matches. The association row already carries that hash, so no join column is needed. Two associations with the same features therefore reference the same rows here — including a DeterministicSingleTimeSeries and the SingleTimeSeries it was derived from, which is why transform_single_time_series writes no feature rows at all.

An empty feature map stores no rows.

There is deliberately no foreign key to time_series_associations and no cascade: rows here are shared, so deleting one association must not delete a set another association still uses. Removing the last association that referenced a set instead leaves it unreachable — the same deletion semantics as the NetCDF side's unreachable standalone variables. Store::compact sweeps unreachable sets and reports the count as feature_sets_reclaimed; clearing a store drops them all outright.

supplemental_attribute_associations

Which supplemental attributes are attached to which components. Columns match infrasys' table of the same name, whose logic this replaces. See Associations Between Entities for the data model.

CREATE TABLE supplemental_attribute_associations (
    id             INTEGER PRIMARY KEY,
    component_id   INTEGER NOT NULL,
    component_type TEXT    NOT NULL,
    attribute_id   INTEGER NOT NULL,
    attribute_type TEXT    NOT NULL
);

CREATE UNIQUE INDEX uq_sa_assoc
    ON supplemental_attribute_associations(component_id, attribute_id);
CREATE INDEX idx_sa_assoc_attribute
    ON supplemental_attribute_associations(attribute_id, component_id, component_type);

uq_sa_assoc makes the (component_id, attribute_id) pair the row's identity — the type columns are denormalized labels for filtering, so the same pair under different type names is a duplicate and surfaces as DuplicateAssociation. That index also serves lookups keyed on the component; idx_sa_assoc_attribute serves the reverse direction ("which components carry this attribute").

One attribute may be attached to many components; one component may carry many attributes. Only the exact pair is constrained.

parent_child_associations

Directed edges between components — a generator (parent) connected to a bus (child), say. Both endpoints are always components, which is why there is no category column: a supplemental attribute cannot appear here by construction.

CREATE TABLE parent_child_associations (
    id          INTEGER PRIMARY KEY,
    parent_id   INTEGER NOT NULL,
    parent_type TEXT    NOT NULL,
    child_id    INTEGER NOT NULL,
    child_type  TEXT    NOT NULL
);

CREATE UNIQUE INDEX uq_parent_child
    ON parent_child_associations(parent_id, child_id);
CREATE INDEX idx_parent_child_child
    ON parent_child_associations(child_id, parent_id, parent_type);

Identity is the ordered (parent_id, child_id) pair, so the reversed pair is a different edge. There is no relationship-kind column: two components may be related at most once. Recording a second kind of edge between the same pair would need such a column added and uq_parent_child widened — which is a format change, unlike the tables themselves.

Properties shared by both association tables

Neither table has a foreign key — to time_series_associations or anywhere else — and neither cascades. The endpoints live in the consumer's object graph, not in this store, so the store never observes a component or attribute being deleted and a cascade could never fire. Removing a time series therefore leaves both tables untouched, and vice versa; consumers issue both calls when they want both effects.

Both are also independent of each other: the same integer may name a row in each without collision.

Compatibility

Both tables were added without bumping data_format_version, which is why a 0.10.0 store may or may not contain them:

• Opening a store for writing applies the full DDL, which is idempotent, so a store written before they existed gains both on first writable open.
• Opening read-only cannot run DDL. Reads of a missing table return empty results (0, false, no rows) rather than failing.

The version gate is exact equality with no upgrade path, so bumping for purely additive tables would have made every existing store unreadable in exchange for nothing.

schema_version

CREATE TABLE schema_version (version INTEGER NOT NULL);

A single-column table holding the catalog schema version. Creating the catalog inserts version = 1 if the table is empty; 1 is the current value. This tracks the SQLite schema alone and is distinct from the NetCDF data_format_version attribute, which governs the artifact as a whole and is the value open validates.

Indexes

CREATE UNIQUE INDEX uq_ts_assoc ON time_series_associations
    (owner_id, owner_category, time_series_type, name, resolution, interval, features_hash);
CREATE UNIQUE INDEX uq_ts_assoc_coalesced ON time_series_associations
    (owner_id, owner_category, time_series_type, name,
     COALESCE(resolution, ''), COALESCE(interval, ''), features_hash);

CREATE INDEX idx_hash       ON time_series_associations(data_hash);
CREATE INDEX idx_owner      ON time_series_associations(owner_id, owner_category);
CREATE INDEX idx_resolution ON time_series_associations(resolution);

Together the two unique indexes enforce key uniqueness; a violation surfaces as DuplicateTimeSeries. Both owner_id and owner_category are part of the key, so a component and a supplemental attribute that share an owner_id are independent owners. interval is part of the key, so two forecasts of one variable at the same resolution but different intervals are distinct series. SQLite treats NULL values as distinct in a UNIQUE index, so uq_ts_assoc does not constrain rows with a NULL resolution or interval (e.g. NonSequentialTimeSeries, or any static series, which carry no interval). uq_ts_assoc_coalesced covers that case by folding NULL to the empty-string sentinel via COALESCE before enforcing uniqueness (the empty string is never a valid ISO-8601 period).

These two were named uq_assoc and uq_assoc_coalesced before the association tables above existed and made a bare "assoc" ambiguous. The DDL drops the old names before creating the new ones, so a store written by an earlier build is renamed in place on its first writable open rather than accumulating two equivalent index pairs. Index names are not part of the on-disk contract, so this carries no data_format_version change.

Field Encoding Notes

• Timestamps are RFC 3339 strings in UTC.
• Periods (resolution, horizon, interval) are canonical ISO-8601 duration strings in SQLite (PT1H, P1M, P1Y), and the packed dataset name's {res} field uses the same encoding. Calendar periods (Month/Quarter/Year) are stored distinctly from fixed spans.
• Hashes are raw 32-byte BLOBs in SQLite, lowercase hex in NetCDF (the …_h variable for packed arrays, the arr_ variable name for standalone arrays).
• element_shape is the per-step shape only (the trailing axes); the time length is a separate column.

Inspecting a Store by Hand

ncdump -h system.nc                      # groups, variables, dtypes, shapes
sqlite3 system.nc.sqlite '.schema'
sqlite3 system.nc.sqlite \
  'SELECT name, time_series_type, dtype, element_shape, length FROM time_series_associations;'
# Both association tables are absent in stores written before they existed.
sqlite3 system.nc.sqlite 'SELECT * FROM supplemental_attribute_associations;'
sqlite3 system.nc.sqlite 'SELECT * FROM parent_child_associations;'

To map an association to its values: read its data_hash. For a packed array, hex-encode it and find the matching column in the relevant sts_…_h variable; for a standalone array, read the variable named arr_<hex_hash> directly.

Rust API

The public surface of infrastore-core. Import paths below are relative to the crate root.

#![allow(unused)]
fn main() {
use infrastore_core::{
    create_store, open_store, Store, BulkAdd, TimeSeriesKey, KeyIdentity,
    SingleTimeSeries, NonSequentialTimeSeries, Deterministic, Probabilistic, Scenarios,
    TimeSeriesData, TimeSeriesType, RequestedType, Period,
    TypedArray, Dtype, Compression, OwnerCategory, FeatureValue, Features, TimeSeriesMetadata,
    ListFilter, AddRequest,
    SupplementalAttributeAssociation, SupplementalAttributeFilter, SupplementalAttributeSummaryRow,
    ParentChildAssociation, ParentChildFilter,
    StaticReader, StaticGroup, ForecastReader, ForecastEntry, WindowSlot,
    TimeSeriesCounts, TimeSeriesCountsDetailed, StaticSummaryRow, ForecastSummaryRow,
    ForecastParameters, StaticConsistency, CompactionReport, IntegrityReport,
    TimeSeriesError, Result, DATA_FORMAT_VERSION,
};

// `array_hash` and `hash_hex` are also re-exported at the crate root; `features_hash` is not:
use infrastore_core::hash::{array_hash, features_hash, hash_hex};
use infrastore_core::storage::StorageBackend;
}

All time spans in this API — resolutions, horizons, and intervals — are the crate's Period, a calendar-aware span. Builders and constructors take impl Into<Period>, so you can pass a fixed chrono::Duration (e.g. Duration::hours(1), via From<Duration>) or a calendar span (Period::months(n), for the monthly/annual resolutions a fixed Duration cannot represent). Values read back — struct fields, get_resolutions, and the reader accessors — are always Period. Instants (DateTime<Utc>) remain chrono types.

Constructors

#![allow(unused)]
fn main() {
pub fn create_store(path: Option<&Path>, in_memory: bool) -> Result<Store>
pub fn create_store_with_compression(
    path: Option<&Path>,
    in_memory: bool,
    compression: Compression,
) -> Result<Store>
pub fn open_store(path: &Path, read_only: bool) -> Result<Store>
}

• create_store(None, true) — in-memory store, no filesystem I/O.
• create_store(Some(path), false) — creates path (NetCDF) and path.sqlite (metadata).
• create_store_with_compression(...) — as above but with an explicit NetCDF compression policy.
• open_store(path, read_only) — opens an existing pair. read_only = true rejects all writes.

Store::create / Store::create_with_compression / Store::open are the inherent-method equivalents.

#![allow(unused)]
fn main() {
pub enum Compression {
    None,
    Deflate { level: u8, shuffle: bool }, // level 0–9
}
}

create_store uses Compression::default() (DEFLATE level 3 + shuffle). The policy is persisted and restored when the store is reopened for appends, applies only to on-disk stores, and never changes how data is read back — see the storage model.

Store

#![allow(unused)]
fn main() {
impl Store {
    pub fn read_only(&self) -> bool;

    // The compression policy applied to writes (restored from the file on open;
    // `Compression::None` for in-memory stores).
    pub fn compression(&self) -> Compression;

    pub fn add_time_series(
        &mut self,
        owner_id: i64,
        owner_type: &str,
        owner_category: OwnerCategory,
        data: TimeSeriesData,
        features: Features,
        units: Option<String>,
    ) -> Result<TimeSeriesKey>;

    // A managed batch: packed series are written into batch-sized datasets that
    // fill whole HDF5 chunks (the optimized bulk-write path).
    pub fn add_time_series_bulk(&mut self, items: Vec<AddRequest>) -> Result<Vec<TimeSeriesKey>>;

    // Begin a buffered bulk add. Requests pushed onto the returned guard are
    // accumulated in memory and written together by `BulkAdd::commit` (same
    // block-write path as `add_time_series_bulk`); dropping without committing
    // discards the buffer.
    pub fn bulk_add(&mut self) -> BulkAdd<'_>;

    // Copy one association onto another owner (metadata only; the array is shared).
    pub fn copy_time_series(
        &mut self,
        src: &KeyIdentity,
        dst_owner_id: i64,
        dst_owner_type: &str,
        new_name: Option<&str>,   // None keeps the source name
    ) -> Result<TimeSeriesKey>;

    pub fn get_time_series(
        &self,
        key: &KeyIdentity,
        time_range: Option<(DateTime<Utc>, DateTime<Utc>)>,
    ) -> Result<TimeSeriesData>;

    // Read many full series at once (no time-range slicing). Packed
    // `SingleTimeSeries` are read in one decompress-once pass per dataset; other
    // types reuse the per-key path. Returns a `TimeSeriesData` per key, in order.
    pub fn bulk_read(&self, keys: &[&KeyIdentity]) -> Result<Vec<TimeSeriesData>>;

    pub fn transform_single_time_series(
        &mut self,
        horizon: impl Into<Period>,
        interval: impl Into<Period>,
        owner_category: Option<OwnerCategory>,
        resolution: Option<Period>,
    ) -> Result<usize>;

    pub fn remove_time_series(&mut self, key: &KeyIdentity) -> Result<()>;
    pub fn clear_time_series(
        &mut self,
        owner: Option<(i64, OwnerCategory)>,
    ) -> Result<usize>;
    pub fn replace_owner(
        &mut self,
        old_owner: i64,
        new_owner: i64,
        owner_category: OwnerCategory,
    ) -> Result<usize>;

    pub fn list_time_series(&self, filter: ListFilter) -> Result<Vec<TimeSeriesMetadata>>;
    // Key-centric listing: the same rows reduced to their keys, dropping storage
    // detail (`data_hash`, `dtype`, `ext`, `percentiles`).
    pub fn list_keys(&self, filter: ListFilter) -> Result<Vec<TimeSeriesKey>>;
    // …and with each key's array content hash, so callers can group series that
    // share stored data (dedup'd arrays; an STS and any DST derived from it).
    pub fn list_keys_with_hash(
        &self,
        filter: ListFilter,
    ) -> Result<Vec<(TimeSeriesKey, [u8; 32])>>;
    pub fn get_time_series_keys(
        &self,
        owner_id: i64,
        owner_category: OwnerCategory,
    ) -> Result<Vec<TimeSeriesKey>>;
    pub fn has_time_series(&self, key: &KeyIdentity) -> Result<bool>;

    // Resolve a forecast addressed by attributes + a `RequestedType` to the one
    // matching key. `NotFound` if nothing matches; `InvalidParameter` if ambiguous.
    pub fn resolve_forecast_key(
        &self,
        owner_id: i64,
        owner_category: OwnerCategory,
        name: &str,
        resolution: Option<Period>,
        interval: Option<Period>,
        features: Features,
        requested: RequestedType,
    ) -> Result<TimeSeriesKey>;

    pub fn get_metadata(&self, key: &KeyIdentity) -> Result<TimeSeriesMetadata>;
    pub fn get_array_by_hash(&self, hash: &[u8; 32]) -> Result<TypedArray>;
    // (SingleTimeSeries, DeterministicSingleTimeSeries) associations on one array.
    pub fn count_array_references(&self, data_hash: &[u8; 32]) -> Result<(usize, usize)>;

    pub fn get_resolutions(
        &self,
        time_series_type: Option<TimeSeriesType>,
    ) -> Result<Vec<Period>>;
    pub fn get_time_series_counts(&self) -> Result<TimeSeriesCounts>;
    pub fn get_forecast_parameters(
        &self,
        resolution: Option<Period>,
        interval: Option<Period>,
    ) -> Result<ForecastParameters>;

    // Catalog introspection (each one catalog query; see "Introspection" below).
    pub fn check_static_consistency(
        &self,
        resolution: Option<Period>,
    ) -> Result<Vec<StaticConsistency>>;
    pub fn counts_by_type(&self) -> Result<Vec<(TimeSeriesType, i64)>>;
    pub fn num_distinct_arrays(&self) -> Result<i64>;
    pub fn time_series_counts_detailed(&self) -> Result<TimeSeriesCountsDetailed>;
    pub fn list_owner_ids(
        &self,
        category: OwnerCategory,
        time_series_type: Option<TimeSeriesType>,
        resolution: Option<Period>,
    ) -> Result<Vec<i64>>;
    pub fn static_summary(&self) -> Result<Vec<StaticSummaryRow>>;
    pub fn forecast_summary(&self) -> Result<Vec<ForecastSummaryRow>>;

    // Per-timestamp readers (see "Readers" below).
    pub fn build_static_reader(&self, filter: ListFilter) -> Result<StaticReader>;
    pub fn static_read(&self, reader: &mut StaticReader, at: DateTime<Utc>) -> Result<()>;
    pub fn build_forecast_reader(&self, filter: ListFilter) -> Result<ForecastReader>;
    pub fn forecast_read(&self, reader: &mut ForecastReader, at: DateTime<Utc>) -> Result<()>;

    // The supplemental-attribute catalog (see "Associations" below). Independent
    // of time series: none of these touch, or are touched by, a time-series call.
    pub fn add_supplemental_attribute_association(
        &mut self,
        assoc: SupplementalAttributeAssociation,
    ) -> Result<()>;
    pub fn add_supplemental_attribute_associations(
        &mut self,
        assocs: Vec<SupplementalAttributeAssociation>,
    ) -> Result<usize>;
    pub fn has_supplemental_attribute_association(
        &self,
        filter: &SupplementalAttributeFilter,
    ) -> Result<bool>;
    pub fn list_supplemental_attribute_associations(
        &self,
        filter: &SupplementalAttributeFilter,
    ) -> Result<Vec<SupplementalAttributeAssociation>>;
    pub fn list_supplemental_attribute_ids(
        &self,
        filter: &SupplementalAttributeFilter,
    ) -> Result<Vec<i64>>;
    pub fn list_components_with_attributes(
        &self,
        filter: &SupplementalAttributeFilter,
    ) -> Result<Vec<i64>>;
    pub fn remove_supplemental_attribute_associations(
        &mut self,
        filter: &SupplementalAttributeFilter,
    ) -> Result<usize>;
    pub fn replace_supplemental_attribute_component_id(
        &mut self,
        old_id: i64,
        new_id: i64,
    ) -> Result<usize>;
    pub fn count_supplemental_attribute_associations(
        &self,
        filter: &SupplementalAttributeFilter,
    ) -> Result<i64>;
    pub fn count_supplemental_attributes(
        &self,
        filter: &SupplementalAttributeFilter,
    ) -> Result<i64>;
    pub fn count_components_with_attributes(
        &self,
        filter: &SupplementalAttributeFilter,
    ) -> Result<i64>;
    pub fn supplemental_attribute_counts_by_type(&self) -> Result<Vec<(String, i64)>>;
    pub fn supplemental_attribute_summary(
        &self,
    ) -> Result<Vec<SupplementalAttributeSummaryRow>>;

    // The parent/child catalog (see "Associations" below). Same independence
    // from time series.
    pub fn add_parent_child_association(&mut self, assoc: ParentChildAssociation) -> Result<()>;
    pub fn add_parent_child_associations(
        &mut self,
        assocs: Vec<ParentChildAssociation>,
    ) -> Result<usize>;
    pub fn has_parent_child_association(&self, filter: &ParentChildFilter) -> Result<bool>;
    pub fn list_parent_child_associations(
        &self,
        filter: &ParentChildFilter,
    ) -> Result<Vec<ParentChildAssociation>>;
    pub fn list_children(&self, filter: &ParentChildFilter) -> Result<Vec<i64>>;
    pub fn list_parents(&self, filter: &ParentChildFilter) -> Result<Vec<i64>>;
    pub fn remove_parent_child_associations(
        &mut self,
        filter: &ParentChildFilter,
    ) -> Result<usize>;
    pub fn replace_parent_child_component_id(
        &mut self,
        old_id: i64,
        new_id: i64,
    ) -> Result<usize>;
    pub fn count_parent_child_associations(&self, filter: &ParentChildFilter) -> Result<i64>;

    pub fn compact(&mut self) -> Result<CompactionReport>;
    pub fn verify_integrity(&self) -> Result<IntegrityReport>;
    pub fn flush(&mut self) -> Result<()>;
    // Write the whole store (arrays + catalog) to `path` + `<path>.sqlite`,
    // overwriting them. Works for on-disk *and* in-memory stores.
    pub fn persist_to(&mut self, path: &Path) -> Result<()>;
}
}

Store is Send but not Sync (the SQLite catalog holds a rusqlite::Connection): a store can be moved between threads, but sharing one requires external synchronization — Arc<Mutex<Store>>, serializing reads as well as writes, which is what the gRPC server does.

Method notes

• add_time_series — Accepts any TimeSeriesData variant — SingleTimeSeries, NonSequentialTimeSeries, or a dense forecast (Deterministic, Probabilistic, Scenarios). Hashes the array, stores it (deduplicating on the hash), inserts a metadata association, and returns its key. Errors with DuplicateTimeSeries if the key already exists or ReadOnlyStore on a read-only store. It is a convenience wrapper over add_time_series_bulk.
• transform_single_time_series — Derives a DeterministicSingleTimeSeries from every stored SingleTimeSeries, sharing the underlying array (with count derived from the series length), and returns the number of series transformed. This is the only way to create a DeterministicSingleTimeSeries; it is never added directly. The optional owner_category and resolution filters restrict the transform to a single owner category and/or resolution, leaving other series untouched.
• add_time_series_bulk — All-or-nothing: every array put and association insert in the call commits together or rolls back together.
• get_time_series — Reconstructs the stored type as a TimeSeriesData variant (static series and all forecast types). With time_range = Some((start, end)), slices on the time axis; the returned series's initial_timestamp and length reflect the slice. For forecasts the window is resolved over the count axis (resolve_windows). end is exclusive.
• clear_time_series — Some((id, category)) removes one owner's series (the owner is the (owner_id, owner_category) pair); None removes all. Returns the count removed. Underlying arrays are freed only when their last reference is gone.
• replace_owner — Reassigns every series owned by (old_owner_id, owner_category) to (new_owner_id, owner_category), returning the number of associations updated. The category is unchanged by the move and scopes which owner's series are reassigned.
• copy_time_series — Copies one association onto (dst_owner_id, dst_owner_type), keeping the source's owner_category and every descriptive column — crucially time_series_type, so a DeterministicSingleTimeSeries stays one rather than being materialized into a dense Deterministic (what a read-then-write copy through the bindings would produce). Only a metadata row is written: the array is content-addressed and shared. new_name = None keeps the source name. Errors with DuplicateTimeSeries if the destination identity already exists.
• get_time_series_keys — Lists every key for the owner identified by the (owner_id, owner_category) pair.
• list_keys / list_keys_with_hash — The key-centric listing path (what the bindings use). list_keys_with_hash pairs each key with its array's content hash in the same single catalog query, so callers can group keys by the data behind them.
• resolve_forecast_key — Resolves a forecast addressed by attributes plus a RequestedType to the single matching key, whose time_series_type is the concrete type that matched. resolution and interval are optional filters; leave them None to match across them. NotFound if nothing matches, InvalidParameter if more than one does.
• get_metadata / get_array_by_hash — The low-level pair used by external bindings: resolve a key to metadata (including data_hash), then read the array directly.
• count_array_references — (sts, dst) association counts referencing one data_hash, so a caller can tell whether removing a SingleTimeSeries would orphan a DeterministicSingleTimeSeries derived from (and sharing) its array.
• verify_integrity — Recomputes each stored array's hash and reports mismatches. Covers the NetCDF half only: the SQLite catalog is not inspected, so an empty report does not mean the store as a whole is sound. See content addressing.
• flush — Issues nc_sync so the files can be copied for persistence without closing.
• persist_to — Writes both halves of the artifact to path and <path>.sqlite, overwriting existing targets. An on-disk store is flushed and copied; an in-memory store is materialized (every distinct array by hash, plus the whole catalog). Because arrays are content-addressed, this reproduces every series — static, forecast, non-sequential — without per-type reconstruction.

Introspection

Grouped catalog queries the bindings use instead of listing every association and aggregating in the caller. All are read-only and hit SQLite once.

• check_static_consistency — one StaticConsistency { resolution, initial_timestamp, length } per resolution present (empty Vec when there are no SingleTimeSeries), ordered by resolution; each row is the grid shared by every SingleTimeSeries at that resolution. Consistency is only required within a resolution — series at different resolutions legitimately have different grids — so pass Some(resolution) to scope the check to one grid. Returns IntegrityError when the series at a single resolution disagree.
• counts_by_type — Association count per TimeSeriesType.
• num_distinct_arrays — Distinct stored content hashes; series sharing an array count once.
• time_series_counts_detailed — TimeSeriesCountsDetailed: distinct owners split by category, and distinct arrays (not associations) split into static vs forecast.
• list_owner_ids — Distinct owner ids in one category that have a time series, optionally narrowed by type and/or resolution.
• static_summary / forecast_summary — One StaticSummaryRow / ForecastSummaryRow per distinct owner/name/shape (or window) combination, with the association count. The core groups; the binding formats the table.

Forecasts

Dense forecasts (Deterministic, Probabilistic, Scenarios) are written through the generic add_time_series by wrapping the corresponding object in a TimeSeriesData variant. Build the object with its new constructor — each holds a TypedArray in its native shape, and the constructor validates the shape against the windowing parameters (horizon, interval, count, and for Probabilistic the percentiles):

#![allow(unused)]
fn main() {
use infrastore_core::{Deterministic, TimeSeriesData};

let forecast = Deterministic::new(
    initial_timestamp, resolution, horizon, interval, count, data, name,
)?;
let key = store.add_time_series(
    owner_id, owner_type, OwnerCategory::Component,
    TimeSeriesData::Deterministic(forecast),
    features, units,
)?;
}

Dense forecast arrays (Deterministic / Probabilistic / Scenarios) are stored as standalone NetCDF variables. A DeterministicSingleTimeSeries is not added directly: call transform_single_time_series(horizon, interval, owner_category, resolution) to derive one from every stored SingleTimeSeries (it shares the backing column-packed array, derives count from the series length, and dedups against that series).

Conventional array shapes:

Reading forecasts: get_time_series reconstructs all forecast types, returning the matching TimeSeriesData variant — Deterministic, Probabilistic, or Scenarios. A DeterministicSingleTimeSeries is synthesized into a Deterministic by gathering its windows from the underlying packed array. The low-level pair still works for direct array access: resolve a TimeSeriesMetadata with get_metadata (it carries horizon, interval, count, and percentiles), then fetch the array with get_array_by_hash(&meta.data_hash).

Readers

get_time_series returns a whole series or forecast. To read many whole series at once (e.g. exploration or plotting), bulk_read takes a slice of keys and reads packed SingleTimeSeries in one decompress-once pass per dataset — far cheaper than a get_time_series per key under the timestamp-major chunking, where a single full-series read touches every chunk. For the timestamp-oriented access pattern — walk the timeline and read every series' value at each instant — build a reader instead. A reader is built once over a ListFilter, pins one resolution, and holds reusable buffers that each read overwrites in place, so a tight loop allocates nothing. The reader is a passive plan: it does not borrow the Store, so reads go through Store::static_read / Store::forecast_read, which fill the buffers; the caller then walks the groups/entries. There are two: StaticReader for SingleTimeSeries and ForecastReader for forecasts.

#![allow(unused)]
fn main() {
// Static: value of every SingleTimeSeries at one timestamp, columnar.
let mut reader = store.build_static_reader(ListFilter::new().resolution(res))?;
for k in 0..reader.length() {
    // Grid point k: initial + k·resolution (calendar-aware, hence `Period::add_to`).
    let at = reader.resolution().add_to(reader.initial_timestamp(), k as i64).unwrap();
    store.static_read(&mut reader, at)?;
    for group in reader.groups() {
        let bytes = group.values();        // [num_columns, *element_shape], row-major LE
        // group.keys()[j] identifies column j; group.dtype(), group.element_shape()
    }
}

// Forecast: the window at one timestamp for every matching forecast of one type.
let mut reader = store.build_forecast_reader(
    ListFilter::new().time_series_type(TimeSeriesType::Deterministic).resolution(res),
)?;
for k in 0..reader.count() {
    // Window k: initial + k·interval.
    let at = reader.interval().add_to(reader.initial_timestamp(), k as i64).unwrap();
    store.forecast_read(&mut reader, at)?;
    // `entry_slot` takes the *entry* index, not `entry.slot()` (which indexes `slots()`).
    for (i, entry) in reader.entries().iter().enumerate() {
        let slot = reader.entry_slot(i);
        let bytes = slot.window();         // window of slot.window_shape(), row-major LE
        // entry.key() identifies the forecast/owner
    }
}
}

build_static_reader requires the filter to pin a resolution and that all matched series share one grid (initial_timestamp + length) — validated at build, so there is no presence mask. build_forecast_reader requires a forecast type and a resolution; a Deterministic reader is abstract (also matches DeterministicSingleTimeSeries), and all matched forecasts must share one window timeline (initial_timestamp + interval + count). static_read / forecast_read error (never clamp) if at is off the grid/timeline.

Window-read deduplication. A ForecastReader groups its entries into WindowSlots keyed by (array hash, read plan): forecasts that reference the same array and slice it the same way — deduplicated identical data, or several DeterministicSingleTimeSeries over one SingleTimeSeries — share one slot. forecast_read performs one backend read per slot, not per entry, so a forecast shared by N owners is read once per timestamp (the forecast analog of StaticReader reading a packed column once and gathering it to many columns). reader.slots() / reader.entry_slot(i) expose the slots; note that entry_slot takes the entry index i and returns the slot backing that entry, while entry.slot() is that slot's index into slots() (equal for entries that share data).

Associations

Two catalogs of relationships between entities the store does not otherwise model. They live here so consumers do not each carry their own SQLite database for them, and they are wholly independent of time series: there are no foreign keys and no cascade (both endpoints live in the caller's object graph, so a cascade could never fire), so removing a time series never removes an association and removing an association never removes a time series. A caller that wants both composes the two calls.

Both families share the same filter conventions: every field of the filter is optional, set fields are ANDed, and the default filter matches every row — which is what makes a bulk export/import pair a round trip. The *_types fields are lists of concrete type names rendered as SQL IN (…); expanding an abstract type into its subtypes stays with the caller, where the type hierarchy lives, and an empty list matches nothing. Every remove_* returns the number of rows removed, and removing zero rows is Ok(0) rather than an error: the store has no view of whether the caller expected a hit.

Supplemental-attribute associations

Which supplemental attributes are attached to which components. Identity is the (component_id, attribute_id) pair — the type names are denormalized labels carried for filtering and reporting, not part of identity — so re-attaching the same pair under different type names is still a duplicate. One attribute may be attached to many components.

• add_supplemental_attribute_association — Attaches one SupplementalAttributeAssociation. Errors with DuplicateAssociation if that component already carries that attribute, whatever type names are supplied.
• add_supplemental_attribute_associations — All-or-nothing: a duplicate anywhere in the batch rolls the whole batch back. Returns the number inserted. It is the import half of the round trip whose export is list_supplemental_attribute_associations with a default filter.
• list_supplemental_attribute_associations / has_supplemental_attribute_association / count_supplemental_attribute_associations — The SupplementalAttributeFilter predicate over the table. The list returns rows in insertion order, so a default-filter export/import pair round-trips.
• list_supplemental_attribute_ids / list_components_with_attributes — Distinct ids on one end of the matching rows, ascending: the attributes attached to a component when component_id is set, and the components carrying an attribute when attribute_id is set.
• count_supplemental_attributes / count_components_with_attributes — The same two queries counted rather than listed.
• remove_supplemental_attribute_associations — Removes every matching row and returns the count.
• replace_supplemental_attribute_component_id — Moves every attachment from component old_id to new_id, returning the rows updated. Errors with DuplicateAssociation if new_id already carries one of the attributes being moved.
• supplemental_attribute_counts_by_type / supplemental_attribute_summary — Grouped counts, by attribute type or by both type names (SupplementalAttributeSummaryRow, ordered by attribute type then component type). The core groups; the caller formats.

#![allow(unused)]
fn main() {
use infrastore_core::{SupplementalAttributeAssociation, SupplementalAttributeFilter};

store.add_supplemental_attribute_association(SupplementalAttributeAssociation {
    component_id: 1,
    component_type: "Generator".into(),
    attribute_id: 100,
    attribute_type: "GeographicInfo".into(),
})?;

// The attributes attached to component 1, then the components carrying attribute 100.
let attributes =
    store.list_supplemental_attribute_ids(&SupplementalAttributeFilter::new().component_id(1))?;
let components =
    store.list_components_with_attributes(&SupplementalAttributeFilter::new().attribute_id(100))?;

// Detach them: removing the attachments leaves any time series untouched.
let removed = store.remove_supplemental_attribute_associations(
    &SupplementalAttributeFilter::new().component_id(1),
)?;

// Bulk round trip — the default filter matches every row.
let exported = store.list_supplemental_attribute_associations(&Default::default())?;
target.add_supplemental_attribute_associations(exported)?;
}

Parent/child associations

Directed edges between components — a generator (parent) connected to a bus (child), say. Both endpoints are always components; an attribute cannot appear here. Identity is the ordered (parent_id, child_id) pair, so the reversed pair is a different edge. There is no relationship-kind column, so one ordered pair may be related at most once.

This family is deliberately narrower than the supplemental one: it has no counts-by-type and no grouped summary, because there is no consumer for them yet. Both are additive if one appears.

• add_parent_child_association — Records one ParentChildAssociation. Errors with DuplicateAssociation if that ordered pair is already related.
• add_parent_child_associations — All-or-nothing bulk insert, returning the number inserted; the import half of the round trip whose export is list_parent_child_associations with a default filter.
• list_parent_child_associations / has_parent_child_association / count_parent_child_associations — The ParentChildFilter predicate over the table. The list returns rows in insertion order.
• list_children / list_parents — Distinct ids on one end of the matching edges, ascending: the children of a component when parent_id is set, and its parents when child_id is set.
• remove_parent_child_associations — Removes every matching edge and returns the count.
• replace_parent_child_component_id — Rewrites component old_id to new_id on both ends of every edge, returning the rows updated. Errors with DuplicateAssociation if the rewrite would duplicate an edge new_id already has.

#![allow(unused)]
fn main() {
use infrastore_core::{ParentChildAssociation, ParentChildFilter};

store.add_parent_child_association(ParentChildAssociation {
    parent_id: 1,
    parent_type: "Generator".into(),
    child_id: 7,
    child_type: "Bus".into(),
})?;

// The reversed pair is a different edge, not a duplicate.
store.add_parent_child_association(ParentChildAssociation {
    parent_id: 7,
    parent_type: "Bus".into(),
    child_id: 1,
    child_type: "Generator".into(),
})?;

let children = store.list_children(&ParentChildFilter::new().parent_id(1))?;   // [7]
let parents = store.list_parents(&ParentChildFilter::new().child_id(7))?;      // [1]

// Bulk round trip — the default filter matches every row.
let exported = store.list_parent_child_associations(&ParentChildFilter::default())?;
target.add_parent_child_associations(exported)?;
}

Neither association catalog is exposed over the gRPC server or the infrastore CLI.

Types

TimeSeriesKey and KeyIdentity

TimeSeriesKey is an enum: one variant per series family, each pairing the shared identity with a per-variant descriptive snapshot (window/shape parameters). Equality is identity-only — two keys with the same KeyIdentity are equal even if their descriptive snapshots differ, so a key stays a reliable handle.

#![allow(unused)]
fn main() {
pub enum TimeSeriesKey {
    Single(SingleTimeSeriesKey),
    NonSequential(NonSequentialTimeSeriesKey),
    Forecast(ForecastTimeSeriesKey),
}

pub struct SingleTimeSeriesKey {
    pub identity: KeyIdentity,
    pub initial_timestamp: DateTime<Utc>,
    pub length: usize,
}

pub struct NonSequentialTimeSeriesKey {
    pub identity: KeyIdentity,
    pub length: usize,
}

pub struct ForecastTimeSeriesKey {
    pub identity: KeyIdentity,
    pub initial_timestamp: DateTime<Utc>,
    pub horizon: Period,
    pub count: usize,
}
}

KeyIdentity is the identifying tuple shared by every variant — and the only thing that determines equality and is looked up in the catalog. interval is part of the identity (Some for every forecast type, None for the static types); resolution is Option because NonSequentialTimeSeries has none.

#![allow(unused)]
fn main() {
pub struct KeyIdentity {
    pub owner_id: i64,
    pub owner_category: OwnerCategory,
    pub time_series_type: TimeSeriesType,
    pub name: String,
    pub resolution: Option<Period>,
    pub interval: Option<Period>,
    pub features: Features,
}
}

TimeSeriesKey::identity() returns &KeyIdentity; the lookup methods (get_time_series, get_metadata, has_time_series, remove_time_series, bulk_read) take &KeyIdentity. See Data Model.

SingleTimeSeries

#![allow(unused)]
fn main() {
pub struct SingleTimeSeries {
    pub initial_timestamp: DateTime<Utc>,
    pub resolution: Period,
    pub length: usize,
    pub data: TypedArray,
    pub name: String,
}

impl SingleTimeSeries {
    pub fn new(
        initial_timestamp: DateTime<Utc>, resolution: impl Into<Period>, data: TypedArray,
        name: impl Into<String>,
    ) -> Self;
}
}

length is derived from the array's first axis (data.length()) by new.

TypedArray and Dtype

The storage array type: an element dtype, an N-dimensional shape [length, k1, k2, …] (first axis time, trailing axes the per-step element shape), and raw row-major, little-endian bytes.

#![allow(unused)]
fn main() {
pub enum Dtype { F64, F32, I64, I32, U64, Bool }   // codes 0..=5; size() = 8/4/8/4/8/1

pub struct TypedArray {
    pub dtype: Dtype,
    pub shape: Vec<usize>,
    pub bytes: Vec<u8>,
}

impl TypedArray {
    pub fn new(dtype: Dtype, shape: Vec<usize>, bytes: Vec<u8>) -> Result<Self, String>; // validates len
    pub fn from_f64(shape: Vec<usize>, values: &[f64]) -> Self;
    pub fn to_f64_vec(&self) -> Result<Vec<f64>, String>;
    pub fn length(&self) -> usize;          // shape[0]
    pub fn element_shape(&self) -> &[usize]; // shape[1..]
}
}

Dtype::code() / Dtype::from_code(i32) and Dtype::as_str() / Dtype::parse(&str) convert to and from the stable integer codes and string names used by the bindings and the on-disk format.

Period

The calendar-aware time span used for every resolution, horizon, and interval. A Period is either a fixed span (a chrono::Duration — hours, minutes, days, weeks) or a calendar span (a count of months, so Quarter = 3, Year = 12), letting the store represent monthly/annual grids a fixed Duration cannot.

#![allow(unused)]
fn main() {
pub enum Period {
    Fixed(Duration),   // a fixed chrono::Duration
    Months(i32),       // n calendar months
}

impl Period {
    pub fn fixed(d: Duration) -> Self;     // also: From<Duration> for Period
    pub fn months(n: i32) -> Self;
    pub fn is_irregular(&self) -> bool;    // true for Months
    pub fn is_positive(&self) -> bool;
    pub fn same_kind(&self, other: &Period) -> bool;  // both Fixed, or both Months

    // Grid arithmetic (calendar-aware for `Months`).
    pub fn add_to(&self, dt: DateTime<Utc>, k: i64) -> Option<DateTime<Utc>>;
    // Whole steps from `start` to `at`; errors if `at` is before `start` or off-grid.
    pub fn steps_between(&self, start: DateTime<Utc>, at: DateTime<Utc>) -> Result<usize>;
    // Nearest grid step at or below / at or above `at`; clamps to 0, never errors
    // (used for time-range slicing, where the bounds are arbitrary).
    pub fn floor_steps(&self, start: DateTime<Utc>, at: DateTime<Utc>) -> usize;
    pub fn ceil_steps(&self, start: DateTime<Utc>, at: DateTime<Utc>) -> usize;
    // `other / self` as an exact positive integer (H = horizon / resolution).
    // Mixing a Fixed and a Months period is an error.
    pub fn divide_into(&self, other: &Period) -> Result<usize>;

    // The on-disk / on-the-wire encoding: an ISO-8601 duration ("PT1H", "P1M", "P1Y").
    pub fn to_iso8601(&self) -> String;                 // also the `Display` impl
    pub fn from_iso8601(s: &str) -> Result<Period>;
}
}

to_iso8601 / from_iso8601 are the persistence contract: every resolution, horizon, and interval is stored and transmitted as that string. The encoding is a pure function of the value, so equal periods always encode identically (which is what the catalog's uniqueness key relies on), and it round-trips. Calendar units (Y, M before the T) decode to Months; fixed units (W, D, and H/M/S after the T) decode to Fixed; a string mixing the two is rejected.

Because Period: From<Duration>, anywhere the API takes impl Into<Period> you may pass a chrono::Duration directly (e.g. Duration::hours(1)); use Period::months(n) for calendar spans. Two periods of different kinds (one Fixed, one Months) are never equal, even if a particular month happens to span the same wall-clock time. See the data model for how resolution drives the storage grid.

NonSequentialTimeSeries

#![allow(unused)]
fn main() {
pub struct NonSequentialTimeSeries {
    pub timestamps: Vec<DateTime<Utc>>,
    pub length: usize,
    pub data: TypedArray,
    pub name: String,
}

impl NonSequentialTimeSeries {
    pub fn new(
        timestamps: Vec<DateTime<Utc>>, data: TypedArray, name: impl Into<String>,
    ) -> Result<Self, String>;
}
}

new validates that timestamps are strictly increasing and match the data length.

Deterministic

#![allow(unused)]
fn main() {
pub struct Deterministic {
    pub initial_timestamp: DateTime<Utc>,
    pub resolution: Period,
    pub horizon: Period,
    pub interval: Period,
    pub count: usize,
    pub data: TypedArray,   // shape [H, count, *E]
    pub name: String,
}

impl Deterministic {
    pub fn new(
        initial_timestamp: DateTime<Utc>, resolution: impl Into<Period>,
        horizon: impl Into<Period>, interval: impl Into<Period>, count: usize, data: TypedArray,
        name: impl Into<String>,
    ) -> Result<Self, String>;
}
}

new validates data.shape against [H, count, *E] where H = horizon / resolution.

Probabilistic

#![allow(unused)]
fn main() {
pub struct Probabilistic {
    pub initial_timestamp: DateTime<Utc>,
    pub resolution: Period,
    pub horizon: Period,
    pub interval: Period,
    pub count: usize,
    pub percentiles: Vec<f64>,
    pub data: TypedArray,   // shape [num_percentiles, H, count, *E]
    pub name: String,
}

impl Probabilistic {
    pub fn new(
        initial_timestamp: DateTime<Utc>, resolution: impl Into<Period>,
        horizon: impl Into<Period>, interval: impl Into<Period>, count: usize,
        percentiles: Vec<f64>, data: TypedArray, name: impl Into<String>,
    ) -> Result<Self, String>;
}
}

new also requires percentiles to be non-empty and strictly increasing.

Scenarios

#![allow(unused)]
fn main() {
pub struct Scenarios {
    pub initial_timestamp: DateTime<Utc>,
    pub resolution: Period,
    pub horizon: Period,
    pub interval: Period,
    pub count: usize,
    pub scenario_count: usize,
    pub data: TypedArray,   // shape [scenario_count, H, count, *E]
    pub name: String,
}

impl Scenarios {
    pub fn new(
        initial_timestamp: DateTime<Utc>, resolution: impl Into<Period>,
        horizon: impl Into<Period>, interval: impl Into<Period>, count: usize,
        scenario_count: usize, data: TypedArray, name: impl Into<String>,
    ) -> Result<Self, String>;
}
}

StaticReader and StaticGroup

The columnar SingleTimeSeries reader (see Readers). Period is the crate's resolution type. values() is empty until the first Store::static_read.

#![allow(unused)]
fn main() {
impl StaticReader {
    pub fn initial_timestamp(&self) -> DateTime<Utc>;
    pub fn resolution(&self) -> Period;
    pub fn length(&self) -> usize;            // grid points; valid timestamps initial + k·resolution
    pub fn groups(&self) -> &[StaticGroup];
    pub fn index_at(&self, at: DateTime<Utc>) -> Result<usize>;
}

impl StaticGroup {
    pub fn dtype(&self) -> Dtype;
    pub fn element_shape(&self) -> &[usize];  // trailing per-step dims; empty == scalar
    pub fn keys(&self) -> &[TimeSeriesKey];   // column j identity
    pub fn num_columns(&self) -> usize;
    pub fn values(&self) -> &[u8];            // [num_columns, *element_shape], row-major LE
}
}

ForecastReader, WindowSlot, and ForecastEntry

The forecast-window reader (see Readers). Entries are the per-key forecasts; slots are the deduplicated physical reads. WindowSlot::window() is empty until the first Store::forecast_read.

#![allow(unused)]
fn main() {
impl ForecastReader {
    pub fn time_series_type(&self) -> TimeSeriesType;
    pub fn initial_timestamp(&self) -> DateTime<Utc>;
    pub fn resolution(&self) -> Period;
    pub fn interval(&self) -> Period;
    pub fn count(&self) -> usize;             // windows; valid timestamps initial + k·interval
    pub fn entries(&self) -> &[ForecastEntry];
    pub fn slots(&self) -> &[WindowSlot];     // one backend read each per forecast_read
    pub fn entry_slot(&self, i: usize) -> &WindowSlot;  // slot backing entry i
    pub fn window_index(&self, at: DateTime<Utc>) -> Result<usize>;
}

impl ForecastEntry {
    pub fn key(&self) -> &TimeSeriesKey;
    pub fn slot(&self) -> usize;              // index into slots(); equal for entries sharing data
}

impl WindowSlot {
    pub fn dtype(&self) -> Dtype;
    pub fn window_shape(&self) -> &[usize];   // [H,*E] / [P,H,*E] / [scenarios,H,*E]
    pub fn window(&self) -> &[u8];            // most recent window, row-major LE
}
}

TimeSeriesData

#![allow(unused)]
fn main() {
pub enum TimeSeriesData {
    SingleTimeSeries(SingleTimeSeries),
    NonSequentialTimeSeries(NonSequentialTimeSeries),
    Deterministic(Deterministic),
    Probabilistic(Probabilistic),
    Scenarios(Scenarios),
}

impl TimeSeriesData {
    pub fn time_series_type(&self) -> TimeSeriesType;
    pub fn as_single(&self) -> Option<&SingleTimeSeries>;
    pub fn as_non_sequential(&self) -> Option<&NonSequentialTimeSeries>;
    pub fn as_deterministic(&self) -> Option<&Deterministic>;
    pub fn as_probabilistic(&self) -> Option<&Probabilistic>;
    pub fn as_scenarios(&self) -> Option<&Scenarios>;
}
}

There is no DeterministicSingleTimeSeries variant: a stored DeterministicSingleTimeSeries is read back as a Deterministic (so as_deterministic returns Some for it).

TimeSeriesType

#![allow(unused)]
fn main() {
pub enum TimeSeriesType {
    SingleTimeSeries,
    NonSequentialTimeSeries,
    Deterministic,
    DeterministicSingleTimeSeries,
    Probabilistic,
    Scenarios,
}
}

as_str() / parse(&str) convert to and from the canonical string names used on disk.

OwnerCategory

#![allow(unused)]
fn main() {
pub enum OwnerCategory { Component, SupplementalAttribute }
}

FeatureValue and Features

#![allow(unused)]
fn main() {
pub enum FeatureValue { Int(i64), Float(f64), Bool(bool), Str(String) }
pub type Features = BTreeMap<String, FeatureValue>;
}

Features is sorted by key, which fixes hash order and the uniqueness constraint. FeatureValue canonicalizes NaN for hashing and equality.

TimeSeriesMetadata

The full record returned by list_time_series and get_metadata: owner fields, time_series_type, name, data_hash: [u8; 32], the optional temporal fields (initial_timestamp, resolution, length, horizon, interval, count, timestamps), features, units, percentiles: Option<Vec<f64>> (set for Probabilistic), and the array typing: dtype: Dtype, element_shape: Vec<usize>, and ext: Option<String>. The span fields (resolution, horizon, interval) are Option<Period>.

ListFilter

A builder; every field is an optional filter, combined with AND. ListFilter::new() and ListFilter::default() are the same empty filter (matches everything).

#![allow(unused)]
fn main() {
ListFilter::new()
    .owner_id(42)
    .owner_type("Generator")
    .owner_category(OwnerCategory::Component)
    .time_series_type(TimeSeriesType::SingleTimeSeries)
    .name("load")
    .name_glob("load_*")  // SQLite GLOB (case-sensitive, `*`/`?`); ANDed with .name
    .resolution(Duration::hours(1))   // impl Into<Period>
    .interval(Duration::hours(24))    // impl Into<Period>; forecasts only
    .features(features)   // subset match: rows must contain at least these pairs
}

AddRequest

The element type of add_time_series_bulk (and of BulkAdd::push), mirroring the add_time_series arguments plus an optional ext — an opaque, package-owned payload (typically JSON) stored verbatim. The series name lives on the TimeSeriesData object, not here.

#![allow(unused)]
fn main() {
pub struct AddRequest {
    pub owner_id: i64,
    pub owner_type: String,
    pub owner_category: OwnerCategory,
    pub data: TimeSeriesData,
    pub features: Features,
    pub units: Option<String>,
    pub ext: Option<String>,
}
}

BulkAdd

The buffered bulk-add session returned by Store::bulk_add. Requests accumulate in memory — no validation and no I/O until commit, which writes every array as a batch-sized block and inserts every association in one transaction, all-or-nothing. Dropping the session without committing discards the buffer and writes nothing.

#![allow(unused)]
fn main() {
impl BulkAdd<'_> {
    pub fn push(&mut self, request: AddRequest) -> &mut Self;   // prebuilt request
    pub fn add(                                                 // …or from its parts
        &mut self,
        owner_id: i64,
        owner_type: &str,
        owner_category: OwnerCategory,
        data: TimeSeriesData,
        features: Features,
        units: Option<String>,
    ) -> &mut Self;                                             // ext = None
    pub fn len(&self) -> usize;          // requests buffered so far
    pub fn is_empty(&self) -> bool;
    pub fn commit(self) -> Result<Vec<TimeSeriesKey>>;          // keys in push order
}
}

RequestedType

What Store::resolve_forecast_key is asked to match: one concrete stored type, or the abstract "deterministic" family, which a stored Deterministic or DeterministicSingleTimeSeries satisfies (the two never coexist for one series, so at most one can match).

#![allow(unused)]
fn main() {
pub enum RequestedType {
    Concrete(TimeSeriesType),
    AbstractDeterministic,
}

impl RequestedType {
    pub fn matches(self, concrete: TimeSeriesType) -> bool;
}
}

Association types

The row, predicate, and grouped-row types of the two association catalogs. All derive Serialize/Deserialize, so a binding can hand a whole filter or a whole batch across a language boundary as one JSON value. The rows also derive PartialEq/Eq/Hash, so they work in sets and as map keys.

#![allow(unused)]
fn main() {
// One attachment: a supplemental attribute carried by a component. Identity is
// the (component_id, attribute_id) pair; the type names are denormalized labels.
pub struct SupplementalAttributeAssociation {
    pub component_id: i64,
    pub component_type: String,
    pub attribute_id: i64,
    pub attribute_type: String,
}

// One directed edge between two components. Identity is the *ordered*
// (parent_id, child_id) pair, so the reversed pair is a different edge.
pub struct ParentChildAssociation {
    pub parent_id: i64,
    pub parent_type: String,
    pub child_id: i64,
    pub child_type: String,
}

// One grouped row of `supplemental_attribute_summary`; `count` is how many
// attachments share the (component_type, attribute_type) pair.
pub struct SupplementalAttributeSummaryRow {
    pub component_type: String,
    pub attribute_type: String,
    pub count: i64,
}
}

The two filters are builders, like ListFilter: every field is optional and the set ones are combined with AND, so ::new() / ::default() matches every row.

#![allow(unused)]
fn main() {
pub struct SupplementalAttributeFilter {
    pub component_id: Option<i64>,
    pub component_types: Option<Vec<String>>,
    pub attribute_id: Option<i64>,
    pub attribute_types: Option<Vec<String>>,
}

pub struct ParentChildFilter {
    pub parent_id: Option<i64>,
    pub parent_types: Option<Vec<String>>,
    pub child_id: Option<i64>,
    pub child_types: Option<Vec<String>>,
}
}

#![allow(unused)]
fn main() {
SupplementalAttributeFilter::new()
    .component_id(1)
    .component_types(["Generator", "Load"])   // concrete type names, rendered as SQL `IN (…)`
    .attribute_id(100)
    .attribute_types(["GeographicInfo"])

ParentChildFilter::new()
    .parent_id(1)
    .parent_types(["Generator"])
    .child_id(7)
    .child_types(["Bus"])
}

The *_types lists take concrete type names only; expanding an abstract type into its subtypes stays with the caller, where the type hierarchy lives. An empty list is an empty allow-list and matches nothing (as opposed to leaving the field unset, which matches everything).

Report and count types

#![allow(unused)]
fn main() {
pub struct TimeSeriesCounts {
    pub components_with_time_series: i64,
    pub static_time_series: i64,
    pub forecasts: i64,
}
// Owner- and array-oriented counts (`time_series_counts_detailed`). Unlike
// `TimeSeriesCounts`, the series counts here are deduplicated by array content
// and owners are split by category.
pub struct TimeSeriesCountsDetailed {
    pub components_with_time_series: i64,
    pub supplemental_attributes_with_time_series: i64,
    pub static_time_series_count: i64,
    pub forecast_count: i64,
}
// One grouped row of `static_summary` / `forecast_summary`; `count` is the
// number of associations in the group.
pub struct StaticSummaryRow {
    pub owner_type: String,
    pub owner_category: OwnerCategory,
    pub time_series_type: TimeSeriesType,
    pub name: String,
    pub initial_timestamp: Option<DateTime<Utc>>,
    pub resolution: Option<Period>,
    pub time_step_count: Option<i64>,
    pub count: i64,
}
pub struct ForecastSummaryRow {
    pub owner_type: String,
    pub owner_category: OwnerCategory,
    pub time_series_type: TimeSeriesType,
    pub name: String,
    pub initial_timestamp: Option<DateTime<Utc>>,
    pub resolution: Option<Period>,
    pub horizon: Option<Period>,
    pub interval: Option<Period>,
    pub window_count: Option<i64>,
    pub count: i64,
}
pub struct CompactionReport {
    pub slots_reclaimed: usize,
    pub datasets_dropped: usize,
    pub feature_sets_reclaimed: usize,
}
pub struct IntegrityReport { pub errors: Vec<String> }  // .ok() == errors.is_empty()
pub struct ForecastParameters {
    pub horizon: Option<Period>, pub interval: Option<Period>,
    pub count: Option<usize>, pub resolution: Option<Period>,
    pub initial_timestamp: Option<DateTime<Utc>>,
}
pub struct StaticConsistency {  // one row per resolution from check_static_consistency
    pub resolution: Period,
    pub initial_timestamp: DateTime<Utc>,
    pub length: usize,
}
}

Errors

#![allow(unused)]
fn main() {
pub type Result<T> = std::result::Result<T, TimeSeriesError>;

#[non_exhaustive] // match with a wildcard arm; new variants are not semver breaks
pub enum TimeSeriesError {
    NotFound,
    DuplicateTimeSeries,
    /// An association with the same identity already exists — the
    /// `(component_id, attribute_id)` pair of an attachment, or the ordered
    /// `(parent_id, child_id)` pair of an edge. The payload names the offending
    /// pair; it is a human-readable message, not a parseable encoding.
    DuplicateAssociation(String),
    InvalidParameter(String),
    IntegrityError(String),
    ReadOnlyStore,
    ConnectionError(String),
    IncompatibleForecast,
    /// The store on disk was written in a different, incompatible on-disk
    /// format. There is no in-place upgrade; see the file-format reference.
    IncompatibleFormat { found: String, expected: &'static str },
    Io(std::io::Error),
    Sqlite(rusqlite::Error),
    Serde(serde_json::Error),
}
}

StorageBackend Trait

The seam between Store and array storage. Implemented by MemoryBackend and NetCdfBackend. You rarely call it directly, but it documents the backend contract. It is not re-exported at the crate root — import it (and the backends) from the storage module:

#![allow(unused)]
fn main() {
use infrastore_core::storage::{MemoryBackend, NetCdfBackend, StorageBackend};
}

Every method below with a default is a performance override: the default is correct but naive, and NetCdfBackend implements a faster path (single hyperslab reads, whole-chunk block writes).

#![allow(unused)]
fn main() {
pub trait StorageBackend: Send + Sync {
    // --- required ---

    // `packed = true` column-packs same-shaped arrays (SingleTimeSeries / DST);
    // `packed = false` stores a standalone multi-dim variable (NonSequential, dense forecasts).
    // Idempotent on hash: returns `true` only if this call physically wrote new content.
    fn put_array(
        &mut self,
        hash: &[u8; 32],
        data: &TypedArray,
        resolution: Period,
        packed: bool,
    ) -> Result<bool>;
    fn get_array(&self, hash: &[u8; 32]) -> Result<TypedArray>;
    // Slice along axis 0 (the time axis); `range` end is exclusive.
    fn get_slice(&self, hash: &[u8; 32], range: Range<usize>) -> Result<TypedArray>;
    fn remove_array(&mut self, hash: &[u8; 32]) -> Result<()>;   // no-op if absent
    fn contains(&self, hash: &[u8; 32]) -> Result<bool>;
    fn compact(&mut self) -> Result<CompactionReport>;
    fn verify(&self) -> Result<IntegrityReport>;
    fn flush(&mut self) -> Result<()>;

    // --- provided (overridden by NetCdfBackend) ---

    // Write a block of same-shaped packed arrays at once (the bulk-add write path).
    // The returned Vec is aligned to `hashes`: `true` where this call wrote new content.
    fn put_packed_block(
        &mut self,
        hashes: &[[u8; 32]],
        arrays: &[&TypedArray],
        resolution: Period,
    ) -> Result<Vec<bool>>;
    // Read many whole arrays at once (`Store::bulk_read`): one decompress pass per dataset.
    fn read_arrays(&self, hashes: &[[u8; 32]]) -> Result<Vec<TypedArray>>;
    // One time step across co-located arrays (`StaticReader`); `out` is cleared, then
    // filled row-major as [column, *element_shape]. Reusing the buffer keeps the loop
    // allocation-free.
    fn read_index_into(&self, hashes: &[[u8; 32]], index: usize, out: &mut Vec<u8>) -> Result<()>;
    // Stored (dtype, shape), ideally without reading the data.
    fn array_shape(&self, hash: &[u8; 32]) -> Result<(Dtype, Vec<usize>)>;
    // One forecast window: the `window_index` slice along `count_axis`, that axis dropped.
    fn read_window_into(
        &self,
        hash: &[u8; 32],
        count_axis: usize,
        window_index: usize,
        out: &mut Vec<u8>,
    ) -> Result<()>;
    // `len` consecutive steps from `start` along axis 0 (backs DST window reads).
    fn read_range_into(
        &self,
        hash: &[u8; 32],
        start: usize,
        len: usize,
        out: &mut Vec<u8>,
    ) -> Result<()>;
    // The compression policy applied to writes; defaults to `Compression::None`
    // (in-memory backends never compress).
    fn compression(&self) -> Compression;
}
}

Hashing

In the hash module (infrastore_core::hash). array_hash and hash_hex are also re-exported at the crate root; features_hash is only reachable through the module.

#![allow(unused)]
fn main() {
pub fn array_hash(data: &TypedArray) -> [u8; 32];   // domain: dtype tag + shape + typed bytes
pub fn features_hash(features: &Features) -> [u8; 32];
pub fn hash_hex(hash: &[u8; 32]) -> String;
}

These define the cross-language content-addressing contract; see Content Addressing.

Constants

#![allow(unused)]
fn main() {
pub const DATA_FORMAT_VERSION: &str = "0.11.0";
}

Python API

The PyO3 binding is importable as the infrastore module (package infrastore). It is built as an abi3-py310 wheel, so one build runs on CPython 3.10 and newer.

from infrastore import (
    Store, SingleTimeSeries, NonSequentialTimeSeries, TimeSeriesKey,
    Deterministic, Probabilistic, Scenarios,
    TimeSeriesType, OwnerCategory,
    SupplementalAttributeAssociation, ParentChildAssociation,
    TimeSeriesError, NotFoundError, DuplicateTimeSeriesError,
    DuplicateAssociationError, InvalidParameterError, IntegrityError, ReadOnlyStoreError,
)

infrastore.__version__ reports the wheel version.

Array dtypes. The binding accepts and returns NumPy arrays of float64, float32, int64, int32, uint64, or bool; whatever dtype is given round-trips unchanged. Multi-dimensional arrays (a per-step element shape) are supported via the NumPy array's shape.

Store

Constructors

@classmethod
def create(
    cls,
    path: str | None = None,
    in_memory: bool = False,
    compression: str = "deflate",   # "deflate" or "none"
    compression_level: int = 3,     # 0–9, DEFLATE only
    shuffle: bool = True,           # byte-shuffle filter, DEFLATE only
) -> Store: ...

@classmethod
def open(cls, path: str, read_only: bool = False) -> Store: ...

• create(in_memory=True) — in-memory store; path and compression arguments are ignored.
• create(path=...) — writes path (NetCDF) and path + ".sqlite" (metadata).
• create(path=..., compression="none") — store arrays uncompressed; compression="deflate" with a compression_level / shuffle of your choice tunes the filter. The policy persists with the store and is reused on later appends. An unknown compression or out-of-range level raises InvalidParameterError.
• open(path, read_only=True) — read-only open; writes raise ReadOnlyStoreError.

The store is also a context manager: with Store.create(...) as store: closes it on exit. store.close() drops the underlying handle and releases its files; subsequent operations raise TimeSeriesError (it is idempotent). repr(store) shows the path (or in-memory), the read-only flag, and closed once closed.

Property

store.read_only -> bool

Methods

def add_time_series(
    self,
    owner_id: int,
    owner_type: str,
    owner_category: OwnerCategory,
    time_series: SingleTimeSeries | NonSequentialTimeSeries
        | Deterministic | Probabilistic | Scenarios,
    features: dict[str, int | float | bool | str] | None = None,
    units: str | None = None,
) -> TimeSeriesKey: ...
# `name` comes from the time_series object
# (e.g. SingleTimeSeries(..., name=...)), not from this call.

def add_time_series_bulk(self, items: list[dict]) -> list[TimeSeriesKey]: ...
# Each item dict mirrors add_time_series's parameters: required `owner_id`,
# `owner_type`, `owner_category`, `time_series`; optional `features`, `units`.
# All items commit in ONE metadata transaction (all-or-nothing), which is much
# faster than looping over add_time_series. Keys are returned in input order.

def transform_single_time_series(self, horizon: timedelta | str, interval: timedelta | str) -> int: ...

def get_time_series(
    self,
    key: TimeSeriesKey,
    time_range: tuple[datetime, datetime] | None = None,
) -> SingleTimeSeries | NonSequentialTimeSeries | Deterministic | Probabilistic | Scenarios: ...

def bulk_read(
    self,
    keys: list[TimeSeriesKey],
    *,
    time_range: tuple[datetime, datetime] | None = None,
) -> list[SingleTimeSeries | NonSequentialTimeSeries | Deterministic | Probabilistic | Scenarios]: ...
# `time_range` applies the same window to every key (default: each series in full).
# Results are returned in the same order as `keys`; an empty list of keys returns an empty list.

def remove_time_series(self, key: TimeSeriesKey) -> None: ...
def clear_time_series(
    self,
    owner_id: int | None = None,
    owner_category: OwnerCategory | None = None,
) -> int: ...
# Pass both owner_id and owner_category to clear one owner's series (the owner is
# the (owner_id, owner_category) pair); pass neither to clear the whole store.

def replace_owner(
    self,
    old_owner: int,
    new_owner: int,
    owner_category: OwnerCategory,
) -> int: ...
# Reassign every series owned by (old_owner, owner_category) to
# (new_owner, owner_category). Returns the number of associations moved.

def list_time_series(
    self,
    *,
    owner_id: int | None = None,
    owner_category: OwnerCategory | None = None,
    owner_type: str | None = None,
    time_series_type: TimeSeriesType | None = None,
    name: str | None = None,
    name_glob: str | None = None,   # SQLite GLOB pattern; ANDed with `name`
    resolution: timedelta | str | None = None,
    interval: timedelta | str | None = None,
    features: dict[str, int | float | bool | str] | None = None,
) -> list[dict]: ...

def list_array_groups(self, *, ...) -> list[dict]: ...
# Same keyword-only filter arguments as list_time_series; so do list_keys,
# list_names, list_owner_types, and remove_by_filter.

def get_time_series_keys(
    self,
    owner_id: int,
    owner_category: OwnerCategory,
) -> list[TimeSeriesKey]: ...
def has_time_series(self, key: TimeSeriesKey) -> bool: ...
def get_resolutions(self, time_series_type: TimeSeriesType | None = None) -> list[str]: ...
# resolutions are returned as ISO 8601 duration strings, e.g. "PT1H"
def get_time_series_counts(self) -> dict: ...
def get_forecast_parameters(self, *, resolution: str | None = None,
                            interval: str | None = None) -> dict: ...
def get_compression(self) -> dict: ...
def compact(self) -> dict: ...
def verify_integrity(self) -> dict: ...
# {"ok": bool, "errors": list[str]}
def flush(self) -> None: ...

Keyword-only arguments. Every optional argument in the binding is keyword-only (the * marker): filter kwargs, features=/units=/ext= on the add paths, time_range= on the read paths, and so on. Positional use raises TypeError. The wheel ships a infrastore.pyi stub, so IDEs and type checkers see the full signatures.

Return shapes

• add_time_series accepts a SingleTimeSeries, a NonSequentialTimeSeries, or a dense forecast object (Deterministic / Probabilistic / Scenarios) — see Forecasts. transform_single_time_series derives a DeterministicSingleTimeSeries from every stored SingleTimeSeries and returns the count transformed. get_time_series returns whichever matches the stored type.
• bulk_read returns one typed object per key, in the same order as keys (an empty key list returns an empty list). It is the bulk counterpart to get_time_series: packed SingleTimeSeries are read in one decompress-once pass per dataset instead of one read per key. Pass the keyword-only time_range=(start, end) to apply the same window to every key; by default each series comes back in full.
• list_time_series returns a list of dicts, each with the keys: owner_id, owner_type, owner_category, time_series_type, name, data_hash (hex string), length, resolution (ISO 8601 duration string, e.g. PT1H, or None), timestamps, features, units. timestamps is a list of RFC 3339 strings for non-sequential series and None otherwise. The features filter is a subset match — rows must contain at least the given pairs.
• list_array_groups accepts the same filters as list_time_series and groups the matching series by their underlying stored array. It returns a list of dicts, each with data_hash (hex string) and keys (a list of TimeSeriesKeys that resolve to that array). Keys sharing one dict share one deduplicated array.
• get_time_series_counts returns {"components_with_time_series": int, "static_time_series": int, "forecasts": int}.
• get_forecast_parameters returns {"horizon": str, "interval": str, "count": int, "resolution": str, "initial_timestamp": str}, where horizon, interval, and resolution are ISO 8601 duration strings (e.g. "PT1H") and initial_timestamp is an RFC 3339 string. Every value is None when the store holds no forecasts. The keyword-only resolution / interval arguments scope the query to forecasts matching that grid.
• get_compression returns {"compression": "deflate" | "none", "level": int, "shuffle": bool} — the policy the store was created with (restored from the file on open; "none" for in-memory).
• compact returns {"slots_reclaimed": int, "datasets_dropped": int, "feature_sets_reclaimed": int}. feature_sets_reclaimed counts content-addressed feature rows that no association referenced any more; see the file format.
• verify_integrity returns {"ok": bool, "errors": list[str]}; ok is True when the error list is empty. It checks stored arrays against their recorded hashes and does not inspect the SQLite catalog, so ok is not a statement about the store as a whole — see content addressing.
• get_time_series with time_range=(start, end) slices on the time axis; end is exclusive.

SingleTimeSeries

SingleTimeSeries(
    initial_timestamp: datetime,
    resolution: timedelta,
    data: numpy.ndarray,   # shape (length,) or (length, k1, ...)
    name: str,
)

Read-only properties: initial_timestamp -> datetime, resolution -> str (ISO 8601 duration, e.g. PT1H), length -> int, data -> numpy.ndarray, name -> str. The constructor accepts either a timedelta or an ISO 8601 duration string for resolution; the getter always returns the ISO string. name is a required association attribute (the same array may be stored under different names). It is read off the object by add_time_series and populated on get_time_series. The array's dtype (one of float64, float32, int64, int32, uint64, bool) and per-step element shape are preserved through a round-trip.

NonSequentialTimeSeries

NonSequentialTimeSeries(
    timestamps: list[datetime],
    data: numpy.ndarray,
    name: str,
)

Read-only properties: timestamps, length, data, and name. Timestamps must be timezone-aware, strictly increasing, and match the first data dimension. get_time_series returns this class for a non-sequential key.

TimeSeriesKey

Returned by add_time_series, add_time_series_bulk, and get_time_series_keys, and in the keys list of every list_array_groups row; not constructed directly. Read-only properties:

key.owner_id          -> int
key.owner_category    -> OwnerCategory
key.time_series_type  -> TimeSeriesType
key.name              -> str
key.resolution        -> str | None   # ISO 8601 duration, e.g. "PT1H"
key.interval          -> str | None   # ISO 8601 duration
key.features          -> dict[str, int | float | bool | str]

Enums

TimeSeriesType.SingleTimeSeries
TimeSeriesType.NonSequentialTimeSeries
TimeSeriesType.Deterministic
TimeSeriesType.DeterministicSingleTimeSeries
TimeSeriesType.Probabilistic
TimeSeriesType.Scenarios

OwnerCategory.Component
OwnerCategory.SupplementalAttribute

Forecasts

Dense forecasts are constructed as Deterministic, Probabilistic, or Scenarios objects and then passed to add_time_series. They are read back through get_time_series, which returns the matching object depending on the stored type (a DeterministicSingleTimeSeries is synthesized into a Deterministic on read). A DeterministicSingleTimeSeries is not added directly — derive one from stored SingleTimeSeries with transform_single_time_series. get_time_series_counts reports the forecast total under forecasts.

ts = Deterministic(initial_timestamp, resolution, horizon, interval, count, data, "load_fc")
key = store.add_time_series(42, "Generator", OwnerCategory.Component, ts, units="MW")

data is a NumPy array in the canonical shape for the forecast type, where H is horizon / resolution. As with SingleTimeSeries, every period argument (resolution, horizon, interval) accepts either a timedelta or an ISO 8601 duration string — the string form is required for calendar periods such as "P1M" — and the getters always return the ISO string. Every forecast also takes a required name (after data), exposed as a read-only property:

Deterministic

Deterministic(
    initial_timestamp: datetime,
    resolution: timedelta | str,
    horizon: timedelta | str,
    interval: timedelta | str,
    count: int,
    data: numpy.ndarray,
    name: str,
)

Read-only properties:

forecast.initial_timestamp -> datetime
forecast.resolution        -> str   # ISO 8601 duration, e.g. "PT1H"
forecast.horizon           -> str   # ISO 8601 duration
forecast.interval          -> str   # ISO 8601 duration
forecast.count             -> int
forecast.data              -> numpy.ndarray
forecast.name              -> str

Probabilistic

Probabilistic(
    initial_timestamp: datetime,
    resolution: timedelta | str,
    horizon: timedelta | str,
    interval: timedelta | str,
    count: int,
    percentiles: list[float],
    data: numpy.ndarray,
    name: str,
)

Same properties as Deterministic, plus:

forecast.percentiles -> list[float]

Scenarios

Scenarios(
    initial_timestamp: datetime,
    resolution: timedelta | str,
    horizon: timedelta | str,
    interval: timedelta | str,
    count: int,
    data: numpy.ndarray,   # leading axis is scenario_count
    name: str,
)

Same properties as Deterministic, plus:

forecast.scenario_count -> int

Readers

get_time_series returns one whole series or forecast. For the simulation access pattern — walk every timestamp and, at each, read the value of every matching series — use a reader instead. A reader is built once over a filter, pins one resolution, and reuses its output buffers so a tight loop allocates almost nothing. There are two: StaticReader for SingleTimeSeries, and ForecastReader for forecasts. Both share the lifecycle: build → inspect the layout once → *_read(when) in a loop → pull values per group/entry.

The builders and drivers live on Store:

def build_static_reader(
    self,
    resolution: timedelta | str,
    *,
    owner_id: int | None = None,
    owner_category: OwnerCategory | None = None,
    owner_type: str | None = None,
    name: str | None = None,
    name_glob: str | None = None,
    features: dict[str, int | float | bool | str] | None = None,
) -> StaticReader: ...
def static_read(self, reader: StaticReader, when: datetime) -> None: ...

def build_forecast_reader(
    self,
    time_series_type: TimeSeriesType,
    resolution: timedelta | str,
    *,
    owner_id: int | None = None,
    owner_category: OwnerCategory | None = None,
    owner_type: str | None = None,
    name: str | None = None,
    name_glob: str | None = None,
    features: dict[str, int | float | bool | str] | None = None,
) -> ForecastReader: ...
def forecast_read(self, reader: ForecastReader, when: datetime) -> None: ...

resolution is required on both builders (one resolution per reader). static_read / forecast_read fill the reader's buffers in place and return None; passing a when that is off the reader's grid or timeline raises InvalidParameterError.

StaticReader

Reads the value of every matching SingleTimeSeries at one timestamp. Results are columnar: series are partitioned into (dtype, element_shape) groups, and each group's values come back as one dense (num_columns, *element_shape) numpy array.

class StaticReader:
    def grid(self) -> dict: ...     # {"initial_timestamp": rfc3339 str, "resolution": ISO str, "length": int}
    def groups(self) -> list[dict]: ...  # each: {"dtype": str, "element_shape": list[int], "keys": list[TimeSeriesKey]}
    def timestamps(self) -> list[datetime]: ...   # every timestamp on the grid, in order
    def group_values(self, index: int) -> numpy.ndarray: ...  # last read of group `index`

All matched series must share one grid (initial_timestamp + length); the build validates this and raises on divergence, so there is no presence mask — every column has a value at every valid timestamp. group_values(i) returns a (num_columns, *element_shape) array whose column j corresponds to groups()[i]["keys"][j]; it is empty until the first static_read.

reader = store.build_static_reader(timedelta(hours=1))
grid = reader.grid()
groups = reader.groups()
start = datetime.fromisoformat(grid["initial_timestamp"])
for ts in reader.timestamps():
    store.static_read(reader, ts)
    for i, g in enumerate(groups):
        vals = reader.group_values(i)   # column j ↔ g["keys"][j]

ForecastReader

Reads the forecast window at one timestamp for every matching forecast of one type. The build filter must name a forecast type and pin a resolution; a Deterministic reader is abstract and also includes DeterministicSingleTimeSeries (read into identical (horizon, *element_shape) windows). All matched forecasts must share one window timeline (initial_timestamp + interval + count).

time_series_type must be one of the concrete forecast types — Deterministic, DeterministicSingleTimeSeries, Probabilistic, or Scenarios; any other raises InvalidParameterError.

class ForecastReader:
    def timeline(self) -> dict: ...   # {"initial_timestamp": rfc3339 str, "resolution": ISO str, "interval": ISO str, "count": int, "time_series_type": str}
    def entries(self) -> list[TimeSeriesKey]: ...   # per-entry keys, in order (parallel to entry_values)
    def timestamps(self) -> list[datetime]: ...     # every window-start timestamp, in order
    def entry_values(self, index: int) -> numpy.ndarray: ...  # last read of entry `index`
    def num_slots(self) -> int: ...          # deduplicated window slots (physical reads per forecast_read)
    def entry_slot(self, index: int) -> int: ...  # 0-based slot backing entry `index`

Valid read timestamps are initial_timestamp + k·interval for k in range(count) (each names the window forecast from that instant). entry_values(i) returns the window backing entries()[i], shaped (horizon, *element_shape) for Deterministic / DeterministicSingleTimeSeries, (num_percentiles, horizon, *element_shape) for Probabilistic, and (scenario_count, horizon, *element_shape) for Scenarios; it is empty until the first forecast_read.

reader = store.build_forecast_reader(TimeSeriesType.Deterministic, timedelta(hours=1))
tl = reader.timeline()
entries = reader.entries()
for ts in reader.timestamps():
    store.forecast_read(reader, ts)
    for i, key in enumerate(entries):
        window = reader.entry_values(i)   # window for key's owner

Window-read deduplication. Forecasts that share one backing array and read plan — deduplicated identical data, or several DeterministicSingleTimeSeries over one SingleTimeSeries — collapse to a single window slot. forecast_read performs one backend (.nc) read per slot, not per entry, so a forecast shared by N owners is read once per timestamp. num_slots() is that physical read count (<= len(entries())), and entry_slot(i) (0-based) identifies the slot backing entry i; entries that share data report the same slot. Group by slot to also materialize each unique window only once on the Python side:

store.forecast_read(reader, ts)
windows: dict[int, numpy.ndarray] = {}
for i, key in enumerate(entries):
    window = windows.setdefault(reader.entry_slot(i), reader.entry_values(i))

Associations

Two catalogs of relationships between entities the store does not otherwise model. Both are independent of time series: removing a time series never removes an association, and vice versa (there are no foreign keys and no cascade — both endpoints live in the caller's object graph, so a cascade could never fire), so a caller that wants both makes both calls.

Every query in both families takes the same keyword-only filter arguments as its family's has_* method. All are optional and ANDed; with none set they match every row, which is what makes a no-filter export and an add_* import a round trip. The *_types arguments are lists of concrete type names, matched as SQL IN (…): expanding an abstract type into its subtypes stays in Python, where the type hierarchy lives, and an empty list matches nothing — unlike omitting the argument, which matches everything. Every remove_* returns the number removed; removing nothing returns 0 rather than raising.

Supplemental-attribute associations

Which supplemental attributes are attached to which components. One attribute may be attached to many components.

SupplementalAttributeAssociation(
    component_id: int,
    component_type: str,
    attribute_id: int,
    attribute_type: str,
)

Read-only properties: component_id, component_type, attribute_id, attribute_type. The object is hashable and compares structurally, so attachments work in sets and as dict keys. In the catalog, though, identity is only the (component_id, attribute_id) pair — the type names are denormalized labels carried for filtering — so re-attaching the same pair under different type names raises DuplicateAssociationError.

def add_supplemental_attribute_association(
    self, association: SupplementalAttributeAssociation
) -> None: ...
def add_supplemental_attribute_associations(
    self, associations: list[SupplementalAttributeAssociation]
) -> int: ...
# All-or-nothing: a duplicate anywhere in the batch rolls the whole batch back.
# Returns the number inserted; the import half of the round trip whose export is
# list_supplemental_attribute_associations() with no filter.

def has_supplemental_attribute_association(
    self,
    *,
    component_id: int | None = None,
    component_types: list[str] | None = None,
    attribute_id: int | None = None,
    attribute_types: list[str] | None = None,
) -> bool: ...

def list_supplemental_attribute_associations(
    self, *, ...
) -> list[SupplementalAttributeAssociation]: ...
def list_supplemental_attribute_ids(self, *, ...) -> list[int]: ...
def list_components_with_attributes(self, *, ...) -> list[int]: ...
def remove_supplemental_attribute_associations(self, *, ...) -> int: ...
def count_supplemental_attribute_associations(self, *, ...) -> int: ...
def count_supplemental_attributes(self, *, ...) -> int: ...
def count_components_with_attributes(self, *, ...) -> int: ...
# Every `...` above is the same keyword-only filter as has_supplemental_attribute_association.

def replace_supplemental_attribute_component_id(self, old_id: int, new_id: int) -> int: ...

def supplemental_attribute_counts_by_type(self) -> list[tuple[str, int]]: ...
def supplemental_attribute_summary(self) -> list[dict]: ...

• list_supplemental_attribute_associations returns rows in insertion order, so exporting with no filter and importing the result with add_supplemental_attribute_associations is a round trip.
• list_supplemental_attribute_ids returns the distinct attribute ids of the matching rows, ascending — the attributes attached to component c with component_id=c. list_components_with_attributes is the other end: the components carrying attribute a with attribute_id=a. count_supplemental_attributes and count_components_with_attributes are those two queries counted, and count_supplemental_attribute_associations counts the matching rows themselves.
• replace_supplemental_attribute_component_id moves every attachment from component old_id to new_id, returning the rows updated, and raises DuplicateAssociationError if new_id already carries one of the attributes being moved.
• supplemental_attribute_counts_by_type returns [(attribute_type, count), …] ordered by type; supplemental_attribute_summary returns one dict per distinct pair with keys component_type, attribute_type, count, ordered by attribute type then component type.

from infrastore import SupplementalAttributeAssociation, Store

store = Store.create(in_memory=True)
store.add_supplemental_attribute_association(
    SupplementalAttributeAssociation(1, "Generator", 100, "GeographicInfo")
)
store.add_supplemental_attribute_association(
    SupplementalAttributeAssociation(2, "Load", 100, "GeographicInfo")
)

store.list_supplemental_attribute_ids(component_id=1)     # -> [100]
store.list_components_with_attributes(attribute_id=100)   # -> [1, 2]

store.remove_supplemental_attribute_associations(component_id=1)
# -> 1; any time series of component 1 are untouched

Parent/child associations

Directed edges between components — a generator (parent) wired to a bus (child), say. Both endpoints are always components; an attribute cannot appear here.

ParentChildAssociation(
    parent_id: int,
    parent_type: str,
    child_id: int,
    child_type: str,
)

Read-only properties: parent_id, parent_type, child_id, child_type; hashable and structurally comparable like the attachment object. In the catalog, identity is the ordered (parent_id, child_id) pair, so the reversed pair is a different edge, while repeating the same ordered pair under different type names raises DuplicateAssociationError. There is no relationship-kind column, so one ordered pair may be related at most once.

This family is deliberately narrower than the supplemental one — no counts-by-type and no grouped summary — because there is no consumer for them yet; both are additive if one appears.

def add_parent_child_association(self, association: ParentChildAssociation) -> None: ...
def add_parent_child_associations(self, associations: list[ParentChildAssociation]) -> int: ...
# All-or-nothing, like the supplemental bulk add; returns the number inserted.

def has_parent_child_association(
    self,
    *,
    parent_id: int | None = None,
    parent_types: list[str] | None = None,
    child_id: int | None = None,
    child_types: list[str] | None = None,
) -> bool: ...

def list_parent_child_associations(self, *, ...) -> list[ParentChildAssociation]: ...
def list_children(self, *, ...) -> list[int]: ...
def list_parents(self, *, ...) -> list[int]: ...
def remove_parent_child_associations(self, *, ...) -> int: ...
def count_parent_child_associations(self, *, ...) -> int: ...
# Every `...` above is the same keyword-only filter as has_parent_child_association.

def replace_parent_child_component_id(self, old_id: int, new_id: int) -> int: ...

• list_parent_child_associations returns rows in insertion order, so a no-filter export and an add_parent_child_associations import round-trip.
• list_children returns the distinct child ids of the matching edges, ascending — the children of component p with parent_id=p; list_parents is the other end, the parents of component c with child_id=c.
• replace_parent_child_component_id rewrites old_id to new_id on both ends of every edge, returning the rows updated, and raises DuplicateAssociationError if the rewrite would duplicate an edge new_id already has.

from infrastore import ParentChildAssociation, Store

store = Store.create(in_memory=True)
store.add_parent_child_association(ParentChildAssociation(1, "Generator", 7, "Bus"))
# The reversed pair is a different edge, not a duplicate.
store.add_parent_child_association(ParentChildAssociation(7, "Bus", 1, "Generator"))

store.list_children(parent_id=1)   # -> [7]
store.list_parents(child_id=7)     # -> [1]

store.remove_parent_child_associations(parent_types=["Bus"])   # -> 1

Neither association catalog is exposed over the gRPC server or the infrastore CLI.

Exceptions

All inherit from TimeSeriesError:

A malformed ISO 8601 period string raises InvalidParameterError (inside the hierarchy). Only a period argument that is neither a timedelta nor a str raises a plain TypeError, which except TimeSeriesError will not catch.

Feature-value typing note: because bool is a subtype of int in Python, the binding checks bool first, so True/False features are stored as booleans, not integers.

init_tracing

def init_tracing(filter: str) -> None: ...

Initialize the Rust tracing subscriber with the given EnvFilter directive string. Examples:

init_tracing("debug")                            # all targets at DEBUG
init_tracing("infrastore_core=debug")     # store core only
init_tracing("warn,infrastore_core=trace") # warn globally, trace the core

Silently no-ops if a subscriber is already registered (including the one auto-initialized from RUST_LOG at module import). See the Python developer guide for usage examples.

Julia API

The Julia package is InfraStore.jl (module InfraStore); it wraps the C ABI cdylib. The library is resolved from the INFRASTORE_LIB environment variable (development builds), or from the InfraStore_jll binary package when installed.

using InfraStore

Exported names: Store, SingleTimeSeries, NonSequentialTimeSeries, Deterministic, DeterministicSingleTimeSeries, AbstractDeterministic, Probabilistic, Scenarios, TimeSeriesKey, OwnerCategory, Component, SupplementalAttribute, add_time_series!, AddBatch, add_time_series_bulk!, get_time_series, bulk_read, get_time_series_keys, key_info, list_keys, list_array_groups, remove_time_series!, has_time_series, get_counts, counts_by_type, num_distinct_arrays, time_series_counts, list_owner_ids, static_summary, forecast_summary, SupplementalAttributeAssociation, add_supplemental_attribute_association!, add_supplemental_attribute_associations!, has_supplemental_attribute_association, list_supplemental_attribute_associations, list_supplemental_attribute_ids, list_components_with_attributes, remove_supplemental_attribute_associations!, replace_supplemental_attribute_component_id!, count_supplemental_attribute_associations, count_supplemental_attributes, count_components_with_attributes, supplemental_attribute_counts_by_type, supplemental_attribute_summary, ParentChildAssociation, add_parent_child_association!, add_parent_child_associations!, has_parent_child_association, list_parent_child_associations, list_children, list_parents, remove_parent_child_associations!, replace_parent_child_component_id!, count_parent_child_associations, get_forecast_parameters, check_static_consistency, get_resolutions, get_compression, verify_integrity, compact!, get_metadata, get_forecast_metadata, get_array_by_hash, count_array_references, open_store, flush!, clear!, replace_owner!, transform_single_time_series!, has_typed, remove_typed!, copy_time_series!, close!, persist!, StaticReader, build_static_reader, static_grid, static_groups, static_read!, static_values, ForecastReader, build_forecast_reader, forecast_timeline, forecast_entries, forecast_num_slots, forecast_read!, forecast_values, init_logging.

Constructors

Store(; in_memory::Bool=true, path::Union{Nothing,AbstractString}=nothing,
        compression::Union{Symbol,AbstractString}=:deflate,
        compression_level::Integer=3, shuffle::Bool=true) -> Store
open_store(path::AbstractString; read_only::Bool=false) -> Store

• Store() — in-memory store.
• Store(in_memory=false, path="system.nc") — persists to system.nc plus system.nc.sqlite.
• compression=:none stores arrays uncompressed; :deflate (default) applies DEFLATE at compression_level (0–9) with optional byte shuffle. The policy is persisted with the store and reused on later appends; it is ignored for in-memory stores. An unknown compression throws ArgumentError.
• open_store(path; read_only=true) — opens an existing on-disk pair.

The store registers a finalizer; close it eagerly with close!(store).

Types

Each struct carries the association name (required) and an optional ext. Every constructor takes name as the positional after data and ext= as a keyword — e.g. SingleTimeSeries(initial, resolution, data, name; ext=nothing).

Every data-carrying struct is parameterized {T,N} on the element type and dimensionality of its value array; {T,N} is inferred from data by the constructor (an AbstractArray argument — a view or a range — is normalized to a concrete Array{T,N}).

struct SingleTimeSeries{T,N}
    initial_timestamp :: DateTime
    resolution        :: Period          # e.g. Hour(1), Millisecond(500)
    data              :: Array{T,N}      # any element type; dim 1 = time
    name              :: String          # required association name
    ext      :: Union{Nothing,String}
end
SingleTimeSeries(initial_timestamp, resolution, data, name; ext=nothing)

struct NonSequentialTimeSeries{T,N}
    timestamps   :: Vector{DateTime}     # strictly increasing; one per row of dim 1
    data         :: Array{T,N}
    name         :: String
    ext :: Union{Nothing,String}
end
NonSequentialTimeSeries(timestamps, data, name; ext=nothing)

struct Deterministic{T,N} <: AbstractDeterministic
    initial_timestamp :: DateTime
    resolution        :: Period
    horizon           :: Period
    interval          :: Period
    count             :: Int
    data              :: Array{T,N}      # (H, count, element_dims...)
    name              :: String
    ext      :: Union{Nothing,String}
end
Deterministic(initial_timestamp, resolution, horizon, interval, count, data, name;
              ext=nothing)

struct Probabilistic{T,N}
    initial_timestamp :: DateTime
    resolution        :: Period
    horizon           :: Period
    interval          :: Period
    count             :: Int
    percentiles       :: Vector{Float64}
    data              :: Array{T,N}      # (num_percentiles, H, count, element_dims...)
    name              :: String
    ext      :: Union{Nothing,String}
end
Probabilistic(initial_timestamp, resolution, horizon, interval, count, percentiles, data, name;
              ext=nothing)

struct Scenarios{T,N}
    initial_timestamp :: DateTime
    resolution        :: Period
    horizon           :: Period
    interval          :: Period
    count             :: Int
    scenario_count    :: Int             # set from size(data, 1) by the constructor
    data              :: Array{T,N}      # (scenario_count, H, count, element_dims...)
    name              :: String
    ext      :: Union{Nothing,String}
end
Scenarios(initial_timestamp, resolution, horizon, interval, count, data, name;
          ext=nothing)         # note: scenario_count is NOT a constructor argument

# Marker type; never constructed and with no materialized struct. Derived via
# transform_single_time_series! and read back as a Deterministic. Surfaces as a
# key's time_series_type.
abstract type DeterministicSingleTimeSeries <: AbstractDeterministic end

# Request-only supertype of Deterministic and DeterministicSingleTimeSeries.
# Pass it as the requested type to get_time_series to read whichever of the two
# is stored (see Reading forecast values). It is NOT a valid build_forecast_reader
# type — that call takes a concrete forecast type.
abstract type AbstractDeterministic end

mutable struct Store
    handle :: Ptr{Cvoid}
end

mutable struct TimeSeriesKey
    handle :: Ptr{Cvoid}                 # opaque; finalized automatically
end

@enum OwnerCategory begin
    Component             = 0
    SupplementalAttribute = 1
end

ext is an opaque, package-owned payload (typically JSON) the binding can use to reconstruct a domain object on read; the store stores it verbatim and never interprets it. add_time_series! reads name off the object (it is not a call argument), so the same array can be stored under different names; its ext= keyword defaults to the object's ext. data keeps its Julia element type: the binding maps T to a stored dtype (Float64, Float32, Int64, Int32, UInt64, Bool) and converts to row-major bytes on the way down.

Static Series

add_time_series!(
    store::Store, owner_id, owner_type, owner_category::OwnerCategory,
    ts::SingleTimeSeries;
    features::AbstractDict = Dict(), units = nothing,
    ext = ts.ext,
) -> TimeSeriesKey

add_time_series!(
    store::Store, owner_id, owner_type, owner_category::OwnerCategory,
    ts::NonSequentialTimeSeries;
    features = Dict(), units = nothing,
    ext = ts.ext,
) -> TimeSeriesKey

get_time_series(store::Store, key::TimeSeriesKey; time_range=nothing) -> SingleTimeSeries
get_time_series(SingleTimeSeries, store::Store, key::TimeSeriesKey;
               time_range=nothing) -> SingleTimeSeries
get_time_series(NonSequentialTimeSeries, store::Store, key::TimeSeriesKey;
               time_range=nothing) -> NonSequentialTimeSeries

get_time_series(SingleTimeSeries, store, owner_id, owner_category, name;
               resolution=nothing, features=Dict(), time_range=nothing) -> SingleTimeSeries
get_time_series(NonSequentialTimeSeries, store, owner_id, owner_category, name;
               resolution=nothing, features=Dict(), time_range=nothing) -> NonSequentialTimeSeries

time_range = (start::DateTime, stop::DateTime) (exclusive end) slices the read on every form.

owner_id is an integer identifier (Int64) and owner_category (Component / SupplementalAttribute) completes the owner identity — the owner is the pair (owner_id, owner_category). features is serialized to JSON and must contain only JSON-scalar values (Int, Float64, Bool, String). Pass the type as the first argument to get_time_series to read a non-sequential series back.

get_time_series supports two unified calling conventions for every type: pass the TimeSeriesKey returned by add_time_series! (key-based), or pass owner_id, owner_category, name plus optional resolution / features keywords (attribute-based, the same addressing used by get_metadata / has_time_series / remove_time_series!). Both forms return the same struct. The bare get_time_series(store, key) remains a convenience alias for SingleTimeSeries.

To read every series' value at one timestamp in a loop (the simulation pattern), use a StaticReader rather than calling get_time_series per series.

Bulk reads

bulk_read(store::Store, keys::AbstractVector{TimeSeriesKey};
          time_range::Union{Nothing,Tuple{DateTime,DateTime}}=nothing) -> Vector{SingleTimeSeries}
# time_range slices every series to that window (default: each series in full)

Reads many whole SingleTimeSeries in one call, returning one per key in the same order. Each packed dataset is read and decompressed once instead of per series, so this is the efficient way to load many complete series (exploration, plotting). Every key must identify a SingleTimeSeries; an empty key vector returns an empty vector without touching the store.

series = bulk_read(store, keys)   # keys :: Vector{TimeSeriesKey}

Bulk Adds

batch = AddBatch()
add_time_series!(batch, owner_id, owner_type, owner_category, ts; ...)  # any series type
add_time_series_bulk!(store::Store, batch::AddBatch) -> Vector{TimeSeriesKey}

AddBatch accepts the same add_time_series! methods as Store (every series and forecast type) but only accumulates the requests; add_time_series_bulk! commits the whole batch in one metadata transaction, which is much faster than per-item adds when ingesting many series. The submit is all-or-nothing: on error nothing is committed. The batch is drained by the call in either case and may be reused. length(batch) returns the number of pending requests.

Attribute-based lookups

get_metadata(store, owner_id, owner_category::OwnerCategory, name;
             resolution=nothing, features=Dict()) -> NamedTuple
has_time_series(store, owner_id, owner_category::OwnerCategory, name;
                resolution=nothing, features=Dict()) -> Bool
remove_time_series!(store, owner_id, owner_category::OwnerCategory, name;
                    resolution=nothing, features=Dict()) -> Nothing

owner_category (Component / SupplementalAttribute) is required: the owner identity is the pair (owner_id, owner_category), so a component and a supplemental attribute may share a numeric owner_id and remain distinct. get_metadata returns (initial_timestamp, resolution, length, data_hash, dtype, ext, units, element_shape, features), where data_hash is the 32-byte content hash, element_shape is the per-timestep shape tuple (empty for scalar elements), and features is the feature dictionary. It throws NotFoundError if absent. get_forecast_metadata and get_probabilistic_metadata carry the same element_shape/features fields (there, element_shape is the stored array's trailing dims after its first axis).

get_array_by_hash(store, data_hash::Vector{UInt8}, ::Type{T}=Float64) -> Vector{T}

Fetches the flattened array for a 32-byte content hash, decoded as element type T. Combine with get_metadata (for the dtype and shape) to read values without holding a TimeSeriesKey.

Key-based variants

has_time_series(store, key::TimeSeriesKey) -> Bool
remove_time_series!(store, key::TimeSeriesKey) -> Nothing
remove_time_series!(store, keys::Vector{TimeSeriesKey}) -> Int

The vector form removes every key in one all-or-nothing transaction and returns the count; a single missing key aborts the whole batch.

Enumerating keys

get_time_series_keys(store, owner_id, owner_category::OwnerCategory) -> Vector{TimeSeriesKey}
key_info(key::TimeSeriesKey) -> NamedTuple
# (owner_id, owner_category, name, time_series_type, resolution, features)

get_time_series_keys returns one key per stored association for the owner identified by the (owner_id, owner_category) pair, including DeterministicSingleTimeSeries rows derived by transform_single_time_series! — the way to read a transform-derived forecast by key (it returns no key of its own). The keys are opaque; key_info inspects one. time_series_type is the actual Julia type (SingleTimeSeries, NonSequentialTimeSeries, Deterministic, DeterministicSingleTimeSeries, Probabilistic, or Scenarios), as in InfrastructureSystems.jl — pass it straight to get_time_series. features is a Dict (empty when none) that round-trips the JSON-scalar feature values. For example:

for k in get_time_series_keys(store, 12345, Component)
    info = key_info(k)
    series = get_time_series(info.time_series_type, store, k)
end

key_info also returns owner_category (the Component / SupplementalAttribute half of the owner identity) alongside owner_id.

Reading a DeterministicSingleTimeSeries (by key or attributes) returns a Deterministic, since the type has no materialized form.

Forecasts

Dense forecasts are constructed as Deterministic, Probabilistic, or Scenarios structs (see Types) and added through the generic add_time_series!. Each struct wraps a native AbstractArray of any element type and dimensionality — the binding derives the stored dtype and dims and converts to row-major bytes, just like the static add_time_series! (see the data model for the conventional shapes).

The forecast name comes from the struct, e.g. Deterministic(initial, resolution, horizon, interval, count, data, name).

add_time_series!(
    store, owner_id, owner_type, owner_category::OwnerCategory,
    ts::Deterministic;
    features=Dict(), units=nothing, ext=nothing,
) -> TimeSeriesKey

add_time_series!(
    store, owner_id, owner_type, owner_category::OwnerCategory,
    ts::Probabilistic;
    features=Dict(), units=nothing, ext=nothing,
) -> TimeSeriesKey

add_time_series!(
    store, owner_id, owner_type, owner_category::OwnerCategory,
    ts::Scenarios;
    features=Dict(), units=nothing, ext=nothing,
) -> TimeSeriesKey

A DeterministicSingleTimeSeries is not added directly. Derive one from every stored SingleTimeSeries (sharing the backing array) with:

transform_single_time_series!(store, horizon::Period, interval::Period;
                              owner_category::Union{Nothing,OwnerCategory}=nothing,
                              resolution::Union{Nothing,Period}=nothing) -> Int  # number transformed

count is derived from each series' length. owner_category restricts the transform to one owner category (both are transformed when it is nothing); resolution restricts it to the SingleTimeSeries at that resolution.

has_typed, remove_typed!, and copy_time_series! address a series by its ts_type integer code (0 = SingleTimeSeries, 1 = NonSequentialTimeSeries, 2 = Deterministic, 3 = DeterministicSingleTimeSeries, 4 = Probabilistic, 5 = Scenarios):

has_typed(store, owner_id, owner_category, name, ts_type::Integer;
          resolution=nothing, interval=nothing, features=Dict()) -> Bool
remove_typed!(store, owner_id, owner_category, name, ts_type::Integer;
              resolution=nothing, interval=nothing, features=Dict())

interval (a Period) pins the forecast interval — the only way to disambiguate two forecasts of one owner/name/type that differ solely by interval (e.g. day-ahead vs intra-day). Without it such a lookup is ambiguous and errors.

Copying an association

copy_time_series!(store, owner_id, owner_category, name, ts_type::Integer,
                  dst_owner_id, dst_owner_type::AbstractString;
                  new_name=nothing, resolution=nothing, interval=nothing,
                  features=Dict()) -> Nothing

Copies the time series identified by the source attributes onto dst_owner_id (of Julia/domain type dst_owner_type), optionally renaming it to new_name. Arrays are content-addressed, so this writes only a new association row against the same underlying array: no data is duplicated and the stored time series type is preserved — a DeterministicSingleTimeSeries stays a DST, whereas a read-then-write copy through get_time_series / add_time_series! would materialize it into a dense Deterministic. The copy keeps the source's owner_category. Throws if the destination already holds a matching series.

copy_time_series!(store, 42, Component, "load", 0, 43, "Generator")   # SingleTimeSeries → owner 43

Reading forecast values

The type-dispatched get_time_series(Type, …) functions return the corresponding struct, whose data field is a decoded N-dimensional Julia array (reshaped to the type's logical shape, with native Julia indexing). Pass time_range = (start::DateTime, end::DateTime) (exclusive end) to select a window sub-range.

Like the static readers, forecasts support both calling conventions: attribute-based (owner_id, owner_category, name plus optional resolution / interval / features) or key-based (the TimeSeriesKey returned by add_time_series!).

get_time_series(Deterministic, store, owner_id, owner_category, name;
                resolution=nothing, interval=nothing, features=Dict(),
                time_range=nothing) -> Deterministic
get_time_series(Deterministic, store, key::TimeSeriesKey; time_range=nothing) -> Deterministic
                # data shape: (H, count, element_dims...)

get_time_series(DeterministicSingleTimeSeries, store, owner_id, owner_category, name;
                resolution=nothing, interval=nothing, features=Dict(),
                time_range=nothing) -> Deterministic
get_time_series(DeterministicSingleTimeSeries, store, key::TimeSeriesKey;
                time_range=nothing) -> Deterministic

get_time_series(AbstractDeterministic, store, owner_id, owner_category, name;
                resolution=nothing, interval=nothing, features=Dict(),
                time_range=nothing) -> Deterministic

get_time_series(Probabilistic, store, owner_id, owner_category, name;
                resolution=nothing, interval=nothing, features=Dict(),
                time_range=nothing) -> Probabilistic
get_time_series(Probabilistic, store, key::TimeSeriesKey; time_range=nothing) -> Probabilistic
                # data shape: (num_percentiles, H, count, element_dims...)

get_time_series(Scenarios, store, owner_id, owner_category, name;
                resolution=nothing, interval=nothing, features=Dict(),
                time_range=nothing) -> Scenarios
get_time_series(Scenarios, store, key::TimeSeriesKey; time_range=nothing) -> Scenarios
                # data shape: (scenario_count, H, count, element_dims...)

interval (a Period) pins the forecast interval; it is the only way to address one of two forecasts under the same owner/name/type that differ solely by interval. A read that leaves it nothing when two such forecasts exist raises InvalidParameterError.

Concrete types vs. the AbstractDeterministic family

The concrete request types match only themselves:

• get_time_series(Deterministic, …) matches a directly-stored Deterministic and does not match a DeterministicSingleTimeSeries — asking for Deterministic when only a transformed DST exists raises NotFoundError. There is no fallback.
• get_time_series(DeterministicSingleTimeSeries, …) matches only a transformed DST. Since the type has no materialized struct, it returns a Deterministic.

To read whichever of the two is stored under an identity, request the family type AbstractDeterministic (attribute-based only — a key already names its exact stored type):

transform_single_time_series!(store, Hour(4), Hour(2))
fc = get_time_series(AbstractDeterministic, store, 400, Component, "dst")   # a Deterministic

The Rust core resolves the family in a single call — there is no guess-and-retry. A genuine miss raises NotFoundError, and an identity matching both a Deterministic and a DeterministicSingleTimeSeries is ambiguous and errors (request a concrete type instead).

The key-based readers carry the exact stored type in the key, so the question does not arise: a DeterministicSingleTimeSeries key reads back as a Deterministic.

Alternatively, use get_metadata to obtain the data_hash, then get_array_by_hash for the raw flattened array.

For the per-timestamp simulation access pattern (walk the timeline, read every series at each instant) prefer a reader — see Readers below.

Readers (per-timestamp iteration)

get_time_series returns a whole series or forecast struct. For the simulation access pattern — walk every timestamp and, at each, read the value of every series — use a reader instead. A reader is built once over a filter, pins one resolution, and reuses output buffers that each read overwrites in place, so a tight loop allocates almost nothing. There are two: StaticReader for SingleTimeSeries, and ForecastReader for forecasts. Both follow the same lifecycle: build → inspect the layout once → *_read!(t) in a loop → pull values per group/entry.

StaticReader

Reads the value of every matching SingleTimeSeries at one timestamp. Results are columnar: series are partitioned into (dtype, element_shape) groups, and each group's values come back as one dense (num_columns, element_dims...) array.

build_static_reader(store; resolution::Period, owner_id=nothing,
                    owner_category=nothing, name=nothing, features=Dict()) -> StaticReader

static_grid(reader)   -> NamedTuple  # (initial_timestamp::DateTime, resolution::Period, length::Int)
static_groups(reader) -> Vector{StaticGroup}  # each: .dtype, .element_shape, .keys
static_read!(reader, t::DateTime) -> reader   # fills buffers; errors if t is off the grid
static_values(reader, group_index::Integer) -> Array
       # (num_columns, element_dims...); column j is static_groups(reader)[group_index].keys[j]

All matched series must share one grid (initial_timestamp + length); the build validates this and errors on divergence, so there is no presence mask — every column has a value at every valid timestamp.

reader = build_static_reader(store; resolution = Hour(1))
grid = static_grid(reader)
for k in 0:(grid.length - 1)
    static_read!(reader, grid.initial_timestamp + grid.resolution * k)
    for (gi, g) in enumerate(static_groups(reader))
        vals = static_values(reader, gi)   # column j ↔ g.keys[j]
    end
end

ForecastReader

Reads the forecast window at one timestamp for every matching forecast of one type. The build filter must name a forecast type and pin a resolution; a Deterministic reader is abstract and also includes DeterministicSingleTimeSeries (read into identical [H, *E] windows). All matched forecasts must share one window timeline (initial_timestamp + interval + count).

time_series_type must be one of the four concrete forecast types — Deterministic, DeterministicSingleTimeSeries, Probabilistic, or Scenarios. Any other type (including AbstractDeterministic, which is a get_time_series request type only) raises InvalidParameterError; a Deterministic reader already covers the family.

build_forecast_reader(store, time_series_type::Type; resolution::Period,
                      owner_id=nothing, owner_category=nothing, name=nothing,
                      features=Dict()) -> ForecastReader

forecast_timeline(reader)  -> NamedTuple
       # (initial_timestamp::DateTime, resolution::Period, interval::Period, count::Int)
forecast_entries(reader)   -> Vector{ForecastEntry}  # each: .dtype, .window_shape, .key, .slot
forecast_num_slots(reader) -> Int                    # physical reads per timestamp (see below)
forecast_read!(reader, t::DateTime) -> reader        # fills buffers; errors if t is off the timeline
forecast_values(reader, entry_index::Integer) -> Array  # window of size .window_shape

Valid read timestamps are initial_timestamp + k·interval for k in 0:count-1 (each names the window forecast from that instant). A window's shape is [H, *E] for Deterministic / DeterministicSingleTimeSeries, [num_percentiles, H, *E] for Probabilistic, and [scenario_count, H, *E] for Scenarios.

reader = build_forecast_reader(store, Deterministic; resolution = Hour(1))
tl = forecast_timeline(reader)
for k in 0:(tl.count - 1)
    forecast_read!(reader, tl.initial_timestamp + tl.interval * k)
    for (i, e) in enumerate(forecast_entries(reader))
        window = forecast_values(reader, i)   # shape e.window_shape, for e.key's owner
    end
end

Window-read deduplication

Forecasts that reference the same backing array and read plan — deduplicated identical data, or several DeterministicSingleTimeSeries over one SingleTimeSeries — collapse to a single window slot. forecast_read! performs one backend (.nc) read per slot, not per entry, so a forecast shared by N owners is read once per timestamp. forecast_num_slots(reader) is that physical read count (≤ length(forecast_entries(reader))), and every ForecastEntry.slot (0-based) identifies the slot backing that entry; entries that share data report the same slot. Group entries by slot to also materialize each unique window only once on the Julia side:

forecast_read!(reader, t)
windows = Dict{Int, Any}()
for (i, e) in enumerate(forecast_entries(reader))
    window = get!(() -> forecast_values(reader, i), windows, e.slot)   # materialize once per slot
    # apply `window` to e.key's owner
end

Store-Wide Operations

get_counts(store) -> NamedTuple   # (components_with_time_series, static_time_series, forecasts)
counts_by_type(store) -> Vector{NamedTuple}   # (time_series_type, count) per stored type
num_distinct_arrays(store) -> Int   # distinct content hashes; shared arrays count once
time_series_counts(store) -> NamedTuple   # distinct owners per category + distinct arrays per kind
list_owner_ids(store, owner_category; time_series_type=nothing, resolution=nothing) -> Vector{Int}
list_array_groups(store; owner_id=nothing, owner_category=nothing, time_series_type=nothing,
                  name=nothing, resolution=nothing, features=Dict()) -> Vector{NamedTuple}
                                  # list_keys rows + `data_hash`; group by it to find shared arrays
count_array_references(store, data_hash::Vector{UInt8}) -> NamedTuple  # (sts, dst) refs to a 32-byte hash
static_summary(store) -> Vector{NamedTuple}   # grouped static rows with a `count`; build your own table
forecast_summary(store) -> Vector{NamedTuple}   # grouped forecast rows with a `count`
get_forecast_parameters(store; resolution=nothing, interval=nothing) -> NamedTuple  # (horizon, interval, count, resolution, initial_timestamp); fields `nothing` when none match
check_static_consistency(store; resolution=nothing) -> Vector{NamedTuple}  # one (resolution, initial_timestamp, length) per resolution present (empty when none); throws if the series at one resolution disagree
get_resolutions(store; time_series_type=nothing) -> Vector{Period}  # distinct resolutions, in the core's stored (lexical-by-ISO) order
get_compression(store) -> NamedTuple  # (compression=:deflate|:none, level, shuffle); restored from file on open
verify_integrity(store) -> Int    # number of integrity errors; 0 == intact
compact!(store) -> Nothing
flush!(store) -> Nothing          # sync to disk; afterwards .nc and .sqlite can be copied
clear!(store; owner_id=nothing, owner_category=nothing) -> Nothing
                                  # both `nothing`: remove every series in the store.
                                  # Scope to one owner by passing BOTH keywords — they identify the
                                  # (owner_id, owner_category) pair. `owner_id` without
                                  # `owner_category` throws ArgumentError.
replace_owner!(store, old_owner_id, new_owner_id, owner_category::OwnerCategory) -> Int
                                  # reassign one owner's series to a new id (same category); count moved
close!(store) -> Nothing

list_keys(store; owner_id=nothing, owner_category=nothing, time_series_type=nothing,
          name=nothing, resolution=nothing, features=Dict()) -> Vector{NamedTuple}

list_keys lists the key of every stored series as NamedTuples (identity plus the per-type descriptive snapshot: initial_timestamp, resolution, length, horizon, interval, count, features; fields that do not apply to a key's type are nothing). All six filters are optional and independent, and combine as a conjunction; with none set the whole store is listed:

• owner_id, owner_category — scope to one owner.
• time_series_type — a INFRASTORE_TYPE_* integer code (0 = SingleTimeSeries … 5 = Scenarios), not a Julia type. (The time_series_type field on a returned row is the Julia type.)
• name — exact association name.
• resolution — a Period.
• features — match keys whose features include all the given entries (subset match).

Physical storage detail (data_hash, ext, percentiles) is not on a key — read it via get_metadata / get_forecast_metadata.

list_array_groups takes the same six filters and returns the same row fields as list_keys, but each row additionally carries data_hash — the 64-character lowercase hex content hash of the array the row resolves to (note this is a hex String, whereas get_metadata returns the hash as a 32-byte Vector{UInt8}). Rows that share a stored array share their data_hash: both deduplicated identical arrays and a SingleTimeSeries together with any DeterministicSingleTimeSeries derived from it. Group rows by data_hash to discover which time series share their underlying data — the foundation for reading a shared series once (see Window-read deduplication). It is one catalog query (the hash is read off each metadata row); there are no per-row get_metadata round-trips.

count_array_references(store, data_hash) returns (; sts, dst) — how many SingleTimeSeries and DeterministicSingleTimeSeries associations reference the given 32-byte hash, across all owners. Because a DST shares its backing SingleTimeSeries array, a caller uses these counts to decide whether removing a SingleTimeSeries would orphan a derived DST.

Associations

Two catalogs of relationships between entities the store does not otherwise model, replacing the association tables IS3.jl used to keep itself. Both are independent of time series: there are no foreign keys and no cascade (both endpoints live in the caller's object graph, so a cascade could never fire), so removing a time series never removes an association and vice versa; a caller that wants both makes both calls.

Every query in a family takes that family's four optional keyword filters, ANDed; with none set they match every row, which is what makes a bare list_* call a whole-catalog export that the matching add_*! re-imports unchanged. The *_types keywords take a vector of concrete type names, matched as SQL IN (…): expanding an abstract type into its subtypes stays on the Julia side, where the type hierarchy lives, and an empty vector matches nothing, unlike omitting the keyword, which matches everything. Every remove_*! returns the number of rows removed; removing nothing is 0, not an error.

Supplemental-attribute associations

Which supplemental attributes are attached to which components. One attribute may be attached to many components.

struct SupplementalAttributeAssociation
    component_id::Int64
    component_type::String
    attribute_id::Int64
    attribute_type::String
end

SupplementalAttributeAssociation overloads ==, hash, and show (a compact SupplementalAttributeAssociation(Generator 1 <- GeographicInfo 100)), so attachments work as Dict/Set members. In the catalog, identity is only the (component_id, attribute_id) pair — the type names are denormalized labels carried for filtering — so re-attaching the same pair under different type names throws DuplicateAssociationError.

add_supplemental_attribute_association!(store, association::SupplementalAttributeAssociation) -> Nothing
add_supplemental_attribute_associations!(store, associations::AbstractVector{SupplementalAttributeAssociation}) -> Int
                                  # one all-or-nothing transaction; count inserted
has_supplemental_attribute_association(store; filters...) -> Bool
list_supplemental_attribute_associations(store; filters...) -> Vector{SupplementalAttributeAssociation}
                                  # insertion order
list_supplemental_attribute_ids(store; filters...) -> Vector{Int}
                                  # distinct attribute ids, ascending
list_components_with_attributes(store; filters...) -> Vector{Int}
                                  # distinct component ids, ascending
remove_supplemental_attribute_associations!(store; filters...) -> Int   # count removed
replace_supplemental_attribute_component_id!(store, old_id, new_id) -> Int   # rows updated
count_supplemental_attribute_associations(store; filters...) -> Int
count_supplemental_attributes(store; filters...) -> Int
count_components_with_attributes(store; filters...) -> Int
supplemental_attribute_counts_by_type(store) -> Vector{NamedTuple}   # (type, count), by type
supplemental_attribute_summary(store) -> Vector{NamedTuple}
                                  # (component_type, attribute_type, count), by attribute then component type

The four keyword filters are component_id, component_types, attribute_id, and attribute_types.

list_supplemental_attribute_ids is "the attributes attached to this component" when component_id is set; list_components_with_attributes is the other end, "the components carrying this attribute" when attribute_id is set. count_supplemental_attributes and count_components_with_attributes are those two queries counted, and count_supplemental_attribute_associations counts the matching rows themselves.

replace_supplemental_attribute_component_id! moves every attachment from component old_id to new_id, and throws DuplicateAssociationError if new_id already carries one of the attributes being moved.

store = Store(in_memory=true)
add_supplemental_attribute_association!(
    store, SupplementalAttributeAssociation(1, "Generator", 100, "GeographicInfo"))
add_supplemental_attribute_association!(
    store, SupplementalAttributeAssociation(2, "Load", 100, "GeographicInfo"))

list_supplemental_attribute_ids(store; component_id=1)     # [100]
list_components_with_attributes(store; attribute_id=100)   # [1, 2]

remove_supplemental_attribute_associations!(store; component_id=1)
# 1; component 1's time series are untouched

Parent/child associations

Directed edges between components — a generator (parent) wired to a bus (child), say. Both endpoints are always components; an attribute cannot appear here.

struct ParentChildAssociation
    parent_id::Int64
    parent_type::String
    child_id::Int64
    child_type::String
end

ParentChildAssociation overloads ==, hash, and show (a compact ParentChildAssociation(Generator 1 -> Bus 7)) the same way. In the catalog, identity is the ordered (parent_id, child_id) pair, so the reversed pair is a different edge, while repeating the same ordered pair under different type names throws DuplicateAssociationError. There is no relationship-kind column, so one ordered pair may be related at most once.

This family is deliberately narrower than the supplemental one — no counts-by-type and no grouped summary — because there is no consumer for them yet; both are additive if one appears.

add_parent_child_association!(store, association::ParentChildAssociation) -> Nothing
add_parent_child_associations!(store, associations::AbstractVector{ParentChildAssociation}) -> Int
                                  # one all-or-nothing transaction; count inserted
has_parent_child_association(store; filters...) -> Bool
list_parent_child_associations(store; filters...) -> Vector{ParentChildAssociation}
                                  # insertion order
list_children(store; filters...) -> Vector{Int}   # distinct child ids, ascending
list_parents(store; filters...) -> Vector{Int}    # distinct parent ids, ascending
remove_parent_child_associations!(store; filters...) -> Int   # count removed
replace_parent_child_component_id!(store, old_id, new_id) -> Int   # rows updated
count_parent_child_associations(store; filters...) -> Int

The four keyword filters are parent_id, parent_types, child_id, and child_types.

replace_parent_child_component_id! rewrites old_id to new_id on both ends of every edge, and throws DuplicateAssociationError if the rewrite would duplicate an edge new_id already has.

store = Store(in_memory=true)
add_parent_child_association!(store, ParentChildAssociation(1, "Generator", 7, "Bus"))
# The reversed pair is a different edge, not a duplicate.
add_parent_child_association!(store, ParentChildAssociation(7, "Bus", 1, "Generator"))

list_children(store; parent_id=1)   # [7]
list_parents(store; child_id=7)     # [1]

remove_parent_child_associations!(store; parent_types=["Bus"])   # 1

Neither association catalog is exposed over the gRPC server or the infrastore CLI.

Errors

All subtype TimeSeriesException:

The message text comes from the FFI layer's thread-local error buffer.

Base Interface

The package overloads Base so the wrapped types behave like native Julia values:

• == and hash on TimeSeriesKey delegate to the Rust core's identity semantics (owner, category, type, name, resolution, interval, features), so keys work as Dict/Set members.
• show renders compact one-liners for Store, TimeSeriesKey, and the five value types.
• length, eltype, getindex, and iterate on SingleTimeSeries / NonSequentialTimeSeries delegate to the wrapped data array (element count, not time steps, for multi-dimensional values). Forecast types define length = window count.
• Do-block forms guarantee close! even on throw:

Store(in_memory=true) do store
    add_time_series!(store, 1, "Generator", Component, ts)
end

open_store(path; read_only=true) do store
    get_metadata(store, 1, Component, "load")
end

Time and Resolution Conversions

• DateTime is converted to/from Unix milliseconds at the boundary.
• resolution is passed as a Period and converted to an ISO-8601 duration string; reads return resolution as a Period (Millisecond for fixed durations).

Tracing

init_logging(level::AbstractString = "") -> Int32  # the FFI status code

Initialize the Rust tracing subscriber. level is an EnvFilter directive string such as "debug" or "infrastore_core=debug". Pass an empty string (the default) to read RUST_LOG; if that variable is also unset, no output is produced.

The subscriber is initialized at most once per process — subsequent calls are no-ops. The module's __init__ hook calls init_logging("") automatically when RUST_LOG is set, so the common case requires no code change:

export RUST_LOG=infrastore_core=debug
julia --project=. myscript.jl

For programmatic control without environment variables:

using InfraStore
init_logging("infrastore_core=debug")

See Julia developer guide for usage examples and a table of available span targets.

C ABI

infrastore-ffi compiles a cdylib (libinfrastore_ffi.{dylib,so,dll}) exposing a C-compatible API over Store. The header crates/infrastore-ffi/include/infrastore.h is generated by cbindgen — do not hand-edit it. The Julia binding is the primary consumer.

Conventions

• Return codes. Every function returns int32_t. INFRASTORE_OK (0) is success; any other value is an error. Retrieve the human-readable detail with infrastore_last_error_message.
• Opaque handles. InfraStore and InfraStoreKey are incomplete struct types; you only ever hold pointers. Free them with infrastore_store_free and infrastore_key_free.
• Out-parameters. Results are written through caller-provided pointers (**out, *out_len, …).
• Caller-owned buffers. Returned arrays come back through an out-pointer + length and must be freed: f64 percentile buffers (double **) with infrastore_buffer_free_f64, raw element-byte buffers (uint8_t **) with infrastore_buffer_free_u8, timestamp and shape buffers (int64_t **) with infrastore_buffer_free_i64, and the u64 dims buffer from infrastore_store_get_forecast (uint64_t **) with infrastore_buffer_free_u64.
• Typed arrays. Add functions take the element dtype code (0=f64, 1=f32, 2=i64, 3=i32, 4=u64, 5=bool), ndims plus a dims_ptr shape array ([length, k1, …]), and the raw little-endian data_ptr of data_byte_len bytes. Reads return the dtype code and a raw byte buffer for the caller to decode; infrastore_store_get_single follows the same dtype-generic convention, returning the dtype, shape, and raw element bytes.
• owner_category is an int32_t (0 = Component, 1 = SupplementalAttribute) passed to the add functions. The owner identity is the pair (owner_id, owner_category) — a component and a supplemental attribute may share a numeric owner_id and stay distinct — and the category is recorded with the association at add time.
• ext is an optional opaque, package-owned payload (typically JSON) passed verbatim to the add functions and stored uninterpreted.
• Strings are null-terminated UTF-8. Optional string arguments (ext, features_json, units) accept NULL.
• Features are passed as a JSON object string whose values are int / float / bool / string.
• Timestamps are int64_t Unix milliseconds. Resolutions/horizons/intervals are ISO-8601 duration strings (e.g. "PT1H", "P1M", "P1Y"); a NULL (or empty) string means unset. On output they are owned char * strings — free each with infrastore_string_free.

Status Codes

Lifecycle

int32_t infrastore_store_create(const char *path, bool in_memory, struct InfraStore **out);
/* compression_kind: 0 = none, 1 = DEFLATE (deflate_level 0-9 + shuffle). */
int32_t infrastore_store_create_with_compression(const char *path, bool in_memory, uint8_t compression_kind,
                                         uint8_t deflate_level, bool shuffle, struct InfraStore **out);
int32_t infrastore_store_open(const char *path, bool read_only, struct InfraStore **out);
void    infrastore_store_free(struct InfraStore *handle);
void    infrastore_key_free(struct InfraStoreKey *key);
/* Frees any owned `char *` this library returns (resolutions, horizons, intervals, …). */
void    infrastore_string_free(char *s);
void    infrastore_buffer_free_f64(double *ptr, uint64_t len);
void    infrastore_buffer_free_u8(uint8_t *ptr, uint64_t len);
void    infrastore_buffer_free_i64(int64_t *ptr, uint64_t len);
void    infrastore_buffer_free_u64(uint64_t *ptr, uint64_t len);

SingleTimeSeries

int32_t infrastore_store_add_single(struct InfraStore *handle,
                            int64_t owner_id, const char *owner_type,
                            int32_t owner_category,           /* 0=Component, 1=SupplementalAttribute */
                            const char *name,
                            int64_t initial_ts_unix_ms, const char *resolution,  /* ISO-8601 */
                            int32_t dtype, uint64_t ndims, const uint64_t *dims_ptr,
                            const uint8_t *data_ptr, uint64_t data_byte_len,
                            const char *ext,         /* optional */
                            const char *features_json,        /* optional */
                            const char *units,                /* optional */
                            struct InfraStoreKey **out_key);          /* owned; infrastore_key_free */

int32_t infrastore_store_get_single(const struct InfraStore *handle, const struct InfraStoreKey *key,
                            int64_t *out_initial_ts_unix_ms,
                            char **out_resolution,            /* ISO-8601; infrastore_string_free */
                            int32_t *out_dtype,
                            int64_t **out_shape, uint64_t *out_shape_len,  /* infrastore_buffer_free_i64 */
                            uint8_t **out_data, uint64_t *out_data_byte_len);  /* infrastore_buffer_free_u8 */

int32_t infrastore_store_remove(struct InfraStore *handle, const struct InfraStoreKey *key);
/* All-or-nothing batched remove: on any error (including one missing key)
   nothing is removed. *out_removed receives the count on success. */
int32_t infrastore_store_remove_bulk(struct InfraStore *handle,
                             const struct InfraStoreKey *const *keys, uint64_t len,
                             uint64_t *out_removed);
int32_t infrastore_store_has(const struct InfraStore *handle, const struct InfraStoreKey *key, bool *out_present);

/* Key identity comparison and hashing (consistent with each other; the hash is
   stable only within one process). */
int32_t infrastore_key_eq(const struct InfraStoreKey *a, const struct InfraStoreKey *b, bool *out_eq);
int32_t infrastore_key_identity_hash(const struct InfraStoreKey *key, uint64_t *out_hash);

NonSequentialTimeSeries

infrastore_store_add_non_sequential takes an explicit int64_t Unix-millisecond timestamp array alongside the typed data buffer. infrastore_store_get_non_sequential returns owned timestamp, shape, and raw-byte buffers (free with infrastore_buffer_free_i64, infrastore_buffer_free_i64, and infrastore_buffer_free_u8) plus the dtype code. The shape is the full [length, *element_shape] array shape (the first dim is time, so callers can recover an N-dimensional per-step element shape). out_ext is the optional opaque package-owned payload copied into a caller-allocated buffer of ext_cap bytes, with the full length reported in out_ext_len — probe with a NULL/zero-capacity buffer first, then call again with a buffer of that length (as infrastore_store_get_single documents for its shape and ext outputs).

int32_t infrastore_store_add_non_sequential(struct InfraStore *handle,
                                    int64_t owner_id, const char *owner_type,
                                    int32_t owner_category, const char *name,
                                    const int64_t *timestamps_unix_ms, uint64_t timestamps_len,
                                    int32_t dtype, uint64_t ndims, const uint64_t *dims_ptr,
                                    const uint8_t *data_ptr, uint64_t data_byte_len,
                                    const char *ext, const char *features_json,
                                    const char *units,
                                    struct InfraStoreKey **out_key);

int32_t infrastore_store_get_non_sequential(const struct InfraStore *handle, const struct InfraStoreKey *key,
                                    int64_t **out_timestamps, uint64_t *out_timestamps_len,
                                    int32_t *out_dtype,
                                    int64_t **out_shape, uint64_t *out_shape_len,  /* infrastore_buffer_free_i64 */
                                    uint8_t **out_data, uint64_t *out_data_byte_len,  /* infrastore_buffer_free_u8 */
                                    char *out_ext, uint64_t ext_cap,
                                    uint64_t *out_ext_len);

Attribute-Based Access

Resolve a series by its attributes instead of a key handle. A NULL/empty resolution means unset.

int32_t infrastore_store_get_metadata(const struct InfraStore *handle,
                              int64_t owner_id,
                              int32_t owner_category,         /* 0=Component, 1=SupplementalAttribute */
                              const char *name,
                              const char *resolution,         /* ISO-8601; NULL = unset */
                              const char *features_json,
                              int64_t *out_initial_ts_unix_ms,
                              char **out_resolution,          /* ISO-8601; infrastore_string_free */
                              uint64_t *out_length, uint8_t *out_data_hash, /* 32-byte buffer */
                              int32_t *out_dtype,
                              char *out_ext, uint64_t ext_cap,
                              uint64_t *out_ext_len,
                              char *out_units, uint64_t units_cap,
                              uint64_t *out_units_len,        /* empty => unset */
                              uint64_t *out_element_shape, uint64_t element_shape_cap,
                              uint64_t *out_element_shape_len, /* 0 => scalar elements */
                              char *out_features_json, uint64_t features_json_cap,
                              uint64_t *out_features_json_len); /* "{}" => none */

int32_t infrastore_store_has_by_attrs(const struct InfraStore *handle,
                              int64_t owner_id, int32_t owner_category, const char *name,
                              const char *resolution, const char *features_json, bool *out_present);

/* Name-less existence query: true iff owner_id has any series, optionally
   filtered to a single ts_type (use_type selects whether ts_type applies). */
int32_t infrastore_store_has_for_owner(const struct InfraStore *handle,
                               int64_t owner_id, int32_t owner_category,
                               int32_t ts_type, bool use_type, bool *out_present);

int32_t infrastore_store_remove_by_attrs(struct InfraStore *handle,
                                 int64_t owner_id, int32_t owner_category, const char *name,
                                 const char *resolution, const char *features_json);

int32_t infrastore_store_get_array_by_hash(const struct InfraStore *handle, const uint8_t *data_hash,
                                   int32_t *out_dtype,
                                   uint8_t **out_data, uint64_t *out_byte_len); /* infrastore_buffer_free_u8 */

/* Count SingleTimeSeries (*out_sts) and DeterministicSingleTimeSeries (*out_dst)
   associations that reference the 32-byte content hash data_hash, across all
   owners — one catalog query to decide whether removing a SingleTimeSeries would
   orphan a DST that shares its backing array. */
int32_t infrastore_store_count_array_references(const struct InfraStore *handle, const uint8_t *data_hash,
                                        uint64_t *out_sts, uint64_t *out_dst);

/* Build a key handle from attributes for any ts_type, so an attribute-addressed
   caller can reuse the key-based readers (infrastore_store_get_single,
   infrastore_store_get_non_sequential, infrastore_store_get_forecast_by_key). A NULL resolution
   or interval means unset. The returned key is owned; free it with infrastore_key_free. */
int32_t infrastore_make_key_from_attrs(int64_t owner_id, int32_t owner_category, const char *name,
                               int32_t ts_type,
                               const char *resolution, const char *interval,  /* ISO-8601; NULL = unset */
                               const char *features_json,
                               struct InfraStoreKey **out_key);

/* List every key for owner_id (one per association, including derived
   DeterministicSingleTimeSeries rows). Ownership is two-tiered: free each InfraStoreKey
   with infrastore_key_free, then free the array with infrastore_keys_buffer_free. An owner with
   no series yields *out_keys = NULL and *out_len = 0. */
int32_t infrastore_store_get_time_series_keys(const struct InfraStore *handle,
                                      int64_t owner_id, int32_t owner_category,
                                      struct InfraStoreKey ***out_keys, uint64_t *out_len);
void    infrastore_keys_buffer_free(struct InfraStoreKey **ptr, uint64_t len);

/* Inspect an opaque key: type code, resolution (an owned ISO-8601 string, NULL
   when unset — free with infrastore_string_free), owner id, owner category (0 = Component,
   1 = SupplementalAttribute), name, and features (a JSON object string, "{}" when
   empty — the shape the attribute-addressed entry points accept). The name and
   features strings use probe-then-fetch — pass NULL buffers / 0 caps to read the
   required lengths, then call again with len+1-byte buffers. */
int32_t infrastore_key_attributes(const struct InfraStoreKey *key,
                          int32_t *out_type, char **out_resolution,  /* ISO-8601; infrastore_string_free */
                          int64_t *out_owner_id, int32_t *out_owner_category,
                          char *name_buf, uint64_t name_cap, uint64_t *out_name_len,
                          char *features_buf, uint64_t features_cap, uint64_t *out_features_len);

/* Read an association's name by key, resolved through the stored metadata (the
   per-association name is not carried on the key itself). name_buf uses the
   probe-then-fetch convention (like infrastore_key_attributes): call with name_buf = NULL,
   name_cap = 0 to learn *out_name_len, then again with a name_len+1-byte buffer. */
int32_t infrastore_store_get_association(const struct InfraStore *handle, const struct InfraStoreKey *key,
                                 char *name_buf, uint64_t name_cap, uint64_t *out_name_len);

infrastore_store_get_metadata + infrastore_store_get_array_by_hash is the read path used by bindings that maintain their own key objects (such as an InfrastructureSystems.jl-side store). infrastore_make_key_from_attrs bridges the two addressing styles: it materializes a InfraStoreKey from attributes that the key-based read functions accept directly. infrastore_store_get_time_series_keys enumerates an owner's keys (the only way to obtain a key for a transform-derived DeterministicSingleTimeSeries), and infrastore_key_attributes reads back an opaque key's type, name, features, and addressing so the caller can pick the matching key-based reader.

Forecasts

The forecast types are created and read through the C ABI. ts_type is the TimeSeriesType discriminant — 0 = SingleTimeSeries, 1 = NonSequentialTimeSeries, 2 = Deterministic, 3 = DeterministicSingleTimeSeries, 4 = Probabilistic, 5 = Scenarios — plus one request-only sentinel, INFRASTORE_TYPE_ABSTRACT_DETERMINISTIC (100), described below. Forecast values are dtype-generic raw little-endian byte buffers with explicit dimensions — the same dtype, ndims, dims_ptr, data_ptr, data_byte_len convention as the static add functions (see the data model for the conventional shapes); the store records the windowing parameters in metadata and does not interpret the layout. A DeterministicSingleTimeSeries (3) is read like any other forecast but cannot be written through infrastore_store_add_forecast — it is derived via infrastore_store_transform_single_time_series.

The C ABI keeps these per-type add functions as low-level transport (the higher-level bindings layer the generic add_time_series over them). infrastore_store_add_forecast accepts only ts_type 2 = Deterministic or 5 = Scenarios; infrastore_store_add_probabilistic adds the percentile vector for Probabilistic. DeterministicSingleTimeSeries (3) is not addable through infrastore_store_add_forecast — it errors and directs you to infrastore_store_transform_single_time_series, which derives a DeterministicSingleTimeSeries from every stored SingleTimeSeries (sharing the backing array) and writes the number transformed to *out_count:

int32_t infrastore_store_add_forecast(struct InfraStore *handle,
                              int64_t owner_id, const char *owner_type, int32_t owner_category,
                              const char *name, int32_t ts_type,
                              int64_t initial_ts_unix_ms,
                              const char *resolution, const char *horizon, const char *interval,  /* ISO-8601 */
                              uint64_t count,
                              int32_t dtype, uint64_t ndims, const uint64_t *dims_ptr,
                              const uint8_t *data_ptr, uint64_t data_byte_len,
                              const char *ext,          /* optional */
                              const char *features_json, const char *units,
                              struct InfraStoreKey **out_key);

int32_t infrastore_store_add_probabilistic(struct InfraStore *handle,
                                   int64_t owner_id, const char *owner_type,
                                   int32_t owner_category, const char *name,
                                   int64_t initial_ts_unix_ms,
                                   const char *resolution, const char *horizon, const char *interval,  /* ISO-8601 */
                                   uint64_t count,
                                   const double *percentiles_ptr, uint64_t percentiles_len,
                                   int32_t dtype, uint64_t ndims, const uint64_t *dims_ptr,
                                   const uint8_t *data_ptr, uint64_t data_byte_len,
                                   const char *ext,     /* optional */
                                   const char *features_json, const char *units,
                                   struct InfraStoreKey **out_key);

int32_t infrastore_store_transform_single_time_series(struct InfraStore *handle,
                                              const char *horizon, const char *interval,  /* ISO-8601 */
                                              int32_t owner_category,  /* <0 = all categories; else 0=Component, 1=SupplementalAttribute */
                                              const char *resolution,  /* NULL = all resolutions */
                                              uint64_t *out_count);

infrastore_store_get_forecast is the forecast read function: it resolves a forecast by attributes and returns the decoded data buffer, its out-dimensions, the metadata, and — for Probabilistic — the percentile vector. A stored DeterministicSingleTimeSeries is synthesized into Deterministic values (its dense windows are materialized from the backing SingleTimeSeries), but it remains a distinct stored type for addressing purposes.

Its ts_type argument is a read request, not merely a stored-type filter:

• A concrete code (2 = Deterministic, 3 = DeterministicSingleTimeSeries, 4 = Probabilistic, 5 = Scenarios) matches only that exact stored type. Passing 2 does not find a stored DeterministicSingleTimeSeries, and passing 3 does not find a stored Deterministic. The non-forecast codes 0 and 1 are rejected with INFRASTORE_ERR_INVALID_PARAMETER.
• INFRASTORE_TYPE_ABSTRACT_DETERMINISTIC (100) is a request-only sentinel — never a stored type — for the AbstractDeterministic family: it matches a stored Deterministic or a DeterministicSingleTimeSeries. This is the only way to address a deterministic forecast whose concrete type the caller does not know in advance. The catalog resolves the family authoritatively (no client-side guess-and-retry) and reports the concrete type that matched through *out_matched_type. If both concrete types share the identity the request is ambiguous and returns INFRASTORE_ERR_INVALID_PARAMETER; a genuine miss returns the usual not-found error.

*out_matched_type always receives the concrete TimeSeriesType that was matched — so a stored DeterministicSingleTimeSeries reports 3, never 2, and the 100 sentinel is never returned.

When time_range_present is true, only the windows whose start timestamp falls in [time_range_start_ms, time_range_end_ms) are returned; pass false to retrieve all windows. The caller owns the returned buffers: free *out_data with infrastore_buffer_free_u8, *out_dims with infrastore_buffer_free_u64, *out_percentiles (non-NULL only for Probabilistic) with infrastore_buffer_free_f64, and each of the *out_resolution / *out_horizon / *out_interval ISO-8601 strings with infrastore_string_free.

int32_t infrastore_store_get_forecast(const struct InfraStore *handle,
                              int64_t owner_id, int32_t owner_category,
                              const char *name,
                              int32_t ts_type,  /* 2..5, or INFRASTORE_TYPE_ABSTRACT_DETERMINISTIC (100) */
                              const char *resolution, const char *interval,  /* ISO-8601 filters; NULL = none */
                              const char *features_json,
                              bool time_range_present,
                              int64_t time_range_start_ms, int64_t time_range_end_ms,
                              int64_t *out_initial_ts_unix_ms,
                              char **out_resolution, char **out_horizon, char **out_interval,  /* ISO-8601; infrastore_string_free */
                              uint64_t *out_count, uint64_t *out_scenario_count,
                              uint64_t *out_ndims, uint64_t **out_dims,  /* dims: infrastore_buffer_free_u64 */
                              int32_t *out_dtype,
                              uint8_t **out_data, uint64_t *out_data_byte_len, /* infrastore_buffer_free_u8 */
                              double **out_percentiles, uint64_t *out_percentiles_len, /* infrastore_buffer_free_f64 */
                              int32_t *out_matched_type);  /* concrete matched TimeSeriesType */

infrastore_store_get_forecast_by_key is the key-based counterpart: it takes a InfraStoreKey handle (the type comes from the key) instead of the owner_id, name, ts_type, resolution, interval, features_json arguments, and produces identical outputs with the same buffer-ownership rules. Because the key already names the concrete stored type there is no family to resolve: *out_matched_type is simply the key's type (a DeterministicSingleTimeSeries key reports 3, though its values are decoded into a dense Deterministic window array). There is no key-level equivalent of the 100 sentinel — use infrastore_store_get_forecast for a family request.

int32_t infrastore_store_get_forecast_by_key(const struct InfraStore *handle, const struct InfraStoreKey *key,
                                     bool time_range_present,
                                     int64_t time_range_start_ms, int64_t time_range_end_ms,
                                     int64_t *out_initial_ts_unix_ms,
                                     char **out_resolution, char **out_horizon, char **out_interval,  /* ISO-8601; infrastore_string_free */
                                     uint64_t *out_count, uint64_t *out_scenario_count,
                                     uint64_t *out_ndims, uint64_t **out_dims, /* infrastore_buffer_free_u64 */
                                     int32_t *out_dtype,
                                     uint8_t **out_data, uint64_t *out_data_byte_len, /* infrastore_buffer_free_u8 */
                                     double **out_percentiles, uint64_t *out_percentiles_len, /* infrastore_buffer_free_f64 */
                                     int32_t *out_matched_type);

The metadata-only accessors read the windowing parameters and content hash without decoding the array. infrastore_store_get_probabilistic_metadata additionally returns the percentile vector (free it with infrastore_buffer_free_f64):

int32_t infrastore_store_get_forecast_metadata(const struct InfraStore *handle,
                                       int64_t owner_id, int32_t owner_category,
                                       const char *name, int32_t ts_type,
                                       const char *resolution, const char *interval,  /* ISO-8601 filters */
                                       const char *features_json,
                                       int64_t *out_initial_ts_unix_ms,
                                       char **out_resolution, char **out_horizon, char **out_interval,  /* ISO-8601; infrastore_string_free */
                                       uint64_t *out_count, uint64_t *out_length,
                                       uint8_t *out_data_hash,
                                       char *ext_buf, uint64_t ext_cap,
                                       uint64_t *out_ext_len,
                                       char *out_units, uint64_t units_cap,
                                       uint64_t *out_units_len,
                                       uint64_t *out_element_shape, uint64_t element_shape_cap,
                                       uint64_t *out_element_shape_len,
                                       char *out_features_json, uint64_t features_json_cap,
                                       uint64_t *out_features_json_len);

int32_t infrastore_store_get_probabilistic_metadata(const struct InfraStore *handle,
                                            int64_t owner_id, int32_t owner_category,
                                            const char *name,
                                            const char *resolution, const char *interval,  /* ISO-8601 filters */
                                            const char *features_json,
                                            int64_t *out_initial_ts_unix_ms,
                                            char **out_resolution, char **out_horizon,
                                            char **out_interval,  /* ISO-8601; infrastore_string_free */
                                            uint64_t *out_count,
                                            uint64_t *out_length, uint8_t *out_data_hash,
                                            double **out_percentiles, uint64_t *out_percentiles_len,
                                            char *out_units, uint64_t units_cap,
                                            uint64_t *out_units_len,
                                            uint64_t *out_element_shape, uint64_t element_shape_cap,
                                            uint64_t *out_element_shape_len,
                                            char *out_features_json, uint64_t features_json_cap,
                                            uint64_t *out_features_json_len);

int32_t infrastore_store_has_typed(const struct InfraStore *handle,
                           int64_t owner_id, int32_t owner_category, const char *name,
                           int32_t ts_type,
                           const char *resolution, const char *interval,  /* ISO-8601; NULL = unset */
                           const char *features_json,
                           bool *out_present);
int32_t infrastore_store_remove_typed(struct InfraStore *handle,
                              int64_t owner_id, int32_t owner_category, const char *name,
                              int32_t ts_type,
                              const char *resolution, const char *interval,  /* ISO-8601; NULL = unset */
                              const char *features_json);

infrastore_store_copy_time_series copies one association onto another owner (optionally under a new name). Arrays are content-addressed, so only a new association row is written — no array data is duplicated, and the stored type is preserved (a DeterministicSingleTimeSeries stays one rather than being materialized into a dense Deterministic). The copy keeps the source's owner category. The leading owner_id / owner_category / name / ts_type / resolution / interval / features_json arguments identify the source series, exactly as for infrastore_store_remove_typed; a NULL new_name keeps the source name.

int32_t infrastore_store_copy_time_series(struct InfraStore *handle,
                                  /* source series: */
                                  int64_t owner_id, int32_t owner_category, const char *name,
                                  int32_t ts_type,
                                  const char *resolution, const char *interval,  /* ISO-8601; NULL = unset */
                                  const char *features_json,
                                  /* destination: */
                                  int64_t dst_owner_id, const char *dst_owner_type,
                                  const char *new_name);  /* NULL = keep the source name */

Readers

The per-timestamp read path is exposed as two opaque reader handles — InfraStoreStaticReaderHandle for SingleTimeSeries and InfraStoreForecastReaderHandle for forecasts. A reader is built once over a filter (the same has_owner / owner_id / has_owner_category / owner_category / name / resolution / features_json convention as the attribute-based access, with a forecast reader also taking a ts_type), then driven per timestamp. The lifecycle is: build → read the layout once → *_read in a loop → fetch values per group/entry → free. Each reader pins one resolution and owns reusable buffers that each read overwrites in place.

Ownership rules: the *_grid / *_timeline resolution/interval out-strings (char **) are owned — free each with infrastore_string_free. Keys from *_group_key / *_entry_key are owned InfraStoreKey * — free with infrastore_key_free. The *_values buffers (const uint8_t **) are borrowed: they point into reader memory, stay valid only until the next read or *_free, and must not be freed. Group/entry shapes follow the probe-then-fetch convention — call *_info with shape_buf = NULL / shape_cap = 0 to learn *out_shape_len, then again with a buffer of that length. *_read errors (never clamps) if at_unix_ms is off the reader's grid/timeline.

StaticReader

Reads every matching SingleTimeSeries at one timestamp, partitioned into (dtype, element_shape) groups; each group's values are one dense [num_columns, *element_shape] little-endian buffer whose column j is the key from infrastore_static_reader_group_key(reader, group_idx, j, …).

int32_t infrastore_store_build_static_reader(const struct InfraStore *handle,
                                     bool has_owner, int64_t owner_id,
                                     bool has_owner_category, int32_t owner_category,
                                     const char *name, const char *resolution,
                                     const char *features_json,
                                     struct InfraStoreStaticReaderHandle **out_reader);

int32_t infrastore_static_reader_grid(const struct InfraStoreStaticReaderHandle *reader,
                              int64_t *out_initial_ms, char **out_resolution,  /* free with infrastore_string_free */
                              uint64_t *out_length);
int32_t infrastore_static_reader_num_groups(const struct InfraStoreStaticReaderHandle *reader, uint64_t *out_n);
int32_t infrastore_static_reader_group_info(const struct InfraStoreStaticReaderHandle *reader, uint64_t group_idx,
                                    int32_t *out_dtype, uint64_t *out_num_columns,
                                    int64_t *shape_buf, uint64_t shape_cap, uint64_t *out_shape_len);
int32_t infrastore_static_reader_group_key(const struct InfraStoreStaticReaderHandle *reader,
                                   uint64_t group_idx, uint64_t col_idx,
                                   struct InfraStoreKey **out_key);  /* free with infrastore_key_free */
int32_t infrastore_static_reader_read(struct InfraStoreStaticReaderHandle *reader,
                              const struct InfraStore *store, int64_t at_unix_ms);
int32_t infrastore_static_reader_group_values(const struct InfraStoreStaticReaderHandle *reader, uint64_t group_idx,
                                      const uint8_t **out_ptr,  /* borrowed; valid until next read/free */
                                      uint64_t *out_byte_len);
void infrastore_static_reader_free(struct InfraStoreStaticReaderHandle *reader);

All matched series must share one grid (initial_timestamp + length); the build validates this and errors on divergence, so every column has a value at every valid timestamp (no presence mask).

ForecastReader

Reads the forecast window at one timestamp for every matching forecast of one type. The build ts_type names the forecast type; a Deterministic reader (2) is abstract and also includes DeterministicSingleTimeSeries (3), read into identical [H, *E] windows. (Note the asymmetry with infrastore_store_get_forecast, where 2 matches only a stored Deterministic and the family request must be spelled INFRASTORE_TYPE_ABSTRACT_DETERMINISTIC; the reader build takes no such sentinel.) All matched forecasts must share one window timeline (initial_timestamp + interval + count). Each entry's window is a little-endian buffer of its *_entry_info shape.

int32_t infrastore_store_build_forecast_reader(const struct InfraStore *handle,
                                       bool has_owner, int64_t owner_id,
                                       bool has_owner_category, int32_t owner_category,
                                       int32_t time_series_type,
                                       const char *name, const char *resolution,
                                       const char *features_json,
                                       struct InfraStoreForecastReaderHandle **out_reader);

int32_t infrastore_forecast_reader_timeline(const struct InfraStoreForecastReaderHandle *reader,
                                    int64_t *out_initial_ms,
                                    char **out_resolution, char **out_interval,  /* free each with infrastore_string_free */
                                    uint64_t *out_count);
int32_t infrastore_forecast_reader_num_entries(const struct InfraStoreForecastReaderHandle *reader, uint64_t *out_n);
int32_t infrastore_forecast_reader_num_slots(const struct InfraStoreForecastReaderHandle *reader, uint64_t *out_n);
int32_t infrastore_forecast_reader_entry_slot(const struct InfraStoreForecastReaderHandle *reader,
                                      uint64_t entry_idx, uint64_t *out_slot);
int32_t infrastore_forecast_reader_entry_info(const struct InfraStoreForecastReaderHandle *reader, uint64_t entry_idx,
                                      int32_t *out_dtype,
                                      int64_t *shape_buf, uint64_t shape_cap, uint64_t *out_shape_len);
int32_t infrastore_forecast_reader_entry_key(const struct InfraStoreForecastReaderHandle *reader,
                                     uint64_t entry_idx, struct InfraStoreKey **out_key);  /* free with infrastore_key_free */
int32_t infrastore_forecast_reader_read(struct InfraStoreForecastReaderHandle *reader,
                                const struct InfraStore *store, int64_t at_unix_ms);
int32_t infrastore_forecast_reader_entry_values(const struct InfraStoreForecastReaderHandle *reader, uint64_t entry_idx,
                                        const uint8_t **out_ptr,  /* borrowed; valid until next read/free */
                                        uint64_t *out_byte_len);
void infrastore_forecast_reader_free(struct InfraStoreForecastReaderHandle *reader);

Window-read deduplication. Forecasts that reference the same backing array and read plan (deduplicated identical data, or several DeterministicSingleTimeSeries over one SingleTimeSeries) collapse to a single window slot. infrastore_forecast_reader_read performs one backend read per slot, not per entry, so a forecast shared by N owners is read once per timestamp. infrastore_forecast_reader_num_slots is that physical read count, and infrastore_forecast_reader_entry_slot gives the 0-based slot backing each entry (entries that share data report the same slot) — group entries by slot to also decode each unique window only once.

Bulk Adds

A batch accumulates add requests client-side (no store I/O); infrastore_store_add_batch commits them all in one metadata transaction, which is much faster than per-item adds when ingesting many series. It is also the fast NetCDF write path: same-shaped SingleTimeSeries are packed into batch-sized datasets so the timestamp-major chunks are filled whole rather than a column at a time. The infrastore_batch_add_* functions take the same arguments as their infrastore_store_add_* counterparts minus the store handle and out_key; data buffers are copied into the batch, so they only need to stay valid for the call. The submit is all-or-nothing and drains the batch in either case (on error nothing was committed and the batch is left empty). On success the caller owns the key-handle array: free each key with infrastore_key_free, then the buffer with infrastore_keys_buffer_free (same contract as infrastore_store_get_time_series_keys). The batch handle itself is reusable after submit and must eventually be released with infrastore_batch_free.

struct InfraStoreBatch *infrastore_batch_new(void);
void            infrastore_batch_free(struct InfraStoreBatch *batch);

int32_t infrastore_batch_add_single(struct InfraStoreBatch *batch, /* infrastore_store_add_single args sans handle/out_key */ ...);
int32_t infrastore_batch_add_non_sequential(struct InfraStoreBatch *batch, ...);
int32_t infrastore_batch_add_forecast(struct InfraStoreBatch *batch, ...);       /* 2=Deterministic, 5=Scenarios */
int32_t infrastore_batch_add_probabilistic(struct InfraStoreBatch *batch, ...);

int32_t infrastore_store_add_batch(struct InfraStore *handle, struct InfraStoreBatch *batch,
                           struct InfraStoreKey ***out_keys, uint64_t *out_len);

Bulk Reads

infrastore_store_bulk_read_single reads many full SingleTimeSeries in one call, reading each packed dataset's column span once instead of re-reading every chunk per series — the efficient way to load many whole series (e.g. for exploration or plotting), where a single full-series read otherwise touches every chunk under the timestamp-major layout. Every key must identify a SingleTimeSeries; otherwise the call fails with INFRASTORE_ERR_INVALID_PARAMETER. The results are held in a InfraStoreBulkReadHandle (input order preserved) and read out element-by-element with infrastore_bulk_result_get_single, whose out-parameters match infrastore_store_get_single — the caller owns the returned resolution string and the shape/data buffers and frees them with infrastore_string_free, infrastore_buffer_free_i64, and infrastore_buffer_free_u8. The handle is not consumed by a read (elements may be read more than once) and must be released with infrastore_bulk_result_free.

int32_t infrastore_store_bulk_read_single(const struct InfraStore *handle,
                                  const struct InfraStoreKey *const *keys, uint64_t n,
                                  struct InfraStoreBulkReadHandle **out_result);
int64_t infrastore_bulk_result_len(const struct InfraStoreBulkReadHandle *result);   /* -1 if null */
int32_t infrastore_bulk_result_get_single(const struct InfraStoreBulkReadHandle *result, uint64_t index,
                                  /* same out-params as infrastore_store_get_single */ ...);
void    infrastore_bulk_result_free(struct InfraStoreBulkReadHandle *result);

Store-Wide Operations

int32_t infrastore_store_counts(const struct InfraStore *handle, int64_t *out_components_with_time_series,
                        int64_t *out_static_time_series, int64_t *out_forecasts);
/* Association count per type as a JSON array of {time_series_type, count};
   probe-then-fetch. */
int32_t infrastore_store_counts_by_type(const struct InfraStore *handle,
                                char *buf, uint64_t cap, uint64_t *out_len);
/* Distinct stored arrays (content hashes); shared arrays count once. */
int32_t infrastore_store_num_distinct_arrays(const struct InfraStore *handle, int64_t *out_count);
/* Grouped static / forecast summaries as JSON arrays (one object per group with a
   `count` field); probe-then-fetch. */
int32_t infrastore_store_static_summary(const struct InfraStore *handle,
                                char *buf, uint64_t cap, uint64_t *out_len);
int32_t infrastore_store_forecast_summary(const struct InfraStore *handle,
                                  char *buf, uint64_t cap, uint64_t *out_len);
/* Distinct owners per category + distinct arrays per kind (static/forecast). */
int32_t infrastore_store_counts_detailed(const struct InfraStore *handle, int64_t *out_components,
                                 int64_t *out_supplemental_attributes,
                                 int64_t *out_static_time_series, int64_t *out_forecasts);
/* Distinct owner ids of owner_category as a JSON array; optional type/resolution
   filters (NULL resolution = none). Probe-then-fetch. */
int32_t infrastore_store_list_owner_ids(const struct InfraStore *handle, int32_t owner_category,
                                bool has_time_series_type, int32_t time_series_type,
                                const char *resolution, char *buf, uint64_t cap, uint64_t *out_len);
/* out_present = false when no matching forecast. The out_horizon/out_interval/
   out_resolution ISO-8601 strings are NULL when absent (free with infrastore_string_free);
   out_count and out_initial_ms are -1 when absent. filter_* NULL = no filter. */
int32_t infrastore_store_get_forecast_parameters(const struct InfraStore *handle,
                                         const char *filter_resolution, const char *filter_interval,
                                         bool *out_present, char **out_horizon,
                                         char **out_interval, int64_t *out_count,
                                         char **out_resolution, int64_t *out_initial_ms);
/* Per-resolution static grids as a JSON array of {"resolution","initial_timestamp_ms",
   "length"} objects, ordered by resolution (empty array when no SingleTimeSeries);
   error when the series at one resolution disagree. filter_resolution NULL = every
   resolution, else scope to that ISO-8601 grid. Probe-then-fetch (buf=NULL, cap=0 to size). */
int32_t infrastore_store_check_static_consistency(const struct InfraStore *handle,
                                          const char *filter_resolution, char *buf,
                                          uint64_t cap, uint64_t *out_len);
/* Distinct resolutions as a JSON array of ISO-8601 duration strings, ascending;
   optional type filter. Probe-then-fetch (buf=NULL, cap=0 to size). */
int32_t infrastore_store_get_resolutions(const struct InfraStore *handle,
                                 bool has_time_series_type, int32_t time_series_type,
                                 char *buf, uint64_t cap, uint64_t *out_len);
/* out_kind: 0 = none, 1 = DEFLATE (out_level 0-9 + out_shuffle). */
int32_t infrastore_store_get_compression(const struct InfraStore *handle, uint8_t *out_kind,
                                 uint8_t *out_level, bool *out_shuffle);
int32_t infrastore_store_verify(const struct InfraStore *handle, uint64_t *out_error_count);
int32_t infrastore_store_compact(struct InfraStore *handle);
int32_t infrastore_store_flush(struct InfraStore *handle);
/* Persist the store's data to `path` (NetCDF) and `<path>.sqlite` (metadata),
   materializing an in-memory store to disk. Existing target files are overwritten. */
int32_t infrastore_store_persist(struct InfraStore *handle, const char *path);
/* has_owner=false clears all; when true, the owner is the pair (owner_id, owner_category). */
int32_t infrastore_store_clear(struct InfraStore *handle, bool has_owner, int64_t owner_id,
                       int32_t owner_category); /* owner_category: 0=Component, 1=SupplementalAttribute */
/* Reassign every time series owned by old_owner_id (in owner_category) to
   new_owner_id; *out_updated (when non-NULL) receives the number of associations
   changed. */
int32_t infrastore_store_replace_owner(struct InfraStore *handle,
                               int64_t old_owner_id, int64_t new_owner_id,
                               int32_t owner_category, uint64_t *out_updated);
/* List keys as a JSON array (identity + per-type descriptive snapshot, no physical
   storage detail). has_owner / has_owner_category are independent filters; with
   neither set the whole store is listed. Probe-then-fetch: call with buf=NULL,
   cap=0 to learn the length via out_len, then again with len+1 bytes. */
int32_t infrastore_store_list_keys(const struct InfraStore *handle,
                           bool has_owner, int64_t owner_id,
                           bool has_owner_category, int32_t owner_category,
                           bool has_time_series_type, int32_t time_series_type,
                           const char *name, const char *resolution, const char *features_json,
                           char *buf, uint64_t cap, uint64_t *out_len);
/* Like infrastore_store_list_keys, but each row is annotated with the hex content hash of
   the array it resolves to (keys_to_json's shape plus a `data_hash` field); rows
   that share a stored array share their `data_hash`, so a caller can group time
   series by their underlying data in one query. Same filters and probe-then-fetch
   convention as infrastore_store_list_keys. */
int32_t infrastore_store_list_array_groups(const struct InfraStore *handle,
                                   bool has_owner, int64_t owner_id,
                                   bool has_owner_category, int32_t owner_category,
                                   bool has_time_series_type, int32_t time_series_type,
                                   const char *name, const char *resolution, const char *features_json,
                                   char *buf, uint64_t cap, uint64_t *out_len);

Associations

Two catalogs of relationships between entities the store does not otherwise model. Both are independent of time series: there are no foreign keys and no cascade (both endpoints live in the caller's object graph, so a cascade could never fire), so removing a time series never removes an association and vice versa; a caller that wants both makes both calls.

Each family's query predicate crosses the boundary as one JSON object string rather than a set of positional arguments, because half its fields are string lists. Every field is optional and the set ones are ANDed:

{ "component_id": 1, "component_types": ["Generator", "Load"],
  "attribute_id": 100, "attribute_types": ["GeographicInfo"] }

{ "parent_id": 1, "parent_types": ["Generator"],
  "child_id": 7, "child_types": ["Bus"] }

A NULL (or empty) filter_json is the empty filter and matches every row — which is what makes a bulk export/import round trip one call each way. The *_types lists hold concrete type names, rendered as SQL IN (…); expanding an abstract type into its subtypes stays in the calling language, and an empty list matches nothing. An unknown field or malformed JSON is INFRASTORE_ERR_INVALID_PARAMETER. Every remove function reports its count through out_removed and treats removing nothing as success, not an error.

The list functions follow the same probe-then-fetch convention as infrastore_store_list_keys: call with buf = NULL, cap = 0 to learn the length via out_len, then again with a len + 1-byte buffer.

Supplemental-attribute associations

Which supplemental attributes are attached to which components. Identity is the (component_id, attribute_id) pair; the type names are denormalized labels, so re-attaching the same pair under different type names is still a duplicate. One attribute may be attached to many components.

/* Attach supplemental attribute (attribute_id, attribute_type) to component
   (component_id, component_type). INFRASTORE_ERR_DUPLICATE_ASSOCIATION if that component
   already carries that attribute, whatever type names are supplied. */
int32_t infrastore_store_add_supplemental_attribute_association(struct InfraStore *handle,
                                                        int64_t component_id,
                                                        const char *component_type,
                                                        int64_t attribute_id,
                                                        const char *attribute_type);

/* Attach many in one all-or-nothing transaction, from a JSON array of objects with
   component_id, component_type, attribute_id, and attribute_type. The import half
   of the round trip whose export is infrastore_store_list_supplemental_attribute_associations
   with a NULL filter. *out_added (when non-NULL) receives the number inserted. */
int32_t infrastore_store_add_supplemental_attribute_associations(struct InfraStore *handle,
                                                         const char *associations_json,
                                                         uint64_t *out_added);

/* Whether any attachment matches filter_json (NULL = any). */
int32_t infrastore_store_has_supplemental_attribute_association(const struct InfraStore *handle,
                                                        const char *filter_json,
                                                        bool *out_found);

/* Matching attachments as a JSON array, in insertion order; each object carries
   component_id, component_type, attribute_id, attribute_type. Probe-then-fetch. */
int32_t infrastore_store_list_supplemental_attribute_associations(const struct InfraStore *handle,
                                                          const char *filter_json,
                                                          char *buf, uint64_t cap,
                                                          uint64_t *out_len);

/* Distinct attribute ids of the matching rows, ascending, as a JSON array — the
   attributes attached to a component when component_id is set. Probe-then-fetch. */
int32_t infrastore_store_list_supplemental_attribute_ids(const struct InfraStore *handle,
                                                 const char *filter_json,
                                                 char *buf, uint64_t cap,
                                                 uint64_t *out_len);

/* Distinct component ids of the matching rows, ascending, as a JSON array — the
   components carrying an attribute when attribute_id is set. Probe-then-fetch. */
int32_t infrastore_store_list_components_with_attributes(const struct InfraStore *handle,
                                                 const char *filter_json,
                                                 char *buf, uint64_t cap,
                                                 uint64_t *out_len);

/* Remove every matching attachment; *out_removed (when non-NULL) receives the
   count. Removing nothing is success, not an error. */
int32_t infrastore_store_remove_supplemental_attribute_associations(struct InfraStore *handle,
                                                            const char *filter_json,
                                                            uint64_t *out_removed);

/* Move every attachment from component old_id to new_id; *out_updated (when
   non-NULL) receives the rows changed. INFRASTORE_ERR_DUPLICATE_ASSOCIATION if new_id
   already carries one of the attributes being moved. */
int32_t infrastore_store_replace_supplemental_attribute_component_id(struct InfraStore *handle,
                                                             int64_t old_id, int64_t new_id,
                                                             uint64_t *out_updated);

/* Attachment counts through out_count. kind selects what is counted:
   0 = rows matching the filter, 1 = distinct attributes among them,
   2 = distinct components among them. */
int32_t infrastore_store_count_supplemental_attribute_associations(const struct InfraStore *handle,
                                                           const char *filter_json,
                                                           int32_t kind, int64_t *out_count);

/* Grouped counts as JSON arrays: by attribute type, as {"type": …, "count": …}
   ordered by type; and by both type names, as
   {"component_type": …, "attribute_type": …, "count": …} ordered by attribute type
   then component type. Probe-then-fetch. */
int32_t infrastore_store_supplemental_attribute_counts_by_type(const struct InfraStore *handle,
                                                       char *buf, uint64_t cap,
                                                       uint64_t *out_len);
int32_t infrastore_store_supplemental_attribute_summary(const struct InfraStore *handle,
                                                char *buf, uint64_t cap,
                                                uint64_t *out_len);

/* The attribute ids attached to component 1. */
const char *filter = "{\"component_id\":1}";
uint64_t len = 0;
infrastore_store_list_supplemental_attribute_ids(store, filter, NULL, 0, &len);
char *json = malloc(len + 1);
infrastore_store_list_supplemental_attribute_ids(store, filter, json, len + 1, &len); /* e.g. "[100]" */
free(json);

Parent/child associations

Directed edges between components — a generator (parent) connected to a bus (child), say. Both endpoints are always components. Identity is the ordered (parent_id, child_id) pair, so the reversed pair is a different edge, and with no relationship-kind column one ordered pair may be related at most once.

This family is deliberately narrower than the supplemental one — no counts-by-type and no grouped summary — because there is no consumer for them yet; both are additive if one appears.

/* Record a directed edge from component (parent_id, parent_type) to component
   (child_id, child_type). INFRASTORE_ERR_DUPLICATE_ASSOCIATION if that ordered pair is
   already related; the reversed pair is a different edge. */
int32_t infrastore_store_add_parent_child_association(struct InfraStore *handle,
                                              int64_t parent_id, const char *parent_type,
                                              int64_t child_id, const char *child_type);

/* Record many edges in one all-or-nothing transaction, from a JSON array of objects
   with parent_id, parent_type, child_id, and child_type. *out_added (when non-NULL)
   receives the number inserted. */
int32_t infrastore_store_add_parent_child_associations(struct InfraStore *handle,
                                               const char *associations_json,
                                               uint64_t *out_added);

/* Whether any edge matches filter_json (NULL = any). */
int32_t infrastore_store_has_parent_child_association(const struct InfraStore *handle,
                                              const char *filter_json,
                                              bool *out_found);

/* Matching edges as a JSON array, in insertion order; each object carries
   parent_id, parent_type, child_id, child_type. Probe-then-fetch. */
int32_t infrastore_store_list_parent_child_associations(const struct InfraStore *handle,
                                                const char *filter_json,
                                                char *buf, uint64_t cap,
                                                uint64_t *out_len);

/* Distinct ids on one end of the matching edges, ascending, as a JSON array.
   endpoint is 0 for parents and 1 for children — so endpoint = 1 with parent_id
   set is "the children of this component". Probe-then-fetch. */
int32_t infrastore_store_list_parent_child_ids(const struct InfraStore *handle,
                                       const char *filter_json, int32_t endpoint,
                                       char *buf, uint64_t cap, uint64_t *out_len);

/* Remove every matching edge; *out_removed (when non-NULL) receives the count.
   Removing nothing is success, not an error. */
int32_t infrastore_store_remove_parent_child_associations(struct InfraStore *handle,
                                                  const char *filter_json,
                                                  uint64_t *out_removed);

/* Rewrite component old_id to new_id on BOTH ends of every edge; *out_updated
   (when non-NULL) receives the rows changed. INFRASTORE_ERR_DUPLICATE_ASSOCIATION if the
   rewrite would duplicate an edge new_id already has. */
int32_t infrastore_store_replace_parent_child_component_id(struct InfraStore *handle,
                                                   int64_t old_id, int64_t new_id,
                                                   uint64_t *out_updated);

/* Number of edges matching filter_json, through out_count. */
int32_t infrastore_store_count_parent_child_associations(const struct InfraStore *handle,
                                                 const char *filter_json,
                                                 int64_t *out_count);

/* The children of component 1. */
const char *filter = "{\"parent_id\":1}";
uint64_t len = 0;
infrastore_store_list_parent_child_ids(store, filter, 1 /* children */, NULL, 0, &len);
char *json = malloc(len + 1);
infrastore_store_list_parent_child_ids(store, filter, 1, json, len + 1, &len);   /* e.g. "[7]" */
free(json);

Neither association catalog is exposed over the gRPC server or the infrastore CLI.

Error Messages

int32_t infrastore_last_error_message(char *buf, uint64_t buf_len, uint64_t *needed);

Copies the thread-local error message (UTF-8, null-terminated) into buf. *needed receives the length excluding the NUL. If buf_len is too small, the message is truncated but INFRASTORE_OK is still returned — call once with buf = NULL, buf_len = 0 to learn the needed size, then again with a buffer of *needed + 1. This is the pattern TimeSeries.jl uses.

Building

cargo build -p infrastore-ffi --release
# Library at: target/release/libinfrastore_ffi.{dylib,so,dll}

Point consumers at it via the INFRASTORE_LIB environment variable (see the Julia how-to).

Tracing

int32_t infrastore_store_init_logging(const char *filter);

Initialize the Rust tracing subscriber. filter is a null-terminated UTF-8 EnvFilter directive string (e.g. "debug" or "infrastore_core=debug"). Pass NULL to read the RUST_LOG environment variable; if that variable is also unset, no output is produced.

The subscriber is initialized at most once per process — subsequent calls are no-ops. Returns INFRASTORE_OK on success or INFRASTORE_ERR_INVALID_UTF8 if filter is not valid UTF-8.

The Julia binding calls this automatically from its __init__ hook when RUST_LOG is set, and exposes init_logging for explicit control.

gRPC API

The proto contract lives at proto/infrastore/v1/store.proto and is compiled into infrastore-proto with tonic. The service is read-only — every write operation (add, remove, clear, compact) requires local filesystem access and is intentionally absent.

The association catalogs are absent too, reads included: no message or RPC covers supplemental_attribute_associations or parent_child_associations. Consumers of those tables work against a local Store.

• Package: infrastore.v1
• Service: CatalogStore

Methods

TimeSeriesKey messages returned by GetTimeSeriesKeys, ListKeys, and ResolveForecastKey now carry the per-variant descriptive snapshot (initial_timestamp_rfc3339, length, horizon, count) in addition to the identity tuple, so RemoteClient reconstructs the full core TimeSeriesKey enum.

Common Messages

enum TimeSeriesType {
  SINGLE_TIME_SERIES               = 0;
  NON_SEQUENTIAL_TIME_SERIES       = 1;
  DETERMINISTIC                    = 2;
  DETERMINISTIC_SINGLE_TIME_SERIES = 3;
  PROBABILISTIC                    = 4;
  SCENARIOS                        = 5;
}

enum OwnerCategory { COMPONENT = 0; SUPPLEMENTAL_ATTRIBUTE = 1; }

message FeatureValue {
  oneof value {
    int64  int_value   = 1;
    double float_value = 2;
    bool   bool_value  = 3;
    string str_value   = 4;
  }
}

message Features { map<string, FeatureValue> entries = 1; }

message TimeSeriesKey {
  int64          owner_id         = 1;
  TimeSeriesType time_series_type = 2;
  string         name             = 3;
  string         resolution       = 4;   // ISO-8601 duration; empty = unset
  Features       features         = 5;
  OwnerCategory  owner_category   = 6;   // part of the owner identity / key
  string         interval         = 7;   // ISO-8601 duration; empty = unset
  // Descriptive snapshot (NOT part of the identity); variant implied by type.
  optional string initial_timestamp_rfc3339 = 8;
  optional uint64 length                     = 9;
  optional string horizon                    = 10;  // ISO-8601 duration
  optional uint64 count                      = 11;
}

message TimeSeriesMetadata {
  int64           owner_id                  = 1;
  string          owner_type                = 2;
  OwnerCategory   owner_category            = 3;
  TimeSeriesType  time_series_type          = 4;
  string          name                      = 5;
  bytes           data_hash                 = 6;   // 32 bytes
  // Temporal fields are `optional` so genuine values (e.g. length == 0) decode
  // correctly rather than colliding with a zero/empty sentinel.
  optional string initial_timestamp_rfc3339 = 7;
  optional string resolution                = 8;   // ISO-8601 duration
  optional uint64 length                    = 9;
  optional string horizon                   = 10;  // ISO-8601 duration
  optional string interval                  = 11;  // ISO-8601 duration
  optional uint64 count                     = 12;
  repeated string timestamps_rfc3339        = 13;
  Features        features                  = 14;
  optional string units                     = 16;
  int32           dtype                     = 17;  // Dtype code
  repeated uint64 element_shape             = 18;  // per-step trailing dims
  optional string ext                       = 19;  // opaque package-owned payload
  repeated double percentiles               = 20;  // Probabilistic only
}

Request / Response Messages

message ListReq {
  optional int64          owner_id         = 1;
  optional string         owner_type       = 2;
  optional TimeSeriesType time_series_type = 3;
  optional string         name             = 4;
  optional string         resolution       = 5;   // ISO-8601 duration
  Features                features         = 6;   // subset match
  optional OwnerCategory  owner_category   = 7;
  optional string         interval         = 8;   // ISO-8601 duration
}
message ListResp { repeated TimeSeriesMetadata metadata = 1; }

message GetReq {
  TimeSeriesKey   key           = 1;
  optional string start_rfc3339 = 2;   // optional time-axis slice; all-or-nothing with end
  optional string end_rfc3339   = 3;
}
message GetResp {
  string          initial_timestamp_rfc3339 = 1;
  string          resolution                = 2;   // ISO-8601 duration
  uint64          length                    = 3;
  repeated uint64 shape                     = 4;   // array dimensions (multi-dim supported)
  reserved 5;                                      // was: repeated double values
  TimeSeriesType  time_series_type          = 6;
  repeated string timestamps_rfc3339        = 7;   // set for NonSequentialTimeSeries
  int32           dtype                     = 8;   // Dtype code
  bytes           value_bytes               = 9;   // raw little-endian, row-major
  string          ext              = 10;
  // Forecast-specific fields (populated for Deterministic / Probabilistic / Scenarios).
  string          horizon                   = 11;  // ISO-8601 duration
  string          interval                  = 12;  // ISO-8601 duration
  uint64          count                     = 13;
  repeated double percentiles               = 14;  // Probabilistic only
  uint64          scenario_count            = 15;  // Scenarios only
}

message KeysReq  { int64 owner_id = 1; OwnerCategory owner_category = 2; }
message KeysResp { repeated TimeSeriesKey keys = 1; }

message ResolutionsReq  { optional TimeSeriesType time_series_type = 1; }
message ResolutionsResp { repeated string resolution = 1; }   // ISO-8601 durations

message CountsReq  {}
message CountsResp {
  int64 components_with_time_series = 1;
  int64 static_time_series          = 2;
  int64 forecasts                   = 3;
}

message ForecastParamsReq  {
  optional string resolution = 1;   // ISO-8601 duration filter
  optional string interval   = 2;   // ISO-8601 duration filter
}
message ForecastParamsResp {
  optional string horizon                   = 1;   // ISO-8601 duration
  optional string interval                  = 2;   // ISO-8601 duration
  optional uint64 count                     = 3;
  optional string resolution                = 4;   // ISO-8601 duration
  optional string initial_timestamp_rfc3339 = 5;
}

message HasReq  { TimeSeriesKey key = 1; }
message HasResp { bool present = 1; }

message VerifyReq  {}
message VerifyResp { repeated string errors = 1; }

GetReq Time Slice

start_rfc3339 and end_rfc3339 are all-or-nothing: supply both to request a time-axis slice, or neither to fetch the whole series. Setting exactly one is rejected with InvalidArgument ("start_rfc3339 and end_rfc3339 must be supplied together"). Each value must parse as RFC 3339; a malformed timestamp is also InvalidArgument.

Forecasts Over gRPC

The service is read-only, but its read surface covers dense forecasts. Forecast associations created through the Rust core or C ABI appear in ListTimeSeries — TimeSeriesMetadata carries horizon, interval (ISO-8601 durations), count, and (for Probabilistic) percentiles — and GetCounts includes them in forecasts.

GetTimeSeries returns forecast values too. For a Deterministic, DeterministicSingleTimeSeries (synthesized into Deterministic), Probabilistic, or Scenarios key it fills the GetResp array fields (value_bytes + dtype), the window parameters (horizon, interval, count), and the percentiles (Probabilistic) or scenario_count (Scenarios); the client reconstructs the matching type. Arrays are dtype-generic on the wire — value_bytes is the raw little-endian buffer and dtype names the element type (f64/f32/i64/i32/u64/bool), so non-f64 arrays survive the round trip without coercion. One caveat:

• ext is not carried in GetResp. The opaque package-owned payload is returned by ListTimeSeries (on TimeSeriesMetadata) but left empty by GetTimeSeries, so a value fetched directly by key comes back without it.

Authentication

When the server is configured with method = "api_key", clients must send the key in the x-api-key request metadata (header). The server checks the supplied key against every configured key of the same length without early-exit, so a match is not leaked by timing; the comparison is not blinded against the supplied key's length, which is treated as non-secret. A missing or wrong key is rejected before the RPC runs. With method = "none" no metadata is required. See Server Configuration.

Rust Client

infrastore-server ships an async RemoteClient that mirrors the read methods and returns core types, mapping gRPC Status codes back onto the TimeSeriesError taxonomy:

ConnectionError is only the fallback arm, so a remote NotFound or a rejected argument surfaces with the same variant a local Store would return.

#![allow(unused)]
fn main() {
use infrastore_core::OwnerCategory;
use infrastore_server::client::RemoteClient;

let client = RemoteClient::connect("http://127.0.0.1:50051".into()).await?;
let counts = client.get_counts().await?;
let keys = client.get_time_series_keys(42, OwnerCategory::Component).await?;
let data = client.get_time_series(&keys[0], None).await?;
}

RemoteClient methods: connect, from_channel, list_time_series, list_keys, get_metadata, get_time_series, bulk_read, get_time_series_keys, get_resolutions, get_intervals, get_counts, counts_by_type, time_series_counts_detailed, get_forecast_parameters, static_summary, forecast_summary, list_owner_ids, has_time_series, check_static_consistency, resolve_forecast_key, verify_integrity — the full read surface described above. See the gRPC Server guide for end-to-end usage and adding an API key to client requests.

Server Configuration

The gRPC server is configured by a single TOML file passed with --config. The starting point is examples/server.toml.

infrastore-server --config my_server.toml

File Structure

[server]
host = "0.0.0.0"
port = 50051

[data]
files = ["./store.nc"]

[authentication]
# "none" or "api_key". For api_key, populate `keys`; clients must send the
# value in the `x-api-key` request header.
method = "none"
# keys = ["replace-me-with-a-secret-1", "replace-me-with-a-secret-2"]

Sections

[server]

[data]

v0 serves a single file (the first entry). Multiple entries are reserved for a later milestone. The matching <path>.sqlite catalog must sit beside each NetCDF file. The server opens the store read-only.

[authentication]

The whole section is optional; omitting it defaults to method = "none".

Validation runs at startup, so a bad [authentication] section fails loudly rather than at the first request:

• method = "api_key" with an empty keys list is rejected.
• An unknown method value is rejected.

That validation covers the values, not the key names. ServerConfig does not reject unknown fields, so a misspelled TOML key is silently ignored and the default (or a missing-field parse error, for a required key) applies instead. A server that starts with [authenticaton] (sic) is running with method = "none" — check the startup logs and confirm the effective settings rather than assuming a typo would have been caught.

When method = "api_key", each request must carry a matching value in the x-api-key metadata header. Keys are compared without early-exit — every configured key of the same length as the supplied one is checked, so timing does not reveal which key matched or how far a wrong key got. The supplied key's length is not blinded: keys of a different length are rejected before the byte-wise compare, treating length as non-secret.

Startup Behavior

On launch the server:

1. Loads and parses the TOML file.
2. Validates the [authentication] section.
3. Opens the first [data].files entry as a read-only store (errors if the list is empty).
4. Binds host:port and serves the CatalogStore gRPC service.

Logging honors the RUST_LOG environment variable (default info):

RUST_LOG=debug infrastore-server --config my_server.toml

See the gRPC Server guide for the end-to-end workflow and the gRPC API reference for the served methods.

CLI Reference

infrastore-cli builds the infrastore binary, which reads and writes a store directly on disk (NetCDF + SQLite). For a task-oriented walkthrough, see Use the infrastore CLI.

The CLI covers time series only. The association catalogs in the same catalog file have no infrastore commands; reach them through the Rust, Python, or Julia API.

Synopsis

infrastore [--store <PATH.nc>] [-f <FORMAT>] [--log-level <FILTER>] <COMMAND>

Global options

--store (or INFRASTORE_STORE) is required by every command except template and completions.

-f/--format affects the read/inspection commands (list, get, info, export, stats, summary, verify, check-consistency, resolutions, params, compact). It is accepted anywhere because it is global, but the write commands (add, remove, rename, copy, replace-owner, clear, transform, persist) ignore it and print plain text; template always prints a JSON descriptor. export requires -f csv or -f json (there is no table export).

Commands

infrastore --store <PATH> add --descriptor <FILE.json> [--csv <FILE.csv>] [--compression <none|deflate[:LEVEL]>] [--no-shuffle]
infrastore --store <PATH> list    [SELECTOR...]
infrastore --store <PATH> get     [SELECTOR...] [--time-range START..END] [--limit N | --full]
infrastore --store <PATH> info    [SELECTOR...]
infrastore --store <PATH> -f csv|json export [SELECTOR...] [--dir <DIR>]
infrastore --store <PATH> remove  [SELECTOR...] [--all] [--force] [--dry-run]
infrastore --store <PATH> rename  [SELECTOR...] --new-name <NAME> [--dry-run]
infrastore --store <PATH> copy    [SELECTOR...] --dst-owner-id <I> --dst-owner-type <T> [--new-name <NAME>] [--dry-run]
infrastore --store <PATH> replace-owner --old <I> --new <I> --owner-category <C> [--dry-run]
infrastore --store <PATH> clear   [--owner-id <I> --owner-category <C>] [--force] [--dry-run]
infrastore --store <PATH> transform --horizon <DUR> --interval <DUR> [--owner-category <C>] [--resolution <DUR>]
infrastore --store <PATH> persist --dest <PATH.nc>
infrastore --store <PATH> compact [--force]
infrastore --store <PATH> stats
infrastore completions <SHELL>
infrastore --store <PATH> summary [--static-only | --forecast-only]
infrastore --store <PATH> verify
infrastore --store <PATH> check-consistency [--resolution <DUR>]
infrastore --store <PATH> resolutions
infrastore --store <PATH> params [--resolution <DUR>] [--interval <DUR>]
infrastore template <single|non_sequential|deterministic|probabilistic|scenarios>

--csv overrides the csv path inside the descriptor, and only works for a descriptor that describes a single series. Passing it alongside a descriptor array holding more than one object fails with --csv cannot be used with an array descriptor.

transform takes no selector: it rewrites every SingleTimeSeries in the store, deriving a DeterministicSingleTimeSeries from each. --owner-category and --resolution optionally scope it to one category and/or resolution. --horizon must not exceed the shortest matched series (horizon / resolution steps must fit within its length), or the command fails.

remove --all uses the selector as a filter that may match several series, removing them all in one transaction; without --all, the selector must resolve to exactly one series. stats, summary, verify, check-consistency, resolutions, and params are read-only inspection commands and honor -f/--format; verify exits nonzero when the integrity report lists any errors.

export is the read-direction inverse of the batch add: the selector may match many series, and each is written to <owner_id>_<owner_type>_<name>_<type>.csv|json inside --dir. Without --dir the selector must match exactly one series, which goes to stdout. CSV output carries real timestamps (see the CSV Layout section); JSON output is one structured object per series.

--dry-run on remove, clear, replace-owner, rename, and copy prints what would change and exits without opening the store for writing. add --compression sets the NetCDF compression policy for a store this command creates (none, deflate, or deflate:LEVEL with --no-shuffle to disable byte-shuffle); passing it for an existing store is an error, since the persisted policy governs.

Selectors

get, info, and remove identify exactly one series with these flags; list accepts the same flags as filters. Every flag is optional. Only --feature may be repeated; the rest take a single value:

If a selector matches more than one series, infrastore errors and lists the candidates so the query can be narrowed. The owner identity is the pair (owner_id, owner_category), so a component and a supplemental attribute may share a numeric owner_id; add --owner-category to disambiguate when both exist.

Type Spellings

--type (and the descriptor's type key) accepts six values. Matching is case-insensitive and ignores underscores, so each type has a short form and a full form:

deterministic_single is not writable from a descriptor (use transform), but it is selectable, and it is often required: transform derives a series that shares (owner_id, name, resolution) with its source SingleTimeSeries, so after a transform a query like infrastore get --owner-id 42 --name load matches two series and errors. --type single or --type deterministic_single is the only way to pick one.

Note that inputs and outputs use different spellings. You pass the short, lowercase forms (--type single, --owner-category component), but list, get, and info render the canonical CamelCase names (SingleTimeSeries, Component). Piping -f json list output straight back into a filter therefore needs no translation — the CamelCase form is accepted as input too — but string-comparing rendered output against the short form will not match.

Durations and Timestamps

• Durations (resolution, horizon, interval): an integer plus a unit — ms, s, min, h, d (e.g. 500ms, 15min, 24h, 7d). A bare integer is milliseconds. These three also accept ISO-8601 duration strings (e.g. PT1H, P1M, P1Y); a calendar grid (monthly/quarterly/annual) can only be expressed this way, since the human-unit form is always a fixed span.
• Timestamps (initial_timestamp, non-sequential timestamp column): RFC3339 (e.g. 2024-01-01T00:00:00Z) or a bare integer of epoch milliseconds.
• --time-range is a pair of timestamps, not a duration: START..END (half-open), where each side is parsed as a timestamp. For example --time-range 2024-01-01T01:00:00Z..2024-01-01T03:00:00Z. A duration such as --time-range 1h is rejected with invalid --time-range '1h' (expected START..END).

Descriptor Schema

A descriptor JSON file is either a single object (one series) or an array of objects (batch add). The CSV holds only numbers (plus a leading timestamp column for non_sequential).

Unknown keys are rejected. Any key not in the table above — including a typo like resolutionn — is a hard parse error listing the accepted fields, so hand-edited templates fail loudly rather than silently dropping a setting.

CSV Layout

infrastore computes the full array shape from the descriptor and reads the CSV's value cells in row-major order to fill it. The total cell count must equal the product of the shape.

bool cells accept true/false/1/0. For single and non_sequential series, get -f csv re-emits the same layout add consumes, so values round-trip. That output always starts with a header row (value, or timestamp,value... for non_sequential), which is what the has_header: true default — and every template — expects.

For forecasts, get -f csv and export -f csv emit timestamped rows instead of the flat ingest layout: one row per (window, step) with issue_time and target_time columns, and one value column per percentile (value[p10]), scenario (value[s0]), or element entry. This output is for analysis, not re-ingestion — re-adding a forecast needs the flat row-major CSV described above.

Exit Status

Contributing

This page covers the conventions for contributing code and documentation to infrastore.

Code Quality

All changes should pass the standard workspace checks before being committed:

cargo fmt --all -- --check                              # Rust formatting
cargo clippy --workspace --all-targets --all-features -- -D warnings   # Rust linting
cargo test --workspace --all-features                   # Rust tests
dprint check                                            # Markdown formatting
cargo deny check --config deny.toml                     # Dependency policy

The workspace targets edition 2024 and declares an MSRV of Rust 1.94 (rust-version in the root Cargo.toml is the authority; there is no rust-toolchain file). Markdown in docs/ is wrapped at 100 characters — dprint fmt does it for you.

Testing Across Bindings

A change to the core may need re-running the binding tests:

# Rust
cargo test --workspace

# Python
cd crates/infrastore-py && maturin develop && pytest ../../python/tests

# Julia (requires the cdylib + INFRASTORE_LIB)
cargo build -p infrastore-ffi --release
export INFRASTORE_LIB=$PWD/target/release/libinfrastore_ffi.dylib  # .so on Linux
julia --project=julia/InfraStore.jl julia/InfraStore.jl/test/runtests.jl

The On-Disk Format Is a Contract

The NetCDF layout, the SQLite schema, and the hashing rules together form the on-disk format, versioned by DATA_FORMAT_VERSION. Any backward-incompatible change to any of them must bump that version. The hash_golden test pins representative hashes; if it fails because you changed the hashing domain, that is a format-breaking change, not a test to "fix."

Generated Surfaces

• The C header crates/infrastore-ffi/include/infrastore.h is generated by cbindgen — do not hand-edit it; it is refreshed when you build infrastore-ffi.
• The proto-derived Rust types in infrastore-proto are generated by tonic from proto/infrastore/v1/store.proto.

When you change the public surface of a binding, update the matching reference page in this book.

Documentation

Docs follow the Diataxis framework. Put new pages in the right category:

After adding a page, add it to src/SUMMARY.md. Preview locally:

cd docs
mdbook serve --open

Significant design decisions belong in the Explanation section, covering the problem, the approach, the trade-offs, and the alternatives considered.