On-Disk File Format

A persisted store is a pair of files that must be kept together:

<name>.h5          # HDF5   — numerical arrays
<name>.h5.sqlite   # SQLite — metadata associations

The SQLite catalog path is the HDF5 path with .sqlite appended to the file name. This page is the authoritative description of both. For the rationale behind the split, see the Storage Model.

The array file is a plain HDF5 file, written directly against libhdf5. Store::open accepts only files it wrote itself (see storage_backend below). The .h5 extension is a convention — the store never inspects it.

Format Version

The HDF5 root carries four global attributes:

data_format_version = "0.19.0"
compression         = "deflate:3:shuffle"
storage_backend     = "hdf5"
catalog_generation  = "9f2c1ab4e70d5836c41b9e2af0d7c358"

data_format_version is the semver of the on-disk format (DATA_FORMAT_VERSION). It is bumped when the HDF5 layout, the SQLite schema, or the hashing domain changes in a backward-incompatible way. A purely additive SQLite table does not qualify: it costs old readers nothing and old stores pick it up from the idempotent DDL on their first writable open. The associations table landed that way.

compression records the filter policy the store was created with, so that appends made after reopening reuse the same filter. It is not part of the compatibility contract — see Compression below.

storage_backend marks the file as one infrastore wrote. Store::open checks it before reading anything else and rejects a file that lacks it with InvalidParameter — this is what distinguishes an infrastore store from an arbitrary HDF5 file, so a foreign file is refused rather than misread.

catalog_generation pairs this file with exactly one catalog; the same value lives in the catalog's catalog_identity table. Store::open compares the two and rejects a mismatch with MismatchedArtifact. It is written at creation and re-minted by every persist_to, which is what makes a save interrupted between its two renames detectable — see Saving. It is additive, so it carries no data_format_version change: a store written before it existed has neither the attribute nor the table, which reads as "unstamped" and skips the check rather than failing. Compaction preserves the existing value rather than minting a new one, since it rewrites only the HDF5 half.

Opening a store sorts its recorded version into one of three tiers, not two:

• Current — the stamp equals this build's DATA_FORMAT_VERSION. Opened as-is.

• Upgradable — the stamp is at least MIN_UPGRADABLE_VERSION and older. The array layout is compatible, so the file is read as it stands; a writable open runs the catalog migration ladder and then re-stamps the file. A read-only open changes nothing and leaves the stamp alone.

  Whether a read-only open succeeds is decided by the catalog half, not by this tier. CatalogMigrationRequired comes from a stale CATALOG_SCHEMA_REVISION; an upgradable stamp sitting over an already-current catalog opens fine. That is not a hypothetical combination — it is what an interrupted writable open leaves behind, and the step ordering is chosen to make it the harmless outcome.

• Incompatible — older than MIN_UPGRADABLE_VERSION, newer than this build, or unparseable (including an unstamped file). Fails with IncompatibleFormat, naming both versions. There is no upgrade path: regenerate the store with the matching build.

The catalog carries its own revision in schema_version (CATALOG_SCHEMA_REVISION), independent of the artifact version above. Any catalog change the idempotent DDL cannot make to an existing table — a new column, a changed CHECK, a rebuilt table, a backfill — needs a CATALOG_SCHEMA_REVISION bump plus an append-only migration, not a re-created store. A catalog written by a newer build is CatalogTooNew and is refused in both directions. Catalog revision 2, which widens the time_series_type CHECK from BETWEEN 0 AND 5 to >= 0, is the first such change and takes no DATA_FORMAT_VERSION bump at all — nothing in the HDF5 file moves, so a 0.19.0 store upgrades in place on its first writable open. See Upgrade a store in place.

PersistentTimeSeries (storage code 6) is the first type to arrive through that door, and it likewise takes no version bump. The HDF5 layout is unchanged — a persistent series pools into the same nsts_… datasets as a NonSequentialTimeSeries on the same breakpoints, and its breakpoints are an ordinary timestamp vector — so the widened CHECK is the whole of what it needed from the catalog, and every 0.19.0 store already has it.

Version history

Newest first. Each entry is the change that forced the bump; 0.19.0 is both the current DATA_FORMAT_VERSION and the current MIN_UPGRADABLE_VERSION, so only the top entry describes stores this build can open.

0.19.0 — moved a NonSequentialTimeSeries's timestamps out of the SQLite catalog and into the HDF5 file: the timestamp_sets table is gone, and each distinct time axis is now an i64 dataset of unix milliseconds under time_series/timestamps/, named by the same content hash the association row already carried. The hash domain changed with it — timestamps_hash is now the SHA-256 of those milliseconds rather than of the delta-varint blob the table held — so the nsts_… pool names change too. The timestamps are data, and belong in the half of the artifact built for data: a store may hold many distinct axes, each of them long, and a JSON document round trip needs them to travel with the artifact rather than be locked inside the catalog.

0.18.0 — made the id column of all three catalog tables (time_series_associations, supplemental_attribute_associations, and parent_child_associations) an INTEGER PRIMARY KEY AUTOINCREMENT rather than a bare rowid alias, so an id is never reissued once its row is deleted. Recycled ids matter because the id is an external reference — a consumer stores it in its own model — and reuse makes a stale reference resolve to a different, valid row.

This is the one bump that adds no column and changes no value. It was taken because AUTOINCREMENT is part of a table's declaration and there is no ALTER TABLE for it, so re-running the DDL over an older catalog would have left all three tables on recycled ids while reporting success — and, before the migration ladder existed, stranding that catalog was the only way to keep it out. The version floor is what enforces the guarantee, and it holds: MIN_UPGRADABLE_VERSION is 0.19.0, so every catalog predating this entry is Incompatible and refused on open. No store this build can open lacks AUTOINCREMENT. The silent-recycling failure above is what the bump prevented; it is not a state a store can be found in, and nothing needs to check for it at runtime (every_association_table_declares_autoincrement in crates/infrastore-core/tests/association_ids.rs holds a fresh catalog to the property, and sqlite_sequence is what proves the keyword took effect rather than merely being present).

The ladder has since changed what a repeat would cost, so do not reason from this entry to the next one. A table rebuild is now an ordinary migration — revision 2 is exactly that, and carries the AUTOINCREMENT high-water mark across by hand — so the same change made today would be a CATALOG_SCHEMA_REVISION bump that upgrades existing catalogs in place, not a DATA_FORMAT_VERSION bump that strands them.

0.17.0 — added the metadata column time_reference, which records how a series' timestamps were spelled: an instant in UTC, an instant at a fixed offset, an instant in a named IANA zone, or a wall clock naming no instant. It also changes how stored timestamps are interpreted — a row marked zoneless holds wall clocks the store keeps as if UTC, which an older reader would hand back as instants. See Time References.

0.16.0 — added the metadata column component_field.

0.15.0 — renamed the metadata column ext to application_data and added the quantity_kind and unit_system columns. Unlike a new table, new columns are not picked up by the idempotent CREATE TABLE IF NOT EXISTS DDL, so a store one version behind is rejected on open.

0.14.0 — moved a NonSequentialTimeSeries's timestamps out of the association row: the timestamps_json TEXT column became a timestamps_hash BLOB resolving into a then-new content-addressed timestamp_sets catalog table (which 0.19.0 replaced with the HDF5 datasets described above), and irregular arrays sharing a time axis became column-packed into nsts_… datasets keyed by that hash instead of one standalone arr_… dataset each.

0.13.0 — replaced the dtype column with element_type, which names the logical element type and derives the physical dtype from it. See Element types.

0.12.0 — changed owner_category and time_series_type from TEXT names to small INTEGER codes. See Discriminant encoding below.

0.11.0 — renamed the metadata column logical_type to ext, an opaque, package-owned extension payload (typically JSON) the store stores verbatim and never interprets.

0.10.0 — replaced the per-association features table with the content-addressed feature_sets table below, so a feature map is stored once and shared by every association that uses it, dropping the association_id foreign key and its ON DELETE CASCADE.

0.9.0 — changed the packed-dataset chunking to timestamp-major (1, cols, *element_shape) and made the column count cols per-dataset (sized to the writing batch) instead of a fixed 1,000, optimizing reads across series by timestamp and bulk writes.

0.8.0 — added the forecast interval to the association uniqueness key, so two forecasts of one variable differing only by interval became distinct series. This widened both unique indexes (the NULL-folding index now COALESCEs interval as well as resolution).

0.7.0 — made resolution/horizon/interval calendar-aware periods, encoded as ISO-8601 duration strings (e.g. PT1H, P1M, P1Y) rather than integer milliseconds, in both the packed dataset names and the SQLite columns, so irregular periods (Month/Quarter/Year) can be represented distinctly from fixed spans.

0.6.0 — added owner_category to the association uniqueness key, making the owner identity the pair (owner_id, owner_category) and widening the unique indexes and idx_owner.

0.5.0 — changed the owner identifier to a signed 64-bit integer (owner_id).

0.4.0 — the baseline the Rust port of InfrastructureSystems.jl shipped with, and the version that introduced DATA_FORMAT_VERSION itself.

0.3.0 — switched the time unit from nanoseconds to milliseconds, renaming the SQLite *_ns columns to *_ms and encoding the packed dataset name's {res} field in milliseconds instead of whole seconds.

0.2.0 — introduced typed, multi-dimensional arrays and the two-mode array layout below.

0.1.0 — stored only 1-D f64.

Arrays Are Typed and N-Dimensional

Every stored array is a TypedArray: an element dtype, a shape [length, k1, k2, …] whose first axis is time and whose trailing axes are a fixed per-step element shape, and the row-major element values.

Byte order differs between the file and everything outside it. In the HDF5 file each dataset's datatype records its byte order, which is the native order of the host that wrote it, and libhdf5 converts on every read. Outside the file, a TypedArray's buffer is always little-endian: that is what the content hash covers and what every binding and the C ABI exchange. A reader using its own HDF5 tools gets correct values from any host's file without knowing either fact.

The supported dtypes and their stable integer codes (shared with the bindings and the C ABI). Codes 0–5 are the original set and never move; new widths are appended:

A scalar-per-step series has an empty element shape; a per-step tuple (e.g. the 3 coefficients of a quadratic cost curve) has element shape [3].

The dtype says how wide an element is. What those elements mean — and how a ragged piecewise curve is packed into a fixed-width row — is the association's element_type; see Element types. The dtype above is derived from it.

The HDF5 file does not describe its own element typing, and is not meant to: bool and u8 are the same byte on disk, and nothing in a dataspace says whether three f64s are a quadratic curve or three independent samples. Every read takes the dtype from the catalog's element_type instead. The two artifacts are one logical store — an .h5 without its .sqlite is not readable in any case — so this removes a second, weaker source of truth rather than adding a dependency. Where a backend does know a dtype independently (a packed dataset's name encodes it), the catalog's value is checked against it and a mismatch is an integrity error.

HDF5 Layout

Arrays live under a two-level group hierarchy, in one of two storage modes, with the explicit time axes in a sibling group of their own:

<name>.h5
├── attribute  data_format_version = "0.19.0"
├── attribute  compression         = "deflate:3:shuffle"
├── attribute  storage_backend     = "hdf5"
└── group      time_series/
    ├── group  single/
    │   ├── dset  sts_{dtype}_{shape}_{length}_{res}      packed  (length, cols, *element_shape)
    │   ├── dset  sts_{dtype}_{shape}_{length}_{res}_h    u8  (cols, 64)   # per-column hex hashes
    │   ├── dset  sts_{dtype}_{shape}_{length}_{res}__1   packed spill dataset
    │   ├── dset  nsts_{dtype}_{shape}_{length}_{tshash}  packed irregular cohort
    │   ├── dset  nsts_{dtype}_{shape}_{length}_{tshash}_h  u8  (cols, 64)
    │   ├── dset  arr_{hex_hash}                          standalone  [length, *element_shape]
    │   └── ...
    └── group  timestamps/
        ├── dset  tsv_{hex_hash}                          i64  [n]   # unix milliseconds
        └── ...

Datasets carry no dimension scales and no dimension names — shape is read straight off the HDF5 dataspace. On open the backend recovers each packed dataset's column count from the second extent of its dataspace, which is how per-dataset widths round-trip.

The hash companion {dataset}_h is a (cols, 64) array of u8 holding each column's 64 lowercase hex characters as raw bytes — not an HDF5 string dataset. An all-zero row marks a free slot.

Packed mode

Used for every static series: SingleTimeSeries, the underlying array of a DeterministicSingleTimeSeries, NonSequentialTimeSeries, and PersistentTimeSeries. Many arrays that share a (dtype, element_shape, length) and a time axis are column-packed into one dataset:

The trailing element is what pins the time axis, which is what makes packing meaningful at all: the chunking is timestamp-major, so row t of a dataset is "every column at the same instant". A regular series takes its axis from the resolution; an irregular one carries the axis explicitly, and two of them share one exactly when their timestamp vectors are equal — which the store already answers by content-addressing those vectors (see Timestamp vectors). The pool key is the time axis and never the series type, so a PersistentTimeSeries and a NonSequentialTimeSeries on the same instants share one nsts_… dataset, and identical values on them share one content-addressed array. The full 64-char hash is used, not a prefix: the name is parsed back into the pool key when the index is rebuilt at open, so a truncated form could let two distinct time axes collide into one pool.

The dataset shape is (length, cols, *element_shape) and chunking is (rows, cols, *element_shape) with rows = 1 in the ordinary case, so one HDF5 chunk holds a single timestamp across every column — making a read across series by timestamp one chunk, and a buffered bulk write fill whole chunks. cols is chosen per dataset: a managed write sizes it to the batch it has in hand, while an incremental one-at-a-time write path uses a default width (DEFAULT_COLS_PER_DATASET = 1000) and fills one of its slots per call. In both cases cols is capped so one chunk stays within a byte budget (MAX_CHUNK_BYTES = 1 MiB); a batch wider than the cap spills across datasets.

"The batch it has in hand" is the whole of an add_time_series_bulk call — or the whole span of an open transaction, for irregular cohorts as well as regular pools: whether an irregular series packs with others on its axis or stands alone is decided from the block's final membership at the commit, not from the one request an add can see. Nothing a transaction writes is durable until its outermost commit, so a single add inside one is buffered per pool rather than dropped into a growth-pool slot, and the buffered arrays are written with the same block writer at the commit, or when a read needs one of them to have a physical position. A loop of single adds inside one transaction therefore produces exactly the datasets one bulk add of the same items produces — so long as the span reaches its commit without materializing early; only an un-transactioned single add takes the default width.

The buffer is the store's, not the file's: nothing on disk records it, and a reopen never sees one. The byte ceiling below and an early materialization are what can break that equivalence: crossing the ceiling, or asking a buffered array for its physical location (Store::locate_array, which takes &mut self for that reason), writes the span out as it stands. Each split costs an extra dataset and nothing else — the datasets are still block-written and chunk-aligned.

Two limits keep that buffer from being unbounded, and both simply write a block out early — the same spill a too-wide batch already performs, costing an extra dataset and nothing else:

A pool's block width is the lower of two ceilings, and the buffer as a whole has a third:

• Per pool, one chunk row of columns — MAX_CHUNK_BYTES over the element block, exactly the width a bulk add's block spills at, so the span's datasets are the bulk add's datasets. It is deliberately not DEFAULT_COLS_PER_DATASET: a bulk add issued inside a transaction is buffered too, and a thousand-column cap cut a 100,000-series batch into a hundred datasets where the same batch outside a transaction writes ten, which every per-timestep read then paid for chunk by chunk.
• Per pool, MAX_PENDING_BYTES over one column's bytes — length × element_block. A chunk row is a count of columns and says nothing about how long they are, so for anything but a short series this is the ceiling that actually binds: scalar f64 is 131,072 columns by the rule above, but 30,500 steps is 244,000 bytes a column, so the pool spills at 550. The two together are min(MAX_CHUNK_BYTES / element_block, MAX_PENDING_BYTES / (length × element_block)).
• Across every pool, MAX_PENDING_BYTES = 128 MiB of unwritten arrays. Per-pool shares do not add up to a global one, so a span touching several shapes is held to the total as well; crossing it writes out the widest block.

MAX_PENDING_BYTES is the only one of the three a caller can move: Store::set_write_buffer_bytes in the Rust core, store.write_buffer_bytes = n in Python, set_write_buffer_bytes!(store, n) in Julia, infrastore_store_set_write_buffer_bytes across the C ABI. Raising it is how a run of single adds gets the dataset the bulk add of the same series writes, since a batch handed over as a list is written as one block with no budget applied at all — the caller is already holding it. The chunk-row ceiling above it does not move, and neither does the figure travel: it belongs to the handle that set it, not to the artifact, because it is a budget for the writing process rather than a property of the file.

A block of one is not a block. If a span ends up holding a single array for a pool, it fills a growth-pool slot rather than claiming a dataset sized to one column — chunked (1, 1), that would give a scalar f64 series an eight-byte chunk per timestep, whose per-chunk overhead dwarfs the data. It is the same rule add_time_series_bulk applies to a pool its batch gives one array. For an irregular pool the fallback is a standalone arr_ dataset instead of a slot, because an nsts_ pool is shared only by the series on that exact axis: a cohort of one is a dataset spread over length chunks for no reason, where a regular pool is shared by every series on the resolution. From two columns up the block is what the bulk add of those items writes.

This is a write-time policy like every other choice on this page: the layouts it produces are ones the format already had, so it does not affect data_format_version and stores written either way stay mutually readable.

rows rises above one when a single timestamp row would leave the chunk under MIN_CHUNK_BYTES = 32 KiB. A narrow dataset is where that bites hardest, and a dataset is narrow when the width had to be small — which above is MAX_PENDING_BYTES / length for a span, so a long series got a 512-byte chunk. That is small enough to cost on both sides: deflate has too little to work with, and HDF5's fixed per-chunk cost stops being rounding error.

Measured on 512 series of 262,144 f64 steps written one at a time in one span — a 64-column block, so a 512-byte timestamp row — carrying a daily-plus-annual profile with AR(1) noise rather than anything conveniently compressible:

The 512-byte chunking wrote a file larger than the raw data: at that size deflate's per-chunk overhead exceeds what it saves. Write time bottoms out around 16–32 KiB and file size keeps falling past it. Sweeps are flat at every chunk size — a sweep visits every chunk regardless, and the rows a chunk brings along are the next ones it wants, so the reader's main pattern is untouched.

Scattered single-timestamp reads are the one thing taller chunks cost in principle, since they decompress rows nobody asked for. In practice the measurement varied by ±0.4 s run to run and could not separate 4 KiB from 32 KiB; a consumer whose access pattern is genuinely scattered rather than swept should benchmark it rather than trust this note.

The floor reaches past the narrow blocks it is aimed at. A 1,000-column f64 growth pool is an 8,000-byte row and takes five rows per chunk; a year of hourly f64 in a 1,915-column span block is 15 KiB and takes three. On that wide shape the trade measured as about 15% more write time for about 3% less on disk.

Chunking is a write-time policy: it is not recorded in data_format_version, HDF5 readers do not care, and files written under either rule stay readable.

• Rows are timesteps, columns are series. Column i holds one complete series.
• Hash companion dataset. Each packed dataset has a sibling {dataset}_h dataset of u8, shaped (cols, 64). Row i holds the lowercase hex SHA-256 (64 characters) of column i as raw bytes, or 64 zero bytes if the column is free. This is the on-disk index: on open, the backend scans every …_h, decodes the non-zero rows, and rebuilds its hash → (dataset, column) map. (The backend also recovers each dataset's cols from its column dimension length, so per-dataset widths round-trip.)
• Spill. When a family's current dataset is full — a batch exceeds the column cap, or incremental writes fill a default-width dataset — the next write creates a spill dataset …__1, then …__2, and so on.

Standalone mode

Used for the dense forecast arrays (Deterministic, Probabilistic, Scenarios), and for a NonSequentialTimeSeries or PersistentTimeSeries whose time axis nothing else shares. Each array is its own typed, multi-dimensional variable named arr_{hex_hash} in the time_series/single group. There is no column packing and no companion hash variable — the variable name carries the hash.

• A lone explicit-time-axis series (NonSequentialTimeSeries or PersistentTimeSeries) is shaped [length, *element_shape] and chunked as a single whole-array chunk. Its explicit, strictly-increasing timestamps are not in the array: they are a shared time axis, stored once in the timestamps group below. Packing is only a win once a cohort is several columns wide — a packed dataset spreads one array over length chunks — so a series alone on its time axis stays standalone. Whether a given array is packed or standalone is a write-time choice with no effect on reads (they resolve by content hash and handle either), and one cohort can hold columns of both.
• Dense forecasts are shaped [H, count, *element_shape] (Deterministic), [num_percentiles, H, count, *element_shape] (Probabilistic), or [num_scenarios, H, count, *element_shape] (Scenarios), where count is the number of forecast windows. They are chunked in bounded blocks along the count (window) axis — full on every other axis, cols windows wide, where cols is the largest count keeping one chunk within the same 1 MiB budget the packed datasets use. Reading a single window therefore decompresses one block rather than the whole array, and the ForecastReader aligns its in-memory cache to the same block width so sweeping the window timeline decompresses each block exactly once. The chunk width is not recorded anywhere: it is a write-time storage choice that reads transparently regardless of the width a store was written with, so it does not affect the data-format version.

Timestamp vectors

The explicit time axis of a NonSequentialTimeSeries lives in time_series/timestamps/, one dataset per distinct vector, named tsv_{hex_hash} after the vector's content hash — the same hash the association row carries in timestamps_hash and the nsts_… pool name carries as {tshash}. Each is a 1-D i64 dataset of unix milliseconds, one element per timestamp, and carries the file's compression policy once it is big enough to be chunked (short vectors go in the object header, unfiltered, exactly as small standalone arrays do).

Milliseconds are the store's precision floor for every instant it records, so this is exact for anything a write accepts; a finer timestamp is refused rather than truncated. Ordinary numbers in an ordinary dataset: h5py reads a time axis with no help from this library.

Storing it here rather than in the catalog is what makes the timestamps data. Irregular series in a power-systems model overwhelmingly share one axis — event times, an outage schedule, a market timeline — so a thousand components sampled at the same instants hold one copy between them; but a store may hold many distinct axes, each of them long, and a .sqlite file is the wrong place for bulk numeric arrays. It also means the vectors travel with the artifact: a document round trip that carries locators rather than values still finds the timestamps where it finds the arrays. (Before 0.19.0 they were a delta-varint blob in a timestamp_sets catalog table.)

Vectors are shared, so removing one series never deletes one; see Deletion and compaction. verify_integrity reads each referenced vector back and rehashes it, exactly as it does an array — the name records the hash, so a dataset perturbed behind it is caught rather than silently served.

Compression

Compression is chosen at store creation and applies to every data variable, packed and standalone alike (the …_h hash variables are strings and are not compressed). The default is DEFLATE (zlib) level 3 with the byte-shuffle filter; the level (0–9) and shuffle can be changed, or compression turned off entirely. The choice is persisted in the compression global attribute and restored when the store is reopened, so later appends reuse the same filter:

An absent or unparseable attribute falls back to the default (deflate:3:shuffle), which is what such a file was written with. Compression is a storage detail only: arrays decode transparently regardless of the filter, so stores written with different settings stay mutually readable and data_format_version is unaffected by the choice.

Deletion and compaction

• Packed: deletion zero-fills both the column's hash row and the column's data, so no stale values are readable through a reused slot. The slot becomes reusable by the next compatible write. The dataset does not shrink.
• Standalone: deletion unlinks the HDF5 dataset. The object is unreachable immediately and stays gone across a reopen, but the space it occupied is not returned to the filesystem until a compaction (HDF5 cannot reclaim space in place). Re-adding the same content therefore rewrites the dataset rather than re-indexing it.
• Feature sets and timestamp vectors: because both are shared, deleting an association never deletes either; removing the last association that referenced one leaves it unreachable. compact() deletes them — the feature set as a catalog row, the timestamp vector as an unlinked HDF5 dataset — and reports the counts as feature_sets_reclaimed and timestamp_sets_reclaimed. Clearing a store is the exception: it orphans every one of both by construction, so it drops them outright rather than leaving a cleared store's worth of them for a compaction.

compact() on an on-disk store rewrites the .h5 file, because that is the only way HDF5 gives the freed space back:

1. The catalog is swept of unreachable feature sets, and the file of unreachable timestamp vectors; the catalog is then read for the live set — the catalog, not the file, is what makes an array or a time axis live.
2. Every live array is written into a fresh file at <store>.h5.repack, sibling to the original so the two share a filesystem. Layouts are planned from scratch: packed pools are created at exactly their cohort width rather than the growth-sized width an incremental write reserves.
3. The store's HDF5 handle is closed, the temp file is renamed over the original, and the store reopens on the result. A crash before the rename leaves the original untouched plus a stray .repack file, which the next compaction deletes.

What the new file therefore lacks: freed packed slots, unreferenced datasets (an interrupted bulk add's leftovers, unreferenced timestamp vectors, or tombstones left by a store written before deletion unlinked), and the slack in over-wide packed pools. The .sqlite half is not touched — arrays are content-addressed, so a different physical layout is invisible to it — and data_format_version is unchanged, because the rewrite emits the same format.

Compaction assumes the process running it is the file's only user. On Unix another process holding the file open keeps reading the pre-compaction inode; on Windows its lock makes the rename fail, and the error surfaces with the compacting store still open on the original file.

SQLite Schema

The catalog database is created with PRAGMA foreign_keys = ON and the following DDL (idempotent — CREATE TABLE IF NOT EXISTS).

time_series_associations

One row per association between an owner and a stored array.

The two content-address hashes are the last two columns. Column order is not load-bearing — every statement names its columns — so the layout is chosen for readability.

timestamps_hash is set only on NonSequentialTimeSeries rows and is NULL on every other type. It is a locator, not a value: the vector itself lives in the HDF5 file (see Timestamp vectors), and the same hash is the cohort key the nsts_… pools are named by.

time_reference

One TEXT column holds all four spellings: utc, zoneless, a fixed offset (-07:00), or an IANA zone name (America/Denver). That is unambiguous rather than merely hoped for, because the core refuses a zone name that reads as an offset or as either literal — the utc literal is lowercase precisely so the IANA zone UTC stays a distinct value.

NULL means unspecified, never utc. For query bounds it groups with the three zoned spellings (an instant bound is accepted, a wall-clock bound refused), but it is not a claim the timestamps were written as UTC.

Deliberately not indexed, and that does not change now that ListFilter::zoneless exists. The idx_component_field partial-index pattern is wrong here twice over: WHERE time_reference IS NOT NULL would exclude exactly the NULL rows that filter has to return, and the column is low-cardinality — a handful of distinct values across a whole store — so it is not selective enough to earn an index. In practice it is combined with owner_id or name, which are indexed.

The column does not reach storage. array_hash takes no timestamp input at all, the packed dataset names carry no timestamp, and initial_timestamp stays RFC 3339 UTC whatever the reference says — a -07:00 series stores 2024-01-01T07:00:00Z and the label -07:00, and the offset is applied on the way out. Two series with equal values therefore pool into the same dataset and share a data_hash regardless of their references, which is correct: they are the same numbers.

feature_sets

The expanded feature map, one row per key. The typed columns are populated according to value_kind.

Feature sets are content-addressed, exactly as arrays are: the table is keyed by the SHA-256 of the feature map, and one set is stored once and shared by every association whose features_hash matches. The association row already carries that hash, so no join column is needed. Two associations with the same features therefore reference the same rows here — including a DeterministicSingleTimeSeries and the SingleTimeSeries it was derived from, which is why transform_single_time_series writes no feature rows at all.

An empty feature map stores no rows.

There is deliberately no foreign key to time_series_associations and no cascade: rows here are shared, so deleting one association must not delete a set another association still uses. Removing the last association that referenced a set instead leaves it unreachable — the same deletion semantics as the HDF5 side's unreachable standalone variables. Store::compact sweeps unreachable sets and reports the count as feature_sets_reclaimed; clearing a store drops them all outright.

supplemental_attribute_associations

Which supplemental attributes are attached to which components. Columns match infrasys' table of the same name, whose logic this replaces. See Associations Between Entities for the data model.

CREATE TABLE supplemental_attribute_associations (
    id             INTEGER PRIMARY KEY AUTOINCREMENT,
    component_id   INTEGER NOT NULL,
    component_type TEXT    NOT NULL,
    attribute_id   INTEGER NOT NULL,
    attribute_type TEXT    NOT NULL
);

CREATE UNIQUE INDEX uq_sa_assoc
    ON supplemental_attribute_associations(component_id, attribute_id);
CREATE INDEX idx_sa_assoc_attribute
    ON supplemental_attribute_associations(attribute_id, component_id, component_type);

uq_sa_assoc makes the (component_id, attribute_id) pair the row's identity — the type columns are denormalized labels for filtering, so the same pair under different type names is a duplicate and surfaces as DuplicateAssociation. That index also serves lookups keyed on the component; idx_sa_assoc_attribute serves the reverse direction ("which components carry this attribute").

One attribute may be attached to many components; one component may carry many attributes. Only the exact pair is constrained.

parent_child_associations

Directed edges between components — a generator (parent) connected to a bus (child), say. Both endpoints are always components, which is why there is no category column: a supplemental attribute cannot appear here by construction.

CREATE TABLE parent_child_associations (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    parent_id   INTEGER NOT NULL,
    parent_type TEXT    NOT NULL,
    child_id    INTEGER NOT NULL,
    child_type  TEXT    NOT NULL
);

CREATE UNIQUE INDEX uq_parent_child
    ON parent_child_associations(parent_id, child_id);
CREATE INDEX idx_parent_child_child
    ON parent_child_associations(child_id, parent_id, parent_type);

Identity is the ordered (parent_id, child_id) pair, so the reversed pair is a different edge. There is no relationship-kind column: two components may be related at most once. Recording a second kind of edge between the same pair would need such a column added and uq_parent_child widened — which is a format change, unlike the tables themselves.

Properties shared by both association tables

Neither table has a foreign key — to time_series_associations or anywhere else — and neither cascades. The endpoints live in the consumer's object graph, not in this store, so the store never observes a component or attribute being deleted and a cascade could never fire. Removing a time series therefore leaves both tables untouched, and vice versa; consumers issue both calls when they want both effects.

Both are also independent of each other: the same integer may name a row in each without collision.

Compatibility

Both tables were originally added without bumping data_format_version, since the gate is exact equality with no upgrade path and bumping for a purely additive table would have made every existing store unreadable in exchange for nothing.

That is no longer how they land. 0.18.0 gave both an AUTOINCREMENT id, which is part of a table's declaration and so cannot reach an existing table through CREATE TABLE IF NOT EXISTS; the version check rejects an older catalog before this DDL runs. Only a store at the current version can contain them, and it always does.

One consequence of the additive era survives, and is still required: opening read-only cannot run DDL at all, so reads of a missing table return empty results (0, false, no rows) rather than failing. That tolerance is not vestigial — it is what lets a read-only open work on a store whose catalog was never opened for writing.

store_attributes

Free-form key/value provenance about the artifact as a whole — who built it, from what source system, under which of the consumer's own schema versions. The store never interprets a value, in the same spirit as a row's application_data, and nothing here participates in any identity, hash, or query plan. See Store attributes for the model.

CREATE TABLE store_attributes (
    key   TEXT PRIMARY KEY NOT NULL,
    value TEXT NOT NULL
);

key is the primary key, so a set is an upsert rather than an append: an artifact records one creator, not a history of them. Values are TEXT; a caller wanting structure stores JSON, which is what application_data already asks of them. Keys beginning with infrastore. are reserved and refused on the write path — by the store, not by a CHECK, so a future build can stamp its own facts through a deliberate internal write without a format change.

The table is additive, like the two association tables: an existing store gains it on its first writable open, so it needed no CATALOG_SCHEMA_REVISION bump. A read-only open of a catalog written before it existed cannot run that DDL, so reads degrade to the empty answer rather than erroring. The table is not carried by the OpenAPI export or import, which have no place for it.

schema_version

CREATE TABLE schema_version (version INTEGER NOT NULL);

A single-column table holding the catalog's own schema revision — CATALOG_SCHEMA_REVISION, which is 2 as of this build. It holds at most one row: the DDL creates the table, and the migration ladder writes the revision, replacing whatever was there. A fresh catalog is stamped with the current revision directly rather than seeded at 1 and walked up the ladder.

An absent table, or a present one holding no row, reads as revision 1 — the pre-ladder shape, which is where the ladder starts. This tracks the SQLite schema alone and is distinct from the HDF5 data_format_version attribute, which governs the artifact as a whole and is the value open validates; the two move independently.

catalog_identity

CREATE TABLE catalog_identity (generation TEXT NOT NULL);

Holds zero or one row pairing this catalog with one HDF5 file, whose catalog_generation root attribute carries the same value. Store::open compares them and rejects a mismatch with MismatchedArtifact, which is what turns an interrupted persist_to — two renames that cannot be made atomic together — into a loud error instead of a store that quietly disagrees with itself. It also catches one half being copied without the other.

Added additively, so it lands on an existing store's first writable open without a data_format_version change. A store predating it has no row here and no root attribute, and reads as unstamped rather than as a mismatch. Because a read-only open cannot run the DDL, every read of this table tolerates the table being absent.

Indexes

CREATE UNIQUE INDEX uq_ts_assoc ON time_series_associations
    (owner_id, owner_category, time_series_type, name, resolution, interval, features_hash);
CREATE UNIQUE INDEX uq_ts_assoc_coalesced ON time_series_associations
    (owner_id, owner_category, time_series_type, name,
     COALESCE(resolution, ''), COALESCE(interval, ''), features_hash);

CREATE INDEX idx_hash       ON time_series_associations(data_hash);
CREATE INDEX idx_owner      ON time_series_associations(owner_id, owner_category);
CREATE INDEX idx_resolution ON time_series_associations(resolution);

-- Secondary indexes for the filter / discovery surface.
CREATE INDEX idx_ts_type        ON time_series_associations(time_series_type);
CREATE INDEX idx_name           ON time_series_associations(name);
CREATE INDEX idx_owner_type     ON time_series_associations(owner_type);
CREATE INDEX idx_category_owner ON time_series_associations(owner_category, owner_id);
CREATE INDEX idx_interval       ON time_series_associations(interval);
CREATE INDEX idx_component_field ON time_series_associations(component_field)
    WHERE component_field IS NOT NULL;

idx_component_field is the only partial index, because component_field is the only optional column anything filters on: a store that never sets it would otherwise pay index maintenance on every insert to record one NULL per row and buy nothing, so the WHERE clause makes that case cost zero entries. SQLite still uses it for the predicate the filter issues — component_field = ? cannot be true of a NULL whatever the parameter binds to — but it can never serve an IS NULL query, which is the same reason ListFilter::component_field cannot select the rows that left the field unset.

An index over a column that already exists is additive: a store gains any it lacks on its first writable open, at a one-time build cost proportional to catalog size, with no data_format_version change. idx_component_field is the exception, because it names a column introduced by a bump — it applies only to a catalog already carrying component_field, and an older store never reaches the DDL, being rejected by the version check first.

Together the two unique indexes enforce identity uniqueness; a violation surfaces as DuplicateTimeSeries. Both owner_id and owner_category are part of the key, so a component and a supplemental attribute that share an owner_id are independent owners. interval is part of the key, so two forecasts of one variable at the same resolution but different intervals are distinct series. SQLite treats NULL values as distinct in a UNIQUE index, so uq_ts_assoc does not constrain rows with a NULL resolution or interval (e.g. NonSequentialTimeSeries, or any static series, which carry no interval). uq_ts_assoc_coalesced covers that case by folding NULL to the empty-string sentinel via COALESCE before enforcing uniqueness (the empty string is never a valid ISO-8601 period).

These two were named uq_assoc and uq_assoc_coalesced before the association tables above existed and made a bare "assoc" ambiguous. The DDL drops the old names before creating the new ones, so a store written by an earlier build is renamed in place on its first writable open rather than accumulating two equivalent index pairs. Index names are not part of the on-disk contract, so this carries no data_format_version change.

time_series_readable (view)

CREATE VIEW time_series_readable AS
SELECT id, owner_id, owner_type,
       CASE owner_category WHEN 0 THEN 'Component'
                           WHEN 1 THEN 'SupplementalAttribute'
                           ELSE 'unknown(' || owner_category || ')' END AS owner_category,
       CASE time_series_type WHEN 0 THEN 'SingleTimeSeries'
                             WHEN 1 THEN 'NonSequentialTimeSeries'
                             WHEN 2 THEN 'Deterministic'
                             WHEN 3 THEN 'DeterministicSingleTimeSeries'
                             WHEN 4 THEN 'Probabilistic'
                             WHEN 5 THEN 'Scenarios'
                             WHEN 6 THEN 'PersistentTimeSeries'
                             ELSE 'unknown(' || time_series_type || ')' END AS time_series_type,
       name,
       initial_timestamp, resolution, length, horizon, interval, count,
       units, quantity_kind, unit_system, time_reference, component_field,
       element_type, element_shape, application_data,
       lower(hex(data_hash))       AS data_hash,
       lower(hex(features_hash))   AS features_hash,
       lower(hex(timestamps_hash)) AS timestamps_hash
FROM time_series_associations;

This view decodes both discriminants and every content hash, so it is the convenient thing to query by hand. Nothing in the library reads it.

A projection of the association table with both hashes hex-encoded, for humans opening the catalog in sqlite3 — see Reading the catalog by hand. Nothing in the library reads it. It costs no storage, is created by the same idempotent DDL as the tables above, and so lands on an existing store's first writable open without a data_format_version change.

Field Encoding Notes

• Timestamps are RFC 3339 strings in UTC, holding a whole number of milliseconds. The format could carry finer digits and a store written before the precision rule may, so a reader must parse the full string; but no current write path produces one — see timestamp precision.
• Periods (resolution, horizon, interval) are canonical ISO-8601 duration strings in SQLite (PT1H, P1M, P1Y), and the packed dataset name's {res} field uses the same encoding. Calendar periods (Month/Quarter/Year) are stored distinctly from fixed spans.
• Hashes are raw 32-byte BLOBs in SQLite, lowercase hex in HDF5 (the …_h dataset for packed arrays, the arr_ dataset name for standalone arrays). BLOB rather than hex TEXT in SQLite because the two hash columns sit in the association table, in idx_hash, and in both unique indexes — hex would cost roughly 32% more catalog space — and because a BLOB literal (X'…') compares case-insensitively while a hex TEXT column would not. The time_series_readable view supplies the readable form.
• element_shape is the per-step shape only (the trailing axes); the time length is a separate column.

Discriminant encoding

owner_category and time_series_type are stored as small INTEGER codes, not names:

Both columns sit in the two wide unique indexes, and time_series_type additionally in idx_ts_type while owner_category sits in idx_owner and idx_category_owner. A 1-byte code instead of a 9–29 byte string shrinks those indexes by roughly 35% and the whole catalog by roughly 29% on a 400k-row store, and makes type-scoped scans about 1.2x faster warm (1.4–1.5x under page cache pressure). Point lookups by key are unaffected — those are dominated by the 32-byte features_hash.

Two code assignments are load-bearing and cannot be reordered without a version bump:

• Deterministic (2) and DeterministicSingleTimeSeries (3) are adjacent. A request for Deterministic matches both (see the data model), so adjacency lets that widen to time_series_type BETWEEN 2 AND 3 — one index seek instead of a two-value IN.
• The static types (0–1) and forecast types (2–5) are contiguous blocks, so the static and forecast summary queries each scope with a single BETWEEN.

The C ABI uses the same integers for its ts_type arguments, so there is one mapping rather than two. The time_series_readable view decodes both columns for hand inspection.

Inspecting a Store by Hand

The quickest route is the CLI, which resolves the hash-to-location step for you:

infrastore --store system.h5 store-info          # both file paths, format version, compression
infrastore --store system.h5 arrays              # every distinct array, its location and sharers
infrastore --store system.h5 info --name load    # one series: data_hash, hdf5_dataset, hdf5_column

To go straight at the files:

h5ls -r system.h5                        # groups, datasets, dtypes, shapes
h5dump -A -n system.h5                   # root attributes and the object list
sqlite3 system.h5.sqlite '.schema'
# Query the view, not the table, to see decoded discriminants.
sqlite3 system.h5.sqlite \
  'SELECT name, time_series_type, element_type, element_shape, length FROM time_series_readable;'
# Both association tables are present in every current-format store.
sqlite3 system.h5.sqlite 'SELECT * FROM supplemental_attribute_associations;'
sqlite3 system.h5.sqlite 'SELECT * FROM parent_child_associations;'

Reading the catalog by hand

sqlite3 renders a BLOB as raw bytes in its default list mode and in .mode box / .mode json, which corrupts the terminal (and, in box mode, the table borders). Use the time_series_readable view, which is the association table with both hashes hex-encoded:

sqlite3 system.h5.sqlite 'SELECT name, data_hash FROM time_series_readable;'

The view spells hashes in lowercase, matching hash_hex in the core, every binding, and the CLI, so a value copied from it compares equal to one printed anywhere else. Against the base table, use hex(data_hash) or .mode quote; hex() returns uppercase.

The view is created by the same idempotent DDL as the tables, on a store's first writable open, so an older store gains it without a format bump. Nothing in the library reads it.

Mapping an association to its bytes

Read the association's data_hash. For a standalone array, read the dataset named arr_<hex_hash> directly. For a packed array, hex-encode the hash and find the matching row in the relevant sts_…_h dataset; that row index is the column index into the sts_… dataset. Note that a packed pool which fills up spills into {name}__1, {name}__2, so the dataset holding a given array is not derivable from its metadata — scan the _h datasets, or let infrastore info / infrastore arrays report the resolved dataset and column.

A NonSequentialTimeSeries row's time axis needs no scan: hex-encode its timestamps_hash and read time_series/timestamps/tsv_<hex_hash>, an i64 vector of unix milliseconds in the row's own order.