Julia Developer Guide

This guide covers building on InfraStore.jl, the Julia package that wraps the C ABI, from installing it to the calls a consumer package makes. For exact signatures see the Julia API reference.

For complete programs rather than isolated snippets, use the repository's runnable Julia examples. They cover static, non-sequential, persistent, deterministic, probabilistic, and scenario data; fixed tuples; every function-valued element type; feature-based selection; and conversion to DataFrames.

Install

Julia 1.11 or newer. InfraStore.jl is registered in General, and the native library comes with it:

using Pkg
Pkg.add("InfraStore")

Pkg downloads the libinfrastore_ffi artifact for your platform from the matching GitHub Release (Artifacts.toml in the package pins its URL and hash). The library is linked statically against HDF5 and zlib, so there is no HDF5_jll and no system HDF5 involved, and the HDF5 version behind the on-disk format is the one infrastore pinned. Artifacts exist for Linux x86_64 and aarch64 (glibc), macOS x86_64 and Apple Silicon, and Windows x86_64; on any other platform, build the library yourself and use the override below.

That is the whole recipe for a consumer package.

From a checkout

InfraStore.jl resolves the cdylib at first use, in this order:

1. The INFRASTORE_LIB environment variable — the development override.
2. The libinfrastore_ffi artifact Pkg downloaded at install time.

So developing against a working tree means building the library and exporting the variable. This needs the build tools (cmake, a C compiler, protobuf), but no system HDF5:

cargo build -p infrastore-ffi --release
export INFRASTORE_LIB=$PWD/target/release/libinfrastore_ffi.dylib  # .so on Linux

julia --project=julia/InfraStore.jl -e 'using Pkg; Pkg.instantiate()'
julia --project=julia/InfraStore.jl julia/InfraStore.jl/test/runtests.jl

using InfraStore always works; the resolution happens on the first call that reaches the native library, and the path is cached for the rest of the session — so set the variable before that first call, in the same shell that launched Julia. To use the checkout from your own project, Pkg.develop(path="/path/to/infrastore/julia/InfraStore.jl") with the variable set.

If it does not load

• Could not locate libinfrastore_ffi at <path>. — The resolved path is not a file. If <path> is under .julia/artifacts, the artifact download did not complete: Pkg.instantiate() (or Pkg.add("InfraStore") again) fetches it. If it is your own path, export INFRASTORE_LIB before the first store call.
• could not load library — Check the path exists and has the right extension for your OS (.dylib on macOS, .so on Linux, .dll on Windows), and that you built with --release if your variable points at target/release.
• InvalidParameterError on add — owner_id must be an integer (e.g. 42, an Int64), and features values must be JSON scalars.

Load the Package

using Dates, InfraStore

Exported names include Store, SingleTimeSeries, NonSequentialTimeSeries, PersistentTimeSeries, the forecast structs (Deterministic, Probabilistic, Scenarios), OwnerCategory (Component, SupplementalAttribute), the add_time_series! / read_by_id / list_metadata family, and transform_single_time_series!. The store type is named Store.

Open or Create a Store

# In-memory.
store = Store(in_memory=true)

# On disk: writes system.h5 and system.h5.sqlite.
store = Store(in_memory=false, path="system.h5")

# Reopen read-only.
store = open_store("system.h5"; read_only=true)

A Store registers a finalizer, so nothing leaks if you never close one — but the GC decides when, and on disk that means an open file handle and a held SQLite write lock for an unbounded time. Prefer the do-block forms, which release the store on the way out including on a throw:

Store(in_memory=true) do store
    add_time_series!(store, 42, "Generator", Component, ts)
end

open_store("system.h5"; read_only=true) do store
    only(list_metadata(store; owner_id=42, owner_category=Component, name="load"))
end

open_copy has one too. close!(store) is the explicit form when a do-block does not fit — a long-lived store held by a consumer package, say — and is idempotent, so closing a store the finalizer later reaps is fine.

Add a Series

# `name` ("load") is a required field on the struct.
ts = SingleTimeSeries(DateTime(2024, 1, 1), Hour(1), collect(100.0:123.0), "load")

A bare DateTime carries no zone, so this package reads one as a wall clock — the store holds its fields unchanged and records the spelling as ZonelessReference(). If your timestamps are genuinely zoned, using TimeZones and pass a ZonedDateTime instead — it names an instant on its own, and is accepted anywhere a DateTime is:

using TimeZones
ts = SingleTimeSeries(
    ZonedDateTime(DateTime(2024, 1, 1), tz"America/Denver"),   # = 2024-01-01T07:00Z
    Hour(1), collect(100.0:123.0), "load",
)

Reads return a bare DateTime holding the instant either way, with the recorded spelling beside it (zoned_timestamp fuses the two back together); see Time and resolution conversions.

id = add_time_series!(
    store,
    42,
    "Generator",
    Component,
    ts;                                   # name and descriptors come from ts
    features = Dict("model_year" => 2030),
)
# `id` is the catalog row's id: how every read and removal addresses
# the series, and one integer to keep in your own model.

Notes:

• owner_id is an integer (Int64) — the component identifier, e.g. 42.
• resolution is a Period such as Hour(1) or Minute(5).
• features is a Dict serialized to JSON, so values must be JSON scalars (Int, Float64, Bool, String). String features are supported and round-trip unchanged.
• Adding a duplicate identity throws DuplicateTimeSeriesError.
• For a sparse step function — a monthly fuel price, say — use PersistentTimeSeries, worked through in Step Functions.

add_time_series! returns the catalog row's id as an Int64 (see Association ids). Every read, removal and copy takes that id; list_metadata is how you recover one for a series you did not just write.

Two more rules worth knowing up front. Stored instants and periods are millisecond-precision: a Microsecond(1) resolution, a Millisecond(0) one, or a negative period is refused with InvalidParameterError (query bounds are unconstrained). And a Store is not thread-safe: confine it, and any reader built from it, to one task, or guard every call with your own lock — concurrent calls are undefined behavior, not just a race on results.

Descriptors

Beyond units, an association can carry quantity_kind (what the values measure — "ActivePower"; the one record of what per-unit values mean), unit_system (NaturalUnits or ComponentBase; nothing means unspecified, not natural units — the store rescales nothing), component_field (the field on the owning component these values vary — "max_active_power"; also a filter), and application_data (an opaque string returned verbatim — the package-owned slot). They can be set on the struct, where they become the add_time_series! defaults, or passed as keywords:

ts = SingleTimeSeries(DateTime(2024, 1, 1), Hour(1), collect(100.0:123.0), "load";
                      units = "MW", quantity_kind = "ActivePower",
                      unit_system = NaturalUnits, component_field = "max_active_power",
                      application_data = "{\"source\": \"weather_year_2012\"}")
id = add_time_series!(store, 42, "Generator", Component, ts)   # keeps all five

A series also records a time_reference — how its timestamps were spelled — inferred from the timestamp it was built with. A bare DateTime is a wall clock (ZonelessReference()); a ZonedDateTime keeps its zone, so a Denver series renders correctly on both sides of every DST transition. Reads still return a DateTime holding the instant, with the reference beside it; using TimeZones adds zoned_timestamp to fuse them back together losslessly.

None of them is part of a series' identity or of either content hash, so two adds that differ only in a descriptor are a duplicate. See Optional Descriptors and Time references.

Add many series at once

AddBatch accepts the same add_time_series! calls as a Store but only accumulates them; add_time_series_bulk! commits the whole batch in one catalog transaction and takes the block-sized HDF5 write path, so same-shaped series land in the same packed dataset.

batch = AddBatch()
for (id, ts) in series
    add_time_series!(batch, id, "Generator", Component, ts)
end
ids = add_time_series_bulk!(store, batch)   # Vector{Int64}, in input order; all-or-nothing

This is an order of magnitude faster than a bare loop of single adds, which pays one catalog transaction and one HDF5 flush per series. It is not faster than that same loop inside a transaction, which buffers and writes the identical datasets. What separates the two is one thing: the batch is written as a single block whatever its size, because you are already holding it, where the transaction holds its buffered adds to write_buffer_bytes and spills past it. Reach for the batch when the whole cohort is in hand; reach for the loop when you would rather add each series as you build it than hold them all first, and raise the budget if you want the single dataset back.

Transactions

Several operations that must take effect together — replacing a series is an add plus a remove — go inside a transaction. Removals are reversible only there; outside one the array bytes are reclaimed immediately.

transaction(store) do
    new_id = add_time_series!(store, 42, "Generator", Component, updated)
    remove_by_ids!(store, [old_id])
end   # committed if the block returns, rolled back if it throws

Blocks nest (each level is a savepoint), and the store holds the SQLite write lock until the outermost one ends. begin_transaction! / commit_transaction! / rollback_transaction! are the explicit form.

A transaction is also where a run of single adds belongs. Nothing it writes is durable until the outermost commit, so packed adds inside one are buffered per shape group and written as one block at the commit — the datasets add_time_series_bulk! of the same series would produce, without having to hold the batch yourself. Two things qualify it. The buffer spills a block early — an extra dataset, nothing else, the same split a batch that wide gets — once a group reaches the narrower of two widths, or once the unwritten arrays across every group cross 128 MiB:

• the columns one chunk row holds: 131,072 for a scalar Float64;
• 128 MiB divided by one column's bytes, which for anything but a short series is the ceiling that actually binds — a 30,500-step Float64 series is 244 KB a column, so its group spills at 550.

And a span holding a single array fills a shared-pool slot rather than claiming a dataset one column wide.

How wide a dataset a span writes

The 128 MiB is a default, not a law. set_write_buffer_bytes! moves it, and with it how wide a dataset a run of single adds can produce:

set_write_buffer_bytes!(store, 1 << 30)   # 1 GiB
transaction(store) do
    for (id, ts) in series
        add_time_series!(store, id, "Generator", Component, ts)
    end
end
# one dataset, however many series that was

Raised far enough, the loop writes exactly what add_time_series_bulk! of the same series writes — and the memory it costs is the memory the batch was holding anyway. Measured on 400 hourly year-long Float64 series, the two are already the same file in comparable time (~0.09 s either way, and ~0.12–0.14 s for a NonSequentialTimeSeries cohort on one axis); at 2,000 they part only over that extra dataset, ~0.30 s as one batch against ~0.38 s as a loop, and raising the budget closes it.

write_buffer_bytes(store) reads the figure back. It belongs to the handle, not to the artifact — nothing is persisted, and a store reopened elsewhere is back to 128 MiB. It is a budget for the writing process, so a machine that cannot afford another machine's choice does not inherit it. Lowering it mid-transaction writes out whatever the buffer already holds beyond the new figure; zero throws, since a pool's width floors at one column and a zero budget would mean a dataset per array rather than no buffering at all. The chunk-row ceiling above it does not move.

Values that are not numbers

A timestep's value can be a function rather than a number — a cost curve re-offered every hour, a pair of coefficients, a fixed-width tuple. Hand the constructor those values and it does the rest: it packs them into the array the store holds and names the element_type they imply.

curves = [
    PiecewiseLinear([(x = 30.0, y = 1155.0), (x = 100.0, y = 4120.0)]),
    PiecewiseLinear([(x = 30.0, y = 1353.0), (x = 65.0, y = 2730.0), (x = 100.0, y = 4223.0)]),
]

ts = SingleTimeSeries(t0, Hour(1), curves, "variable_cost")
ts.element_type            # "piecewise_linear" -- derived, not declared

id = add_time_series!(store, 42, "Generator", Component, ts)
read_by_id(store, id).data == curves      # true

There is no separate constructor for this and nothing to declare. A Vector{PiecewiseLinear} says what it is, so element_type= is only for the numeric case, where the numbers alone cannot say what they mean — and one that contradicts the values is an error rather than an override. The struct goes on holding the values you gave it; encoding happens at the ABI boundary, which is why a read hands back the same thing a write was given.

A case covered end to end in the runnable single_custom_elements.jl example. InfraStore.jl ships four value types plus NTuple{N,Float64}:

Every series type takes them, including the irregular ones and the forecasts. A forecast's values keep their window shape rather than arriving flat, since a Julia array carries its own:

# [H = 2, count = 2]: a curve for every (horizon step, window) pair.
det = Deterministic(t0, Hour(1), Hour(2), Hour(1), 2, reshape(curves4, 2, 2), "offer")

Two things follow from Julia arrays carrying their element type that are worth knowing. An empty series still names itself — NTuple{3,Float64}[] is a tuple(3,f64) series with no rows, which a language whose empty list is untyped cannot express. And a metadata row's time_series_type is the full parameterized type, SingleTimeSeries{PiecewiseLinear, 1}, so it describes the values rather than their packing.

raw = true on a read hands back the packing instead, and the per-timestamp readers are deliberately never decoded — they are the simulation path, and their numbers are physical. encode_element_values / decode_element_values are the same two directions as free functions, for an array with no series around it, and the door a consumer extends to encode and decode its own domain types with no conversion step. See Element values for both.

Read a Series

got = read_by_id(store, id)
@assert got.data == ts.data
println(got.initial_timestamp, " ", got.resolution)   # resolution comes back as Millisecond

read_by_id also takes a window — start_time plus a len of timesteps or a count of windows — which is checked: an over-long request throws rather than quietly returning less.

To read many whole series at once — e.g. loading everything for a plot — read_by_ids takes a vector of ids and returns one struct per id in the same order (each of its stored type, so the result is a Vector{Any}), reading each packed dataset's column span once instead of re-reading every chunk per series. Its time_range keyword is the other kind of slice, which clips:

series = read_by_ids(store, ids)
window = read_by_ids(store, ids; time_range = (t0, t1))   # the same clip on every series

Attribute-Based Lookups

Beyond key handles, InfraStore.jl can resolve a series directly from its attributes — convenient when a caller keeps its own identifiers (as an InfrastructureSystems.jl-side store does):

meta = get_metadata_by_id(
    store,
    42,
    Component,        # owner_category; the owner is the (owner_id, owner_category) pair
    "load";
    resolution = Hour(1),
    features = Dict("model_year" => 2030),
)
# meta :: TimeSeriesMetadata — the whole record: owner_id/owner_type/owner_category,
#          name, time_series_type, data_hash, initial_timestamp, resolution, length,
#          horizon/interval/count, percentiles, element_type, element_shape, features,
#          units, quantity_kind, unit_system, component_field, application_data

values = get_array_by_hash(store, meta.data_hash)     # Vector{Float64}; pass ::Type{T} for other dtypes

# Identify, then act. `list_metadata` answers which series exist and hands back
# the id; every read and removal takes that id.
row = only(list_metadata(store, owner_id = 42, name = "load",
                         resolution = Hour(1),
                         exact_features = Dict("model_year" => 2030)))
got = read_by_id(store, row.id)
remove_by_ids!(store, [row.id])

# `has_time_series` stays attribute-addressed: it is an index probe that reads no
# row, so routing it through a resolution would cost more than it answers.
present = has_time_series(store, 42, Component, "load";
                          resolution = Hour(1), features = Dict("model_year" => 2030))

exact_features matches the feature map exactly — the series above was added with model_year = 2030, so features (a subset match) would also select a sibling carrying more. The plain features keyword is the right one for "every series tagged scenario = high"; a package that resolves partial user queries lists, then decides what more than one match means.

Forecasts

InfraStore.jl exposes Deterministic, Probabilistic, and Scenarios structs that wrap a native AbstractArray in the type's logical shape (the wrapper derives the dtype and dims and serializes the buffer row-major). Construct one and add it through the generic add_time_series!:

data = zeros(Float64, 24, 7)   # (horizon_count, count)
fc = Deterministic(
    DateTime(2024, 1, 1), Hour(1), Hour(24), Hour(24), 7, data, "load_fc"; units = "MW"
)
id = add_time_series!(
    store,
    42,
    "Generator",
    Component,
    fc,                         # name and units come from fc
)

got = read_by_id(store, id)   # the id add_time_series! returned
values = got.data             # Float64 matrix, shape (24, 7)

Probabilistic(initial_timestamp, resolution, horizon, interval, count, percentiles, data, name) carries the percentile vector, and Scenarios(initial_timestamp, resolution, horizon, interval, count, data, name) takes scenario_count from data's leading axis. Every forecast constructor also accepts a application_data= keyword. A read names only an id, so the row's own type decides what comes back — there is no requested type to disagree with it.

If two forecasts of one owner/name/type differ only by interval (say day-ahead and intra-day), pass interval= to the listing to pin the one you want; without it it returns both:

row = only(list_metadata(store; owner_id = 42, name = "load_fc",
                         resolution = Hour(1), interval = Hour(6)))

A DeterministicSingleTimeSeries is not added directly — derive one from the stored SingleTimeSeries with transform_single_time_series!, which returns a TransformOutcome whose transformed field is the count. It optionally restricts the transform to one owner_category and/or one resolution, and dry_run = true runs every check without writing:

n = transform_single_time_series!(store, Hour(24), Hour(24)).transformed
outcome = transform_single_time_series!(store, Hour(24), Hour(24);
                                        owner_category = Component, resolution = Hour(1),
                                        normalize_single_window = true,
                                        require_uniform_forecast_grid = true)

The two policy flags reproduce InfrastructureSystems.jl's rules (it passes both as true); see the reference for what each enforces.

Filtering for Deterministic also matches a transformed DeterministicSingleTimeSeries, so you find a forecast the same way whether it was added densely or derived — and either reads back as a Deterministic, since a DST has no materialized struct. Each row still reports the concrete form it is, so transform_single_time_series! needs no separate enumeration path:

for row in list_metadata(store; owner_id = 42, time_series_type = Deterministic)
    row.time_series_type <: DeterministicSingleTimeSeries   # derived, or densely stored?
    series = read_by_id(store, row.id)   # a Deterministic either way
end

transform_single_time_series! also reports the ids it wrote on its TransformOutcome.written, so a caller can reference a view it just derived without listing the store to find it again.

has_time_series takes the time series type as its first argument to address anything other than a SingleTimeSeries (and takes the same resolution / interval / features keywords). copy_time_series! takes the source id and re-points that series at another owner without duplicating data — it writes one association row against the same content-addressed array, preserving the stored type (a DST stays a DST) — and returns the copy's own id:

has_time_series(Scenarios, store, 42, Component, "wind"; resolution = Hour(1))
src = only(list_metadata(store; owner_id = 42, name = "load")).id
copy_time_series!(store, src, 43, "Generator")

Every time_series_type filter keyword takes the Julia type as well:

list_metadata(store; time_series_type = Deterministic)
get_resolutions(store; time_series_type = SingleTimeSeries)

A metadata row's time_series_type is the full type — SingleTimeSeries{Float64,1}, Deterministic{Float32,3} — so a row names what a read of it hands back rather than only which of the six kinds it is, and a consumer holding InfrastructureSystems.jl-style parameterized types gets them back intact:

md = get_metadata_by_id(store, id)
md.time_series_type == typeof(read_by_id(store, id))   # every stored type but DST
md.time_series_type <: SingleTimeSeries                # ask for the kind with <:, not ==

A derived DeterministicSingleTimeSeries is the exception: its row keeps the DST tag while a read of it hands back a Deterministic with the same {T,N}. Dispatch on the read's type when the two have to agree, and on the row's when you mean "was this derived?".

That type passes straight back into any filter, has_time_series, or reader. The parameters are ignored there — a series is addressed by identity, which carries no element type — so they never narrow a match; they are accepted so a row you just read round-trips without being taken apart first.

The low-level get_metadata_by_id + get_array_by_hash path is still available for raw access. See the Julia API reference.

Per-Timestamp Reads (Simulation Loop)

read_by_id hands back a whole series or forecast. Simulations instead walk the timeline and, at each timestamp, want the value of every series at that instant. For that, build a reader once and drive it in a loop — it pins one resolution and reuses its output buffers, so the loop allocates almost nothing. StaticReader serves SingleTimeSeries; ForecastReader serves forecasts. (Full signatures: Julia API reference.)

Static series

reader = build_static_reader(store; resolution = Hour(1))
grid = static_grid(reader)                 # StaticGrid: initial_timestamp, resolution, length
for k in 0:(grid.length - 1)
    static_read!(reader, grid.initial_timestamp + grid.resolution * k)
    for (gi, g) in enumerate(static_groups(reader))
        vals = static_values(reader, gi)   # (num_columns, element_dims...); column j ↔ g.ids[j]
    end
end

Series are grouped by (dtype, element_shape); each group's static_values is one dense array whose columns line up with the group's ids. All matched series must share one grid (initial_timestamp + length), validated at build.

Most real systems do not meet that — a year of load beside a week of an outage schedule — and the build then errors, naming the series that diverges. Give the reader a span instead of letting it inherit one:

reader = build_static_reader(store; resolution = Hour(1),
                             window_start = DateTime(2024, 1, 1, 7),
                             window_length = 8760)  # optional: without it, as far as all reach

Each column then reads at an offset of its own, so ragged series sweep together. The span is checked rather than clamped: a series that does not cover it errors, naming that series, and the anchor must land on each series' own step boundaries. See reader windows.

When the odd series out should not take part at all — a stray day of data beside a year of it is a different component, not a shorter view of the same sweep — filter to one grid instead:

reader = build_static_reader(store; resolution = Hour(1),
                             initial_timestamp = DateTime(2024, 1, 1, 7), length = 8784)

The window sweeps a span across whatever matched; the filter matches only the series already on that grid, and reaches list_metadata and remove_by_filter! the same way. See selecting one grid.

Forecasts

reader = build_forecast_reader(store, Deterministic; resolution = Hour(1))
tl = forecast_timeline(reader)             # ForecastTimeline: initial_timestamp, resolution, interval, count
for k in 0:(tl.count - 1)
    forecast_read!(reader, tl.initial_timestamp + tl.interval * k)
    for (i, e) in enumerate(forecast_entries(reader))
        window = forecast_values(reader, i)   # shape e.window_shape, for e.key
    end
end

A Deterministic reader is abstract — it also includes any DeterministicSingleTimeSeries (read into identical windows).

Shared forecasts are read once

Forecasts that share a backing array (deduplicated identical data, or several DeterministicSingleTimeSeries over one SingleTimeSeries) collapse to a single window slot. forecast_read! reads each slot from the .h5 file once per timestamp, so a forecast shared by 10 components costs one read, not ten. forecast_num_slots(reader) is the physical read count, and each ForecastEntry.slot says which slot an entry uses — group by slot to materialize each unique window only once:

forecast_read!(reader, t)
windows = Dict{Int, Any}()
for (i, e) in enumerate(forecast_entries(reader))
    w = get!(() -> forecast_values(reader, i), windows, e.slot)
    # apply w to e.key's owner
end

Step Functions (PersistentTimeSeries)

A PersistentTimeSeries is a sparse step function: a strictly increasing vector of breakpoints plus one value each, where the value at an arbitrary instant is the one belonging to the greatest breakpoint at or before it. The motivating data is a monthly fuel or gas price curve — a dozen breakpoints a simulation reads at timestamps that almost never land on one. The same curve stored as a NonSequentialTimeSeries would throw at nearly every step, because an irregular series has no value between its timestamps; that difference in read semantics is the whole reason this is a separate type. See Time series types for the model.

Build and add

The struct takes the same arguments as NonSequentialTimeSeries — and, like every series struct here, its descriptors as keywords. A bare DateTime is read as a wall clock and a ZonedDateTime as the instant it names, exactly as for the other types:

breakpoints = [DateTime(2024, m, 1) for m in (1, 4, 7, 10)]
prices = PersistentTimeSeries(
    breakpoints,
    [3.5, 4.25, 5.0, 4.75],
    "gas_price";
    units = "USD/MMBtu",
    component_field = "fuel_cost",
    # Whether a curve is expanded to a full series or collapsed to one scalar is
    # your application's policy, and rides here where the store never reads it.
    application_data = """{"as_time_series":false,"force_scalar_mode":"midpoint"}""",
)
id = add_time_series!(store, 7, "ThermalStandard", Component, prices)

got = read_by_id(store, id)              # a PersistentTimeSeries; `timestamps` are the breakpoints
get_metadata_by_id(store, id).time_series_type <: PersistentTimeSeries   # `<:`, not `==`

A metadata row's time_series_type is the full parameterized type (PersistentTimeSeries{Float64, 1}), so ask which kind a row is with <:. Every type-taking call — a time_series_type filter, has_time_series, either reader — accepts the bare spelling and the parameterized one alike.

Read a window

read_by_ids' time_range slices on the step function's own terms: the result begins at the breakpoint in force at the start, so it always defines a value at the start of the window you asked for.

sliced = only(
    read_by_ids(store, [id]; time_range = (DateTime(2024, 4, 10), DateTime(2024, 9, 1))),
)
sliced.timestamps   # [2024-04-01T00:00:00, 2024-07-01T00:00:00] — April, not the first one inside
sliced.data         # [4.25, 5.0]

Past the last breakpoint the last value holds forever, so a window opening after the end comes back with that one row. Before the first breakpoint a step function is undefined: a non-empty window starting there throws InvalidParameterError rather than clamping. (A zero-width range selects nothing, here as for every type.) read_by_id's start_time + len window is checked rather than sliced, so it must name one of the breakpoints; reach for the time_range form when the instant is arbitrary.

Sweep step functions in the simulation loop

A StaticReader filtered to the type is the per-timestamp path, and it is the one place the one-timeline-per-reader rule bends: a step function has a value at every instant from its own first breakpoint on, so the columns need not share a breakpoint vector. Per-fuel curves whose breakpoints do not line up still build one reader.

reader = build_static_reader(
    store;
    time_series_type = PersistentTimeSeries,   # no resolution — passing one throws
    component_field = "fuel_cost",
)
grid = static_grid(reader)                 # grid.resolution === nothing: no constant step
for at in static_timestamps(reader)        # the sorted union of every column's breakpoints
    static_read!(reader, at)
    for (gi, g) in enumerate(static_groups(reader))
        vals = static_values(reader, gi)   # the value in force at `at`; column j ↔ g.ids[j]
    end
end

static_timestamps is the union of the columns' breakpoints, so a position on it is not a storage row for any one column — each column independently reports the value in force there. There is still no presence mask: reading at an instant before some column's first breakpoint throws InvalidParameterError naming that column's association id. Either filter the reader down to columns that start early enough, or begin the sweep at the latest first breakpoint among them.

The sweep need not follow that union axis at all. static_read! accepts any instant every column defines a value at, so driving this reader at your SingleTimeSeries grid's timestamps — the simulation's own clock — works and is usually what an application wants:

for at in static_timestamps(load_reader)   # the hourly grid the simulation runs on
    static_read!(load_reader, at)
    static_read!(reader, at)               # each fuel price, held forward to this hour
end

These rows do not travel in an OpenAPI document

PersistentTimeSeries is an infrastore-local extension, and the vendored wire contract is a oneOf over six canonical Sienna types with no schema for a seventh. So export_time_series_associations_openapi omits persistent rows — a mixed store still exports its six-type rows — a filter naming the type throws InvalidParameterError rather than answering with an empty array, and an import refuses a document that carries one. The series themselves are unaffected: they live in the artifact, which holds them in full. Ask the catalog what a document leaves behind:

left_behind = list_metadata(store; time_series_type = PersistentTimeSeries)

Store-Wide Operations

counts = get_counts(store)        # TimeSeriesCounts: components_with_time_series, static_time_series, forecasts
nerr   = verify_integrity(store)  # 0 == every referenced array and time axis matches its hash
report = compact!(store)          # CompactionReport; rewrites the .h5 from the live set, so a
                                  # delete actually shrinks the file. Nothing else may have the
                                  # store open while it runs.

Associations

Two catalog tables record relationships between entities the store does not otherwise model, wholly independently of time series: which supplemental attributes are attached to which components, and directed parent/child edges between components. Removing a time series never touches either, and vice versa — see Associations Between Entities.

Filter keywords are all optional and ANDed; passing none matches everything.

add_supplemental_attribute_association!(
    store, SupplementalAttributeAssociation(42, "Generator", 100, "GeographicInfo"))

# Bulk add is one all-or-nothing transaction.
add_supplemental_attribute_associations!(store, [
    SupplementalAttributeAssociation(43, "Generator", 100, "GeographicInfo"),
    SupplementalAttributeAssociation(43, "Generator", 101, "Outage"),
])

# Queries run in both directions, returning distinct ids in ascending order.
list_supplemental_attribute_ids(store; component_id=43)      # [100, 101]
list_components_with_attributes(store; attribute_id=100)     # [42, 43]
has_supplemental_attribute_association(store; component_id=42, attribute_id=100)  # true

# `*_types` filters take CONCRETE type names, so expand an abstract type yourself —
# `get_all_subtype_names` in InfrastructureSystems.jl is the usual source. An empty
# vector is a deliberate "none of these" and matches nothing.
list_supplemental_attribute_ids(store; component_id=43, attribute_types=["Outage"])  # [101]

count_supplemental_attributes(store)         # 2, distinct attributes
count_components_with_attributes(store)      # 2, distinct components
supplemental_attribute_counts_by_type(store)
# [SupplementalAttributeTypeCount("GeographicInfo", 2), SupplementalAttributeTypeCount("Outage", 1)]
supplemental_attribute_summary(store)
# [SupplementalAttributeSummaryRow("Generator", "GeographicInfo", 2), ...]

Identity is the (component_id, attribute_id) pair. The type names ride along for filtering and are not part of it, so re-attaching the same pair under different type names is still a duplicate:

try
    add_supplemental_attribute_association!(
        store, SupplementalAttributeAssociation(42, "Load", 100, "Outage"))
catch e
    e isa InfraStore.DuplicateAssociationError || rethrow()
    @info e.msg   # attribute 100 is already attached to component 42
end

# Removal returns a count. Matching nothing returns 0 rather than throwing, so assert on
# the count yourself if you expected a hit.
remove_supplemental_attribute_associations!(store; component_id=43)   # 2

Parent/child edges work the same way, except that identity is the ordered pair — the reverse of an edge is a different edge — and both endpoints are always components:

add_parent_child_association!(store, ParentChildAssociation(42, "Generator", 7, "Bus"))
add_parent_child_associations!(store, [ParentChildAssociation(43, "Generator", 7, "Bus")])

list_children(store; parent_id=42)      # [7]
list_parents(store; child_id=7)         # [42, 43]
count_parent_child_associations(store)  # 2

# Renumbering a component rewrites both ends of every edge.
replace_parent_child_component_id!(store, 42, 99)   # 1
list_parents(store; child_id=7)                     # [43, 99]

Neither table is reachable over gRPC or the infrastore CLI.

Persist to Disk

flush!(store)   # sync HDF5 + SQLite; afterwards system.h5 + system.h5.sqlite can be copied

Keep the .h5 and .h5.sqlite files together.

To change a store you did not build in this process, open a copy: open_store defaults to read-write, and HDF5 has no journal, so an interrupted in-place write is unrecoverable.

store = open_copy(src, joinpath(scratch, "time_series.h5"))   # src is never opened for writing
...
persist!(store, src)                                          # one atomic rename replaces it

open_store(path; read_only=true) is the right call when nothing will be written.

Where the Catalog Lives

By default the catalog is system.h5.sqlite, and every commit is durable. Passing catalog=:memory keeps it in RAM instead, so it reaches disk only via persist!:

# Build in a scratch directory; nothing is durable until the explicit save.
store = Store(; in_memory=false, path=joinpath(scratch, "time_series.h5"), catalog=:memory)
add_time_series!(store, 42, "Generator", Component, ts)
persist!(store, destination)     # writes both halves as a matched pair
persist_catalog!(store)          # or: land only the .sqlite half beside the arrays already at path

Arrays still stream to the HDF5 file, so this does not require the data to fit in memory. It suits building a store beside volatile in-process state — a crash loses that state anyway, so journaling the scratch catalog buys nothing. catalog_mode(store) reports which mode a store is in.

open_store(path; catalog=:memory) loads an existing catalog into RAM the same way. Note that the HDF5 half is still opened in place, so mutations land in the original file; open a copy if you mean to leave the source untouched until an explicit save.

persist! stages both halves and renames them into place, and stamps the pair so that a save interrupted between the two renames is caught on the next open rather than read as a valid store. It does replace the destination, though, so a failed save may have destroyed what was there — recover by calling persist! again on the still-live store rather than assuming the target survived.

Error Handling

Errors subtype InfraStore.TimeSeriesException. The exception types are not exported, so reference them module-qualified. Catch broadly or narrowly:

try
    add_time_series!(store, 42, "Generator", Component, ts)
catch e
    if e isa InfraStore.DuplicateTimeSeriesError
        @warn "already present"
    else
        rethrow()
    end
end

The available types are InfraStore.NotFoundError, InfraStore.DuplicateTimeSeriesError, InfraStore.DuplicateAssociationError, InfraStore.InvalidParameterError, InfraStore.IntegrityError, InfraStore.ReadOnlyStoreError, InfraStore.IOError, InfraStore.StoreExistsError (creating over an existing artifact), InfraStore.MismatchedArtifactError (the .h5 and .sqlite halves came from two saves), InfraStore.IncompatibleFormatError (the on-disk store was written by an incompatible data format version), and InfraStore.GenericError (which carries the raw FFI status code). See the reference for the full table.

InfrastructureSystems.jl Integration Notes

Embedding in a Parent Package is the language-neutral version of this section — the store lifecycle, id mapping, and lookup semantics a package like InfrastructureSystems.jl has to honor. The Julia-specific points:

• Owners are integer component identifiers (Int64), matching InfrastructureSystems.jl component/attribute IDs.
• OwnerCategory distinguishes Component from SupplementalAttribute and is part of the owner identity: the owner is the (owner_id, owner_category) pair, so a component and a supplemental attribute may share a numeric id and stay distinct. Owner-scoped calls take the category alongside the id.
• The attribute-based existence probes (has_time_series, has_any_time_series) plus get_metadata_by_id and get_array_by_hash let an InfrastructureSystems.jl-side store keep its own object model — holding only the catalog id — and reach the array layer directly.
• For the simulation read pattern — iterate every component's value at each timestamp, reading a forecast shared across components only once — use the readers (Per-Timestamp Reads). The ForecastEntry.slot / forecast_num_slots surface lets the wrapping store dedup its own per-component work, mirroring the store's one-read-per-shared-array behavior.

See Language Bindings for how this maps onto the FFI.

Diagnostics and tracing

The store emits structured tracing spans for every significant operation. To see them, initialize a subscriber before your first store call.

Via environment variable — set RUST_LOG before loading the package. The module's __init__ hook calls init_logging("") automatically, which reads RUST_LOG if set:

# shell
export RUST_LOG=infrastore_core=debug
julia --project=. myscript.jl

Programmatically — call init_logging with a filter directive string:

using InfraStore

init_logging("infrastore_core=debug")

store = Store(in_memory=true)
add_time_series!(store, ...)   # spans appear on stderr

init_logging is a no-op if a subscriber is already registered (including the automatic one from RUST_LOG). The filter syntax is the same as RUST_LOG: comma-separated target=level pairs, or a bare level such as "debug" to match everything. Useful targets: