Model Calibration

DEMOS includes a calibration system that adjusts the internal parameters of its estimated models (logit, multinomial logit, and regression) so that the simulation output matches observed real-world data. Without calibration, models will produce output strictly derived from their original estimation (fitting) data, which may not match conditions in the target geography or time period. Calibration shifts model predictions closer to observed aggregate statistics — such as the total number of births, deaths, or employed persons in a given year — while preserving the underlying behavioral relationships captured during model estimation.

This page explains:

  1. How Calibration Works

  2. Modules That Support Calibration

  3. Preparing Observed Data Files

  4. Configuring Calibration in the TOML File

  5. Modules with Simultaneous Calibration: Employment and Household Reorganization

How Calibration Works

DEMOS uses an iterative adjustment procedure based on the Root Mean Square Error (RMSE) between the model’s predictions and a target value drawn from an observed data table. At each iteration, a single parameter (the model’s intercept) is nudged so that the aggregate predicted count (or share) moves toward the observed value. This continues until either the error falls below a user-specified tolerance, or the maximum number of iterations is reached.

The update rule

At each iteration, the intercept \(\beta_0\) is updated as:

\[\beta_0 \leftarrow \beta_0 + \ln\!\left(\frac{\text{target}}{\sum \hat{y}}\right)\]

where \(\sum \hat{y}\) is the sum of the current model predictions and \(\text{target}\) is the observed value for the simulation year. This log-ratio update is a standard technique for intercept adjustment in discrete-choice and regression models: shifting the intercept by this amount is equivalent to re-scaling the aggregate predicted probability so it matches the aggregate observed rate.

Absolute vs relative tolerance

The calibration procedure supports two ways of measuring how close “close enough” is:

  • Absolute (tolerance_type = "absolute"): The error is computed as the RMSE between the total predicted count and the observed count. The tolerance value is therefore expressed in the same units as the counts (e.g., number of people). Use this when your observed data contains raw counts.

  • Relative (tolerance_type = "relative"): The error is computed as the RMSE between the predicted share (predicted count divided by the eligible population size) and the observed share. The tolerance value is a fraction (e.g., 0.01 means 1%). Use this when your observed data contains rates or proportions.

Modules That Support Calibration

The following modules support calibration via the standard calibration_procedure configuration block:

Module

Config key

Calibrated model

What is calibrated

Mortality (fatality)

mortality_module_config

mortality

Total deaths per year

Birth (birth)

birth_module_config

birth

Total births per year

Income (income)

income_module_config

income_nworkers

Aggregate household income per year

The following modules have a different calibration interface (see below):

Module

Config key

Note

Employment (employment)

employment_module_config

Simultaneous calibration across two sub-models

Household Reorganization (household_reorg)

hh_reorg_module_config

Simultaneous calibration across three sub-models

The Kids Moving module (kids_moving) also performs hardcoded calibration internally; its parameters are controlled directly under kids_moving_module_config (see the reference configuration).

Preparing Observed Data Files

Each module that uses the standard calibration procedure requires a CSV file containing observed aggregate statistics, one row per year. The file must have a column named year (used as the row index) and either a count or share column depending on your chosen tolerance_type.

File format for absolute calibration (tolerance_type = "absolute")

The file must contain a column named count with the observed total number of events (deaths, births, etc.) for each year.

year,count
2010,1452
2011,1489
2012,1503
...

File format for relative calibration (tolerance_type = "relative")

The file must contain a column named share with the observed rate (a number between 0 and 1) for each year.

year,share
2010,0.0092
2011,0.0094
2012,0.0096
...

Important: The index column must be named year. Make sure this column contains integer years matching the years in your simulation run (between base_year and forecast_year). If a year is missing from the file, the simulation will raise an error when it tries to look up the target value.

The file can be stored anywhere accessible from the configuration file. A common convention is to place these files in a subdirectory of your data folder, e.g.:

data/
  my_region/
    observed_calibration_values/
      mortalities_over_time.csv
      births_over_time.csv

Configuring Calibration in the TOML File

Calibration is configured inside the relevant module’s configuration block in your demos_config.toml file. Each calibratable module has a calibration_procedure sub-block.

Full example: mortality module with absolute calibration

[mortality_module_config.calibration_procedure]
procedure_type = "rmse_error"
tolerance_type = "absolute"
tolerance = 30
max_iter = 500

[mortality_module_config.calibration_procedure.observed_values_table]
file_type = "csv"
table_name = "observed_fatalities_data"
filepath = "../data/my_region/observed_calibration_values/mortalities_over_time.csv"
index_col = "year"

Full example: birth module with absolute calibration

[birth_module_config.calibration_procedure]
procedure_type = "rmse_error"
tolerance_type = "absolute"
tolerance = 60
max_iter = 1000

[birth_module_config.calibration_procedure.observed_values_table]
file_type = "csv"
table_name = "observed_births_data"
filepath = "../data/my_region/observed_calibration_values/births_over_time.csv"
index_col = "year"

Parameter reference

Parameter

Type

Required

Description

procedure_type

string

Yes

Must be "rmse_error". This is the only available calibration procedure.

tolerance_type

string

No

"absolute" (default) or "relative". Controls how the error is measured. See above.

tolerance

float

Yes

The error threshold below which calibration stops. Units depend on tolerance_type.

max_iter

int

No

Maximum number of calibration iterations (default: 20). Calibration stops here even if the tolerance has not been reached.

observed_values_table

table block

Yes

Defines the CSV file with observed values. Follows the same format as any [[tables]] entry.

The observed_values_table sub-block

The observed_values_table block defines the file containing observed data. It uses the same fields as a regular data source entry:

Field

Description

file_type

Must be "csv" for observed calibration data.

table_name

An internal name for this table in DEMOS memory. Must be unique across all tables. Pick a descriptive name like "observed_fatalities_data".

filepath

Path to the CSV file, relative to your configuration file.

index_col

Must be "year".

Disabling calibration

To disable calibration for a module, simply remove or comment out the entire calibration_procedure block for that module:

# [mortality_module_config.calibration_procedure]
# procedure_type = "rmse_error"
# ...

If the block is absent, DEMOS will run the model using the original estimated parameters with no adjustment.

Modules with Simultaneous Calibration: Employment and Household Reorganization

Two modules — Employment and Household Reorganization — use a more complex calibration strategy called simultaneous calibration. Instead of calibrating a single model against a single target, these modules adjust multiple models at the same time so that their joint output matches aggregate observed counts.

This approach is necessary because the outcome of interest (e.g., total number of employed persons, total number of married persons) depends on the combined output of several sub-models that each influence the same population. Calibrating them independently would cause the adjustments to conflict with each other.

The simultaneous calibration algorithm

Simultaneous calibration uses a momentum-based gradient descent approach. At each iteration:

  1. All sub-models are run on their respective data to produce predictions.

  2. The combined predicted aggregate is compared to the observed aggregate.

  3. An update step \(g\) is computed using a momentum term to smooth out oscillations:

\[g_t = \alpha_t \left( \mu \cdot g_{t-1} + (1-\mu) \cdot \delta_t \right)\]

where:

  • \(\alpha_t\) is a decaying learning rate that starts at learning_rate and decreases toward zero as the iteration count approaches max_iter

  • \(\mu\) is the momentum_weight (a value between 0 and 1 that controls how much the previous gradient update is retained)

  • \(\delta_t = \ln(\text{target} / \hat{y}_t)\) is the log-ratio between the target and current prediction

  1. The intercepts of each sub-model are adjusted by \(g_t\).

  2. Steps 1–4 repeat until the absolute error falls below tolerance or max_iter is reached.

The decaying learning rate helps the algorithm converge: large adjustments happen early, and corrections become finer as the prediction approaches the target.

Employment module

The Employment module calibrates two sub-models simultaneously:

  • enter_labor_force: A logit model predicting which unemployed persons (age ≥ 18) transition into employment.

  • exit_labor_force: A logit model predicting which employed persons (age ≥ 18) exit the workforce.

The calibration target is the total number of workers (employed persons) predicted after running both models, compared against an observed count loaded from the observed_employment table.

Important: The table name observed_employment is hard-coded. You must load this table via the [[tables]] section of your configuration with exactly that name.

Required observed data table

The observed_employment table must be a CSV with a year index and a count column containing the total observed employed persons per year:

year,count
2010,18423
2011,18751
...

Add it to your configuration under [[tables]]:

[[tables]]
file_type = "csv"
table_name = "observed_employment"
filepath = "../data/my_region/observed_calibration_values/employment_obs.csv"
index_col = "year"

Calibration configuration block

[employment_module_config.simultaneous_calibration_config]
tolerance = 100
max_iter = 20
learning_rate = 2.0
momentum_weight = 0.3

Parameter reference

Parameter

Type

Required

Default

Description

tolerance

float

Yes

—

Absolute error threshold (in number of workers) below which calibration stops.

max_iter

int

No

20

Maximum number of calibration iterations.

learning_rate

float

No

2.5

Controls the step size at the first iteration. Decays linearly toward zero.

momentum_weight

float

No

0.3

Weight applied to the previous gradient update. Values closer to 1 introduce more smoothing.

Disabling employment calibration

To disable simultaneous calibration for employment, comment out or remove the block:

# [employment_module_config.simultaneous_calibration_config]
# ...

When the block is absent, DEMOS checks whether individual model calibration procedures are defined under employment_module_config.enter_model_calibration_procedure and employment_module_config.exit_model_calibration_procedure. If neither is present, no calibration is performed. Note that simultaneous calibration and per-model calibration are mutually exclusive: defining both will raise a validation error.

Household Reorganization module

The Household Reorganization module calibrates three sub-models simultaneously:

  • marriage: A multinomial logit model predicting transitions for unmarried, non-cohabitating persons — options are: stay single, start cohabitating, or get married.

  • cohabitation: A multinomial logit model predicting transitions for cohabitating persons — options are: stay cohabitating, break up, or get married.

  • divorce: A binary logit model predicting which married households undergo a divorce.

The combined output of these three models determines the predicted number of married and divorced persons after the step. Calibration targets both of these counts simultaneously, weighting each proportionally to its magnitude in the observed data.

Important: The table name observed_marrital_data is hard-coded. You must load this table via the [[tables]] section with exactly that name.

Required observed data table

The observed_marrital_data table must be a CSV with a year index and columns MAR and count, containing observed counts for each marital status category per year. The values used during calibration are MAR == 1 (married) and MAR == 3 (divorced/separated):

year,MAR,count
2010,1,14200
2010,3,2300
2011,1,14350
2011,3,2280
...

Add it to your configuration under [[tables]]:

[[tables]]
file_type = "csv"
table_name = "observed_marrital_data"
filepath = "../data/my_region/observed_calibration_values/marrital_status_over_time_obs.csv"
index_col = "year"

Calibration configuration block

[hh_reorg_module_config.simultaneous_calibration_config]
tolerance = 5000
max_iter = 100
learning_rate = 1.5
momentum_weight = 0.3

Parameter reference

Parameter

Type

Required

Default

Description

tolerance

float

Yes

—

Combined RMSE threshold (in number of persons) below which calibration stops. The error combines the residuals for the married and divorced counts.

max_iter

int

No

20

Maximum number of calibration iterations.

learning_rate

float

No

2.5

Controls the initial step size. Decays linearly toward zero.

momentum_weight

float

No

0.3

Weight applied to the previous gradient update.

How intercept updates are applied

Each of the three sub-models is adjusted through a different parameter:

  • divorce: The model’s primary intercept (fitted_parameters[0]) is shifted by the divorce gradient.

  • marriage: The married outcome coefficient in the model’s coefficient table is shifted by the marriage gradient.

  • cohabitation: The marriage outcome coefficient in the model’s coefficient table is shifted by the cohabitation gradient (cohabitation → marriage transitions are steered to match the marriage target).

The marriage and cohabitation gradients are weighted by the proportion of the married count relative to the total observed marital transitions. The divorce gradient is weighted by the proportion of the divorced count. This ensures that the calibration effort is distributed in proportion to the size of each phenomenon.

Disabling household reorganization calibration

To disable calibration for household reorganization, comment out or remove the block:

# [hh_reorg_module_config.simultaneous_calibration_config]
# ...

When absent, models run with their original estimated parameters.

Choosing Good Tolerance Values

Choosing a tolerance that is too tight (very small) can cause calibration to run for the full max_iter without converging, adding unnecessary compute time. Choosing one that is too loose means the model output may deviate substantially from observed data.

A practical starting point:

  • For absolute tolerances, set the tolerance to approximately 1–5% of the observed count for the most common year. For example, if you observe ~1,500 deaths per year, a tolerance of 30 (≈2%) is reasonable.

  • For relative tolerances, values between 0.005 and 0.02 (0.5%–2%) work well for most demographic rates.

  • For simultaneous calibration, the tolerance represents the absolute difference in person counts. Values of 100–5,000 are typical depending on the size of the synthetic population.

If calibration is not converging even after increasing max_iter, consider increasing the learning_rate slightly. If the algorithm oscillates (error fluctuates without declining), try increasing momentum_weight toward 0.5–0.7 or decreasing learning_rate.

Income Module: Inflation Adjustment

The income model estimates household income in nominal dollars for the year the model was originally estimated (the reference year). When running a multi-year simulation, predictions must be scaled to reflect price levels in each simulation year. DEMOS handles this through an inflation adjustment step that multiplies the predicted income by a cumulative CPI-based factor derived from a user-supplied table.

How it works

At each simulation year \(t\), DEMOS computes the cumulative inflation factor as:

\[\text{factor}(t) = \prod_{y=r+1}^{t} \big(1 + a_y\big)\]

where \(r\) is inflation_reference_year and \(a_y\) is the annual adjustment rate for year \(y\) from the inflation adjustment table. If \(t < r\) the factor is inverted (backward adjustment). If \(t = r\) the factor is 1 and income is left unchanged.

Inflation adjustment table format

The file must be a CSV indexed by year and must contain an adjustment column with decimal annual inflation rates:

year,adjustment
2011,0.030
2012,0.021
2013,0.015
2014,0.016
2015,0.001
2016,0.013
2017,0.021
2018,0.024
2019,0.018

  • year: Integer calendar year.

  • adjustment: Decimal inflation rate for that year (e.g., 0.030 = 3.0 %).
    Use U.S. Bureau of Labor Statistics CPI-U data or any equivalent price index.

The table must contain a row for every year in the interval (inflation_reference_year, forecast_year]. Missing years will cause a KeyError at runtime.

Loading the table

Add the table to the [[tables]] section of your configuration file:

[[tables]]
file_type = "csv"
table_name = "inflation_adjustment"
filepath = "../data/my_region/observed_calibration_values/inflation_adjustment.csv"
index_col = "year"
custom_dtype_casting = {"year" = "int", "adjustment" = "float"}

Configuring the income module

[income_module_config]
inflation_reference_year = 2010
inflation_adjustment_table = "inflation_adjustment"

Both fields must be set together; specifying only one will raise a validation error at startup.

To disable inflation adjustment, comment out or remove both lines:

# [income_module_config]
# inflation_reference_year = 2010
# inflation_adjustment_table = "inflation_adjustment"

Combining inflation adjustment with calibration

Inflation adjustment and calibration can be used at the same time. Calibration runs before the inflation adjustment, so the calibration target (count) should represent aggregate income expressed in reference-year nominal dollars (i.e., the same price level as the estimated model output).

[income_module_config]
inflation_reference_year = 2010
inflation_adjustment_table = "inflation_adjustment"

[income_module_config.calibration_procedure]
procedure_type = "rmse_error"
tolerance_type = "absolute"
tolerance = 5000
max_iter = 500
[income_module_config.calibration_procedure.observed_values_table]
file_type = "csv"
table_name = "observed_income_data"
filepath = "../data/my_region/observed_calibration_values/income_over_time_obs.csv"
index_col = "year"