6. Check the model¶

Before you trust a suggestion, check how well the model predicts results it has not seen and whether its uncertainty is honest. This page opens the diagnostic plots and says what each shows and how ALchemist computes the numbers.

The parity plot of the first BoTorch model, trained on the ten simulated design points. Each point is one result, predicted with that result held out.

1. Open the plots¶

1. Train a model; see Train a model.
2. In the Model tab, choose Show Visualizations. The centre column becomes Model Visualizations, opened on the parity plot.
3. Choose a plot: Plot Parity, Plot Metrics, Plot Q-Q, Plot Calibration or Plot Contour.
4. Close the panel with the × at its top right.

Open plots redraw when the model retrains, for example after Add Point... with the retrain box ticked. The panel closes when you choose Clear, New Session or Join; choose Show Visualizations again in the new session.

What each plot shows¶

Plot Parity, Plot Q-Q and Plot Calibration use the same set of cross-validated predictions: the data are split into five folds, and each fold is predicted by a model fitted without it. So every result has one prediction, and one uncertainty, from a model that did not see it.

A fixed real variable also has a number box below its slider. Type a value and press Enter (or click away) to set it; values outside the variable's range are clamped to it, and Escape puts back the last value. The arrow keys step the box, or the slider once it has focus, by 1/100 of the range, and the plot updates when you stop pressing.

Scale sets the colour bar. Fixed, the default, spans the observed results, from the smallest to the largest, and stays put while you move the fixed values, so a colour means the same value on every slice. Predictions outside that range take the end colour; hovering still shows their value. Per slice stretches the colour bar over each slice's own predictions: more detail within one slice, but two slices can no longer be compared by colour. The app remembers the choice in this browser. When the results have no spread, as with a single result, only Per slice is available. A multi-objective session takes the range from its first objective.

A standardized residual is the prediction error divided by the predicted uncertainty, \(z = (y - \hat\mu)/\hat\sigma\). If the model is right, the \(z\) values look like draws from a standard normal distribution: Mean(z) near 0 and Std(z) near 1. Use Calibrated Results scales every uncertainty by the factor that makes Std(z) exactly 1 on these data, so the calibrated Q-Q plot passes that test by construction. The factor changes the uncertainties shown, not the suggestions.

How to read each plot, and what to try when one looks wrong, is in Reading the diagnostics.

The metrics¶

Trained Model Info and the parity title show RMSE, MAE and R². Plot Metrics also offers MAPE. With \(e_i = y_i - \hat\mu_i\), the error of the cross-validated prediction for result \(i\) of \(n\):

RMSE, MAE and R² come from sklearn.metrics. The folds are shuffled with a fixed seed. A model trained on fewer than five results has no cross-validation and no metrics. For a given size, Plot Metrics pools the BoTorch folds into one value but averages the scikit-learn folds' values, so its last point need not equal the numbers in Trained Model Info.

From Python or REST¶

import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np
from alchemist_core import OptimizationSession

rng = np.random.default_rng(5)
session = OptimizationSession()
session.add_variable("temperature", "real", bounds=(200.0, 300.0), unit="°C")
session.add_variable("pressure", "real", bounds=(20.0, 80.0), unit="bar")
for p in session.generate_initial_design("lhs", n_points=12, random_seed=5):
    t = (p["temperature"] - 260) / 50
    q = (p["pressure"] - 55) / 30
    session.add_experiment(p, output=10 - 6 * t**2 - 3 * q**2 + rng.normal(0, 0.2))
session.train_model(backend="botorch")

fig, axes = plt.subplots(1, 2, figsize=(11, 4.5))
session.plot_parity(ax=axes[0])
session.plot_contour("temperature", "pressure", ax=axes[1])
fig.tight_layout()
fig.savefig("check.png", dpi=120)
session.plot_qq().savefig("qq.png")
session.plot_calibration().savefig("calibration.png")
session.plot_metrics(metric="rmse").savefig("metrics.png")

cv = session.model.cv_cached_results
z = (np.asarray(cv["y_true"]) - np.asarray(cv["y_pred"])) / np.asarray(cv["y_std"])
print(f"Mean(z) = {z.mean():.2f}, Std(z) = {z.std(ddof=1):.2f}, N = {len(z)}")

GET /api/v1/sessions/{session_id}/visualizations/parity