sup3r.utilities.pytest.helpers.DummySampler

sup3r.utilities.pytest.helpers.DummySampler#

class DummySampler(sample_shape, data_shape, features, batch_size, feature_sets=None, chunk_shape=None)[source]#

Bases: Sampler

Dummy container with random data.

Parameters:

data (Union[Sup3rX, Sup3rDataset],) – Object with data that will be sampled from. Usually the .data attribute of various Container objects. i.e. Loader, Rasterizer, Deriver, as long as the spatial dimensions are not flattened.
sample_shape (tuple) – Size of arrays to sample from the contained data.
batch_size (int) – Number of samples to get to build a single batch. A sample of (sample_shape[0], sample_shape[1], batch_size * sample_shape[2]) is first selected from underlying dataset and then reshaped into (batch_size, *sample_shape) to get a single batch. This is more efficient than getting N = batch_size samples and then stacking.
feature_sets (Optional[dict]) – Optional dictionary describing how the full set of features is split between lr_features, hr_exo_features, and hr_out_features.

lr_featureslist | tuple
List of feature names or patt*erns to use as low-resolution model inputs. If no entry is provided then all available features from the data will be used.

hr_out_featureslist | tuple
List of feature names or patt*erns that should be output by the generative model and available as ground truth targets. If no entry is provided then all features in lr_features will be used.

hr_exo_featureslist | tuple
List of feature names or patt*erns that should be available as high-resolution model inputs (like topography or observations) or for bespoke loss functions. Features used as inputs are injected into the model mid-network to condition output on high-resolution information. The model configuration should have the appropriate layers to use these features. e.g. Sup3rConcat for topography injection, Sup3rObsModel or Sup3rCrossAttention for obs injection. If no entry is provided then hr_exo_features will be empty.

*To include sparse features as inputs or targets the features must have an “_obs” suffix.
proxy_obs_kwargs (dict | None) – Optional dictionary of keyword arguments to pass to the proxy observation generator. This is only used when training with proxy observations. Top-level keys (onshore_obs_frac, offshore_obs_frac, perturbation_scale) apply to all obs features as defaults. A source-feature-named sub-dict (keyed by the gridded feature name, e.g. u_100m for u_100m_obs) overrides any of those keys for that specific feature:
```
proxy_obs_kwargs = {
    'onshore_obs_frac': {'spatial': [0.3, 0.7], 'temporal': 1},
    'perturbation_scale': 0.01,
    'u_100m': {
        'onshore_obs_frac': {'spatial': 0.9},
        'perturbation_scale': 0.05,
    },
}
```
perturbation_scalefloat
If non-zero, gaussian noise scaled by this value times the per-feature batch standard deviation is added to proxy obs.

onshore_obs_fracdict
Fraction of onshore observations per batch. Keys are ‘spatial’ and ‘temporal’. Each value is a float (fixed fraction) or a [min, max] list to sample uniformly per batch.

offshore_obs_fracdict
Same as onshore_obs_frac but applied where topography <= 0. Ignored when topography is not a source feature.
mode (str) – Mode for sampling data. Options are ‘lazy’ or ‘eager’. ‘eager’ mode pre-loads all data into memory as numpy arrays for faster access. ‘lazy’ mode samples directly from the underlying data object, which could be backed by dask arrays or on-disk netCDF files.

Methods

`check_feature_consistency`()	Check that the feature sets are consistent with each other and the obs features are configured correctly.
`check_proxy_obs_consistency`()	Check that the obs features are configured correctly for proxy observations.
`derive`(feature[, strict])	Resolve feature name to a feature in the underlying data.
`get_sample_index`([n_obs])	Randomly gets spatiotemporal sample index.
`post_init_log`([args_dict])	Log additional arguments after initialization.
`preflight`()	Perform shape and feature checks.
`wrap`(data)	Return a `Sup3rDataset` object or tuple of such.

Attributes

`timer`
`data`	Return underlying data.
`hr_exo_features`	Get a list of exogenous high-resolution features that are only used for training e.g., mid-network high-res topo injection.
`hr_features`	List of feature names or patt*erns that the model is shown at high-resolution.
`hr_features_ind`	Get the high-resolution feature channel indices that should be included for loss calculations.
`hr_out_features`	List of feature names or patt*erns that should be output by the generative model.
`hr_sample_features`	List of feature names used in the sample index for the high-resolution training data.
`hr_sample_shape`	Shape of the data sample to select when __next__() is called.
`hr_source_features`	List of feature names or patt*erns that should be available natively as high-resolution.
`lr_features`	List of feature names or patt*erns to use as low-resolution model inputs.
`lr_features_ind`	Get the low-resolution feature channel indices that should be included for training.
`obs_features`	List of feature names or patt*erns that should be treated as observations.
`obs_features_ind`	Get the source feature indices for each obs feature's corresponding base feature (e.g. `u_100m` for `u_100m_obs`).
`real_obs_features`	Obs features that actually exist in the data.
`sample_shape`	Shape of the data sample to select when `__next__()` is called.
`shape`	Get shape of underlying data.
`use_proxy_obs`	Whether to use proxy observations.

check_feature_consistency()#: Check that the feature sets are consistent with each other and the obs features are configured correctly.

check_proxy_obs_consistency()#: Check that the obs features are configured correctly for proxy observations.

property data#

Return underlying data.

Returns:: Sup3rDataset

sup3r.utilities.pytest.helpers.DummySampler

Contents

sup3r.utilities.pytest.helpers.DummySampler#