sup3r.preprocessing.samplers.dual.DualSampler

class DualSampler(data: Sup3rDataset, sample_shape: tuple | None = None, batch_size: int = 16, s_enhance: int = 1, t_enhance: int = 1, feature_sets: dict | None = None, proxy_obs_kwargs: dict | None = None, mode: str = 'lazy')

Bases: Sampler

Sampler for sampling from paired (or dual) datasets. Pairs consist of low and high resolution data, which are contained by a Sup3rDataset. This can also include extra observation data on the same grid as the high-resolution data which has NaNs at points where observation data doesn’t exist. This will be used in an additional content loss term.

Parameters:

• data (Sup3rDataset) – A Sup3rDataset instance with low-res and high-res data members.

• sample_shape (tuple) – Size of arrays to sample from the high-res data. The sample shape for the low-res sampler will be determined from the enhancement factors.

• s_enhance (int) – Spatial enhancement factor

• t_enhance (int) – Temporal enhancement factor

• feature_sets (Optional[dict]) – See Sampler for full documentation.

• proxy_obs_kwargs (dict | None) – See Sampler for full documentation.

• mode (str) – Mode for sampling data. Options are ‘lazy’ or ‘eager’. ‘eager’ mode pre-loads all data into memory as numpy arrays for faster access. ‘lazy’ mode samples directly from the underlying data object, which could be backed by dask arrays or on-disk netCDF files.

Methods

check_feature_consistency()	Make sure features are consistent with the data and with each other.
check_proxy_obs_consistency()	Check that the obs features are configured correctly for proxy observations.
check_shape_consistency()	Make sure container shapes are compatible with enhancement factors.
derive(feature[, strict])	Resolve feature name to a feature in the underlying data.
get_sample_index([n_obs])	Get paired sample index, consisting of index for the low res sample and the index for the high res sample with the same spatiotemporal extent.
post_init_log([args_dict])	Log additional arguments after initialization.
preflight()	Perform shape and feature checks.
wrap(data)	Return a Sup3rDataset object or tuple of such.

Attributes

timer
data	Return underlying data.
hr_exo_features	Get a list of exogenous high-resolution features that are only used for training e.g., mid-network high-res topo injection.
hr_features	List of feature names or patt*erns that the model is shown at high-resolution.
hr_features_ind	Get the high-resolution feature channel indices that should be included for loss calculations.
hr_out_features	List of feature names or patt*erns that should be output by the generative model.
hr_sample_features	List of feature names used in the sample index for the high-resolution training data.
hr_sample_shape	Shape of the data sample to select when __next__() is called.
hr_source_features	Features available natively at high-resolution.
lr_features	List of feature names or patt*erns to use as low-resolution model inputs.
lr_features_ind	Get the low-resolution feature channel indices that should be included for training.
obs_features	List of feature names or patt*erns that should be treated as observations.
obs_features_ind	Get the source feature indices for each obs feature's corresponding base feature (e.g. u_100m for u_100m_obs).
real_obs_features	Obs features that actually exist in the data.
sample_shape	Shape of the data sample to select when __next__() is called.
shape	Get shape of underlying data.
use_proxy_obs	Whether to use proxy observations.

property data

Return underlying data.

Returns:

Sup3rDataset

See also

wrap()

property sample_shape: tuple

Shape of the data sample to select when __next__() is called.

property hr_source_features

Features available natively at high-resolution.

check_feature_consistency()

Make sure features are consistent with the data and with each other.

check_shape_consistency()

Make sure container shapes are compatible with enhancement factors.

get_sample_index(n_obs=None)

Get paired sample index, consisting of index for the low res sample and the index for the high res sample with the same spatiotemporal extent. Optionally includes an extra high res index if the sample data includes observation data.

check_proxy_obs_consistency()

Check that the obs features are configured correctly for proxy observations.

derive(feature, strict=True)

Resolve feature name to a feature in the underlying data. This is used for handling feature aliases and for deriving new features from existing ones.

property hr_exo_features

Get a list of exogenous high-resolution features that are only used for training e.g., mid-network high-res topo injection. These must come at the end of the high-res feature set. These can also be input to the model as low-res features.

property hr_features

List of feature names or patt*erns that the model is shown at high-resolution. This does not include features that are only shown to the model after coarsening. Thus, this includes hr_out_features and and hr_exo_features but not lr_features.

property hr_features_ind

Get the high-resolution feature channel indices that should be included for loss calculations. This includes hr_out_features and hr_exo_features, Any high-resolution features that are only included in the data handler to be coarsened for the low-res input are removed.

property hr_out_features

List of feature names or patt*erns that should be output by the generative model. If no entry is provided then all features in hr_features will be used.

property hr_sample_features

List of feature names used in the sample index for the high-resolution training data. When using proxy obs, obs features that are not present in the raw data are excluded from sampling (they will be generated as proxy obs). Obs features that ARE in the data are included so they get sampled.

property hr_sample_shape: tuple

Shape of the data sample to select when __next__() is called. Same as sample_shape

property lr_features

List of feature names or patt*erns to use as low-resolution model inputs. If no entry is provided then all available features from the data will be used.

property lr_features_ind

Get the low-resolution feature channel indices that should be included for training. This includes lr_features.

property obs_features

List of feature names or patt*erns that should be treated as observations. These features will be included in the high-res data but not the low-res data and won’t necessarily be expected to be output by the generative model. These are different from other hr_exo_features in that they are intended to be used as observation features with NaN values where observations are not available.

property obs_features_ind

Get the source feature indices for each obs feature’s corresponding base feature (e.g. u_100m for u_100m_obs). Used to extract gridded truth for proxy obs generation.

Returns:

list[int] – Indices into hr_sample_features for each obs base feature.

post_init_log(args_dict=None)

Log additional arguments after initialization.

preflight()

Perform shape and feature checks.

property real_obs_features

Obs features that actually exist in the data. In contrast to proxy obs, which are generated by masking the gridded data, these features are present in the raw data and are sampled directly. These are a subset of obs_features.

property shape

Get shape of underlying data.

property use_proxy_obs

Whether to use proxy observations. When True, proxy observation features are generated by masking the corresponding gridded ground truth data and are appended to the samples. The obs features are specified by the obs_features argument and should have a corresponding source feature in the data features that is used for sampling. For example, an obs feature named temperature_obs would be generated from the gridded ground truth feature named temperature.

wrap(data)

Return a Sup3rDataset object or tuple of such. This is a tuple when the .data attribute belongs to a Collection object like BatchHandler. Otherwise this is Sup3rDataset object, which is either a wrapped 3-tuple, 2-tuple, or 1-tuple (e.g. len(data) == 3, len(data) == 2 or len(data) == 1). This is a 3-tuple when .data belongs to a container object like DualSamplerWithObs, a 2-tuple when .data belongs to a dual container object like DualSampler, and a 1-tuple otherwise.