sup3r.utilities.pytest.helpers.BatchHandlerTesterCC

class BatchHandlerTesterCC(train_containers, *, val_containers=None, sample_shape=None, batch_size=16, n_batches=64, s_enhance=1, t_enhance=1, means=None, stds=None, queue_cap=None, transform_kwargs=None, mode='lazy', feature_sets=None, verbose=False, **kwargs)

Bases: BatchHandler

Batch handler with sampler with running index record.

Parameters:

• train_containers (list[Container]) – List of objects with a .data attribute, which will be used to initialize Sampler objects and then used to initialize a batch queue of training data. The data can be a Sup3rX or Sup3rDataset object.

• val_containers (list[Container]) – List of objects with a .data attribute, which will be used to initialize Sampler objects and then used to initialize a batch queue of validation data. The data can be a Sup3rX or a Sup3rDataset object.

• batch_size (int) – Number of observations / samples in a batch

• n_batches (int) – Number of batches in an epoch, this sets the iteration limit for this object.

• s_enhance (int) – Integer factor by which the spatial axes is to be enhanced.

• t_enhance (int) – Integer factor by which the temporal axes is to be enhanced.

• means (str | dict | None) – Usually a file path for loading / saving results, or None for just calculating stats and not saving. Can also be a dict.

• stds (str | dict | None) – Usually a file path for loading / saving results, or None for just calculating stats and not saving. Can also be a dict.

• queue_cap (int) – Maximum number of batches the batch queue can store.

• transform_kwargs (Union[dict, None]) – Dictionary of kwargs to be passed to self.transform. This method performs smoothing / coarsening.

• mode (str) – Loading mode. Default is ‘lazy’, which only loads data into memory as batches are queued. ‘eager’ will load all data into memory right away.

• feature_sets (Optional[dict]) – See DualSampler for full documentation.

• kwargs (dict) – Additional keyword arguments for BatchQueue and / or Samplers. This can vary depending on the type of BatchQueue / Sampler given to the Factory. For example, to build a BatchHandlerDC object (data-centric batch handler) we use a queue and sampler which takes spatial and temporal weight / bin arguments used to determine how to weigh spatiotemporal regions when sampling. Using ConditionalBatchQueue will result in arguments for computing moments from batches and how to pad batch data to enable these calculations.

• sample_shape (tuple) – See DualSampler for full documentation.

• verbose (bool) – Whether to log timing information for batch steps.

Methods

check_enhancement_factors()	Make sure each DualSampler has the same enhancment factors and they match those provided to the BatchQueue.
check_features()	Make sure all samplers have the same sets of features.
check_shared_attr(attr)	Check if all containers have the same value for attr.
derive(feature[, strict])	Resolve feature name to a feature in the underlying data.
enqueue_batches()	Callback function for queue thread.
get_batch()	Get batch from queue or directly from a Sampler through sample_batch.
get_container_index()	Get random container index based on weights
get_queue()	Return FIFO queue for storing batches.
get_random_container()	Get random container based on container weights
init_samplers(train_containers, ...)	Initialize samplers from given data containers.
log_queue_info()	Log info about queue size.
post_init_log([args_dict])	Log additional arguments after initialization.
post_proc(samples)	Performs some post proc on dequeued samples before sending out for training.
preflight()	Run checks before kicking off the queue.
sample_batch()	Get random sampler from collection and return a batch of samples from that sampler.
sample_batches(n_batches)	Sample given number of batches either in serial or with thread pool.
start()	Start the val data batch queue in addition to the train batch queue.
stop()	Stop the val data batch queue in addition to the train batch queue.
transform(samples[, smoothing, smoothing_ignore])	Perform smoothing if requested.
wrap(data)	Return a Sup3rDataset object or tuple of such.

Attributes

timer
BATCH_MEMBERS
container_weights	Get weights used to sample from different containers based on relative sizes
data	Return underlying data.
features	Get all features contained in data.
hr_shape	Shape of high resolution sample in a low-res / high-res pair.
lr_shape	Shape of low resolution sample in a low-res / high-res pair.
queue_futures	Get number of scheduled futures that will eventually add batches to the queue.
queue_len	Get number of batches in the queue.
queue_shape	Shape of objects stored in the queue.
queue_thread	Get new queue thread.
running	Boolean to check whether to keep enqueueing batches.
shape	Get shape of underlying data.
shapes	Shapes of batches returned by __next__

SAMPLER

alias of SamplerTester

TRAIN_QUEUE

alias of DualBatchQueue

VAL_QUEUE

alias of DualBatchQueue

check_enhancement_factors()

Make sure each DualSampler has the same enhancment factors and they match those provided to the BatchQueue.

check_features()

Make sure all samplers have the same sets of features.

check_shared_attr(attr)

Check if all containers have the same value for attr. If they do the collection effectively inherits those attributes.

property container_weights

Get weights used to sample from different containers based on relative sizes

property data

Return underlying data.

Returns:

Sup3rDataset

See also

wrap()

derive(feature, strict=True)

Resolve feature name to a feature in the underlying data. This is used for handling feature aliases and for deriving new features from existing ones.

enqueue_batches() → None

Callback function for queue thread. While training, the queue is checked for empty spots and filled. In the training thread, batches are removed from the queue.

property features

Get all features contained in data.

get_batch() → DsetTuple

Get batch from queue or directly from a Sampler through sample_batch.

get_container_index()

Get random container index based on weights

get_queue()

Return FIFO queue for storing batches.

get_random_container()

Get random container based on container weights

property hr_shape

Shape of high resolution sample in a low-res / high-res pair. (e.g. (spatial_1, spatial_2, temporal, features))

init_samplers(train_containers, val_containers, sample_shape, feature_sets, batch_size, mode, sampler_kwargs)

Initialize samplers from given data containers.

log_queue_info()

Log info about queue size.

property lr_shape

Shape of low resolution sample in a low-res / high-res pair. (e.g. (spatial_1, spatial_2, temporal, features))

post_init_log(args_dict=None)

Log additional arguments after initialization.

post_proc(samples) → DsetTuple

Performs some post proc on dequeued samples before sending out for training. Post processing can include coarsening on high-res data (if Collection consists of Sampler objects and not DualSampler objects), smoothing, etc

Returns:

Batch (DsetTuple) – namedtuple-like object with low_res and high_res attributes. Could also include obs member.

preflight()

Run checks before kicking off the queue.

property queue_futures

Get number of scheduled futures that will eventually add batches to the queue.

property queue_len

Get number of batches in the queue.

property queue_shape

Shape of objects stored in the queue. Optionally includes shape of observation data which would be included in an extra content loss term

property queue_thread

Get new queue thread.

property running

Boolean to check whether to keep enqueueing batches.

sample_batch()

Get random sampler from collection and return a batch of samples from that sampler.

sample_batches(n_batches) → None

Sample given number of batches either in serial or with thread pool.

property shape

Get shape of underlying data.

property shapes

Shapes of batches returned by __next__

start()

Start the val data batch queue in addition to the train batch queue.

stop()

Stop the val data batch queue in addition to the train batch queue.

transform(samples, smoothing=None, smoothing_ignore=None)

Perform smoothing if requested.

Note

This does not include temporal or spatial coarsening like SingleBatchQueue

wrap(data)

Return a Sup3rDataset object or tuple of such. This is a tuple when the .data attribute belongs to a Collection object like BatchHandler. Otherwise this is Sup3rDataset object, which is either a wrapped 3-tuple, 2-tuple, or 1-tuple (e.g. len(data) == 3, len(data) == 2 or len(data) == 1). This is a 3-tuple when .data belongs to a container object like DualSamplerWithObs, a 2-tuple when .data belongs to a dual container object like DualSampler, and a 1-tuple otherwise.