compass.pipeline.runtime.PipelineRuntime

class PipelineRuntime(request, re_run_targets=None)

Bases: object

Context Object for runtime dependencies in one pipeline run

Parameters:

• request (compass.pipeline.data_classes.BaseRequest) – Request object containing all user inputs and settings for this run.

• re_run_targets (dict, optional) – Per-shard metric names mapped to minimum counts for continuing a run. An empty mapping continues with the default nonempty shard requirement. All targets must pass in the re-run. By default, None starts a fresh run.

Methods

Attributes

browser_semaphore	Browser concurrency limiter
crawl_semaphore	Crawl concurrency limiter
dirs	Directories object for this run
extractor_class	Extractor class for the configured tech
file_loader_kwargs	Loader kwargs for remote documents
file_loader_kwargs_no_ocr	Loader kwargs without OCR for website validation
jurisdiction_semaphore	Jurisdiction semaphore or inert context manager
local_file_loader_kwargs	Loader kwargs for local documents
search_engine_semaphore	Search engine concurrency limiter
tpe_kwargs	Thread pool kwargs for this run

property dirs

Directories object for this run

property tpe_kwargs

Thread pool kwargs for this run

property extractor_class

Extractor class for the configured tech

property browser_semaphore

Browser concurrency limiter

property crawl_semaphore

Crawl concurrency limiter

property search_engine_semaphore

Search engine concurrency limiter

property jurisdiction_semaphore

Jurisdiction semaphore or inert context manager

property file_loader_kwargs

Loader kwargs for remote documents

Type:

dict

property local_file_loader_kwargs

Loader kwargs for local documents

Type:

dict

property file_loader_kwargs_no_ocr

Loader kwargs without OCR for website validation

Type:

dict