compass.pipeline.collection.steps.SearchResultsCrawlStep#

class SearchResultsCrawlStep[source]#

Bases: CollectionStep

Crawl HTML pages discovered by the search-engine step

Methods

collect(workflow)

Crawl the configured prefix of HTML search candidates

Attributes

STEP_NAME

Identifier for step
STEP_NAME = 'search_results_crawl'#

Identifier for step

async collect(workflow)[source]#

Crawl the configured prefix of HTML search candidates