compass.web.website_crawl.COMPASSCrawler#

class COMPASSCrawler(validator, url_scorer, file_loader_kwargs=None, already_visited=None, browser_semaphore=None, num_link_scores_to_check_per_page=4, max_pages=100, max_same_score_links_per_page=20, url_ignore_substrings=None, url_keep_substrings=None)[source]#

Bases: object

A simple website crawler to search for ordinance documents

Parameters:

  • validator (callable()) – An async callable that takes a document instance (containing the text from a PDF or a webpage) and returns a boolean indicating whether the text passes the validation check. This is used to determine whether or not to keep (i.e. return) the document.

  • url_scorer (callable()) – An async callable that takes a list of dictionaries containing URL information and assigns each dictionary a score key representing the score for that URL. The input URL dictionaries will each have at least one key: “href”. This key will contain the URL of the link. The dictionary may also have other attributes such as “title”, which contains the link title text.

  • file_loader_kwargs (dict, optional) – Additional keyword-value argument pairs to pass to the AsyncWebFileLoader class. If this dictionary contains the pw_launch_kwargs key, it’s value (assumes to be another dictionary) will be used to initialize the playwright instances used for the crawl. By default, None.

  • already_visited (set, optional) – A set of URLs (either strings or Link objects) that have already been visited. This is used to avoid re-visiting links that have already been checked. By default, None.

  • browser_semaphore (asyncio.Semaphore, optional) – Semaphore instance that can be used to limit the number of playwright browsers open concurrently. If None, no limits are applied. By default, None.

  • num_link_scores_to_check_per_page (int, default 4) – Number of top unique-scoring links per page to use for recursive crawling. This helps the crawl stay focused on the most likely links to contain documents of interest.

  • max_pages (int, default 100) – Maximum number of pages to crawl before terminating, regardless of whether the document was found or not. By default, 100.

  • max_same_score_links_per_page (int, default 20) – Maximum number of links with the same score to check per page. This helps prevent the crawler from spending too much time on links that have identical scores. By default, 20.

  • url_ignore_substrings (Iterable of str, optional) – URL parts that exclude matching crawl candidates. By default, None.

  • url_keep_substrings (Iterable of str, optional) – URL parts that override all crawl blacklist matches. By default, None.

Methods

run(base_url, crawl_timeout_s[, ...])

Run the COMPASS website crawler
async run(base_url, crawl_timeout_s, termination_callback=None, on_new_page_visit_hook=None)[source]#

Run the COMPASS website crawler

Parameters:

  • base_url (str) – URL of the website to start crawling from.

  • crawl_timeout_s (float, optional) – A float representing the maximum time in seconds to allow the crawl to run.

  • termination_callback (callable(), optional) – An async callable that takes a list of documents and returns a boolean indicating whether to stop crawling. If None, the crawl will simply terminates when DOC_THRESHOLD number of documents have been found. By default, None.

  • on_new_page_visit_hook (callable(), optional) – An async callable that is called every time a new page is found during the crawl. The callable should accept a single argument, which is the page Link instance. If None, no additional processing is done on new pages. By default, None.

Returns:

list – List of document instances that passed the validation check. Each document contains the text from a PDF and has an attribute source that contains the URL of the document. This list may be empty if no documents are found.