compass.scripts.download.download_jurisdiction_ordinances_from_website_compass_crawl#

async download_jurisdiction_ordinances_from_website_compass_crawl(website, heuristic, keyword_points, file_loader_kwargs=None, already_visited=None, num_link_scores_to_check_per_page=4, max_urls=100, pb_jurisdiction_name=None, timeout_seconds=3600, url_ignore_substrings=None, url_keep_substrings=None)[source]#

Download ord documents from a website using the COMPASS crawler

The COMPASS crawler is much more simplistic than the Crawl4AI crawler, but is designed to access some links that Crawl4AI cannot (such as those behind a button interface).

Parameters:

  • website (str) – URL of the jurisdiction website to search.

  • heuristic (callable()) – Callable taking an BaseDocument and returning True when the document should be kept.

  • keyword_points (dict) – Dictionary of keyword points to use for scoring links. Keys are keywords, values are points to assign to links containing the keyword. If a link contains multiple keywords, the points are summed up.

  • file_loader_kwargs (dict, optional) – Dictionary of keyword arguments pairs to initialize elm.web.file_loader.AsyncWebFileLoader. If found, the “pw_launch_kwargs” key in these will also be used to initialize the elm.web.search.google.PlaywrightGoogleLinkSearch used for the Google URL search. By default, None.

  • already_visited (set of str, optional) – URLs that have already been crawled and should be skipped. By default, None.

  • num_link_scores_to_check_per_page (int, default 4) – Number of top-scoring links to visit per page. By default, 4.

  • max_urls (int, default 100) – Max number of URLs to check from the website before terminating the search. By default, 100.

  • pb_jurisdiction_name (str, optional) – Optional jurisdiction name to use to update progress bar, if it’s being used. By default, None.

  • timeout_seconds (float, default 3600) – Maximum number of seconds to allow for the crawl before timing out. By default, 3600 (1 hour).

  • url_ignore_substrings (Iterable of str, optional) – URL parts that exclude matching crawl candidates. These are the same values used to filter search results. By default, None.

  • url_keep_substrings (Iterable of str, optional) – URL parts that override all crawl blacklist matches. These are the same values used to filter search results. By default, None.

Returns:

out_docs (list) – List of BaseDocument instances containing potential ordinance information, or an empty list if no ordinance document was found.

Notes

Requires TempFileCache service to be running.