compass.pipeline.collection.dedupe.DocumentDeDuplicator#

class DocumentDeDuplicator(dict=None, /, **kwargs)[source]#

Bases: UserDict

Domain Service for deduplicating collected documents

Methods

add_docs(docs, *[, step_name])

Add documents to the collection mapping

info(doc)

Get the info for a given document
add_docs(docs, *, step_name=None)[source]#

Add documents to the collection mapping

Parameters:

  • docs (list) – Collected document objects to add to the internal de-duplicated mapping.

  • step_name (str, optional) – Identifier for the collection step that produced the documents. If not provided, “from_steps” will not be updated for the added documents. By default, None.

info(doc)[source]#

Get the info for a given document

Parameters:

doc (elm.web.document.BaseDocument) – The document for which to retrieve the deduplication info.

Returns:

object – The deduplication info for the given document.