common-crawl-utils.reader — lt.tokenmill/common-crawl-utils 0.1.0

get-cdx-urls clj

(get-cdx-urls id)

get-urls clj

(get-urls path)

get-warc-urls clj

(get-warc-urls id)

read-coordinates clj

(read-coordinates)

(read-coordinates id)

Given an ID of the Crawl returns a sequence of Common Crawl Coordinates records. ID example: "CC-MAIN-2019-35". By default reads coordinates from the latest Crawl. Returns the sequence of the coordinates.

read-warc clj

(read-warc)

(read-warc id)

Given an ID of the Crawl returns a sequence of WARC records. ID example: "CC-MAIN-2019-35". By default reads WARCs from the latest Crawl. Returns the sequence of the WARC records.