clj-robots-parser.core — isker/clj-robots-parser 0.1.3

is-crawlable? clj/s

(is-crawlable? robots-txt url user-agent)

Does the given parsed robots.txt permit the given URL to be crawled by the given user-agent?

parse clj/s

(parse content)

Parses the given string (content of a robots.txt file) into data that can be queried.

query-crawlable clj/s

(query-crawlable {:keys [agent-groups]} url user-agent)

Determines whether and explains why the given parsed robots.txt does or does not permit the given URL to be crawled by the given user-agent.

robots-txt-line clj/s

stringify-query-result clj/s

(stringify-query-result {:keys [raw-content]}
                        {:keys [because]}
                        &
                        {:keys [context] :or {context 1}})

Creates a user-readable string explanation of a query-crawlable result by providing contextual highlighting of the source robots.txt that produced it.