Liking cljdoc? Tell your friends :D
clj-robots-parser 
What
A Clojure(-script) library to parse robots.txt files as specified by The Great Goog themselves. As robots.txt is woefully underspecified in the "official" docs, this library tolerates anything it doesn't understand, extracting the data it does.
It can use the extracted data to query whether a given user-agent is allowed to crawl a given URL.
Why
Why use Google's (much more stringent) documentation for handling robots.txt? In terms of SEO, googlebot is what you ought to care about the most.
Can you improve this documentation? These fine people already did:
Ian S Kerins & Ian KerinsEdit on GitHub
cljdoc is a website building & hosting documentation for Clojure/Script libraries
× close