nl.jomco.spider — nl.jomco/spider 0.2.0

apply-templ clj

(apply-templ val env)

Apply template expressions in val with given env.

Templates are datastructures that can contain:

• maps and vectors; values are evaluated (not keys)
• lists; evaluated as expressions
• symbols starting with ? evaluated as expressions
• strings; evaluated for expressions within { and } characters.

Example:

(apply-templ "1 + {x} = {(+ 1 x)}" {'+ +, 'x 2}) ;; -> "1 + 2 = 3"

default-env clj

entries clj

(entries coll)

Return key => value pairs for each entry in coll. For vectors, returns index => value pairs.

evaluate clj

(evaluate expr env)

Evaluate parsed expr with given env.

Apart from the values and functions given in env, special forms (if PRED TRUE-FN FALSE-FN), (and PRED-1 PRED-2 ..) and (or PRED-1 PRED-2 ..) are also interpreted. Both and and or return a boolean.

Examples:

(evaluate '(+ 1 2) {'+ +}) ;; => 3 (evaluate '(+ 1 2 x) {'+ +, 'x 3}) ;; => 6 (evaluate '(if (or (< 0 x 5) (< 10 x 15)) "yes" "no") {'< <, 'x 3}) ;; => "yes"

generate clj

(generate templs matches env)

Generate values from templs using matches (a result from multi-select) and env (for evaluate).

harvest clj

(harvest interaction rules env)

Returns a set of requests generated from the given interaction.

An interaction (map containing :request and :response) is matched against rules and the resulting selections are returned.

rules and env arguments apply as in select and generate.

interactions clj

(interactions exec-request generate-requests seed-requests)

(interactions exec-request generate-requests seed-requests seen-requests)

Returns lazy seq of interactions from seed-requests.

exec-request is called once for every request and should return either:

• a response, resulting in an interaction
• :nl.jomco.spider/skip if the request should be skipped
• nil to stop spidering

interactions are maps containing a :request and :response. Skipped requests do not result in an interaction.

seed-requests is a collection of request maps.

generate-requests takes an interaction and returns a collection of requests to be executed.

seen-requests is an inital set of requests to ignore when generated by generate-requests.

multi-select clj

(multi-select data rules)

Run rules (a collection of queries for select against data and join the results.

Like select, returns a (possibly empty, for no matches) relation of bindings

placeholder? clj

(placeholder? x)

select clj

(select data query)

Select query from data. A query is a list of terms navigating into the data. Query terms which are symbols starting with a ? character are considered to be placeholders.

Returns a clojure.set compatible relation result:

• Empty set when no matches are found
• #{{}} (a set containing an empty map) for an exact match
• A set of maps with bindings for every match with placeholders.

Bindings are unified; if a placeholder appears multiple times, the value for that placeholder must be the same for every appearance in a match.

Examples:

(select {:name "fred", :friends [{:name "barney"}]} '[:name ?name]) ;; => #{{?name "fred"}}

(select {:name "fred", :friends [{:name "barney" :name "dino"}]} '[:friends ?i :name ?name]) ;; => #{{?i 0, ?name "barney"} {?i 1, ?name "dino"}}

spider clj

(spider {:keys [env exec-request rules seeds] :or {env default-env}})

Return a lazy seq of interactions by executing requests.

Interactions are maps of {:request .. :response ...}

Response is the result of (exec-request request) After a request is executed, the resulting interaction is used to generate new requests given rules.

Options:

• env - the environment for evaluation rules. Default is default-env

See also harvest and interactions.