Proof Specs

Automatic testing of clojure.spec data generators.

Dependency coordinates

Documentation

API documentation is available inline and at cljdoc.

RATIONALE

clojure.spec generators can fail for unexpected reasons. It requires expertise to see where a complex spec fails to generate.

A practical way to work around the problem is to build and test specs incrementally; the clojure.spec design encourages building nested specs from components; speccing collections independently of their components, building keysets (maps) out of individually specced keys allows testing spec compoments independently. It still takes dicipline to test incrementally and backtracing where a definition went wrong can take much longer than is desirable.

RESOLUTION

Test component specs automatically using proof-specs. This adds the requirement that all specs registered in a particular set of namespaces must have a working generator. proof-spes will attempt to generate data for each spec in the selected namespaces and reports on every failure.

CONSEQUENSES

Specs that are only used as return specs for functions must have a generator, even though they won't be used in any other test.

NON-GOALS

proof-specs will not try to figure out the reason for a spec/generator not working. It will just list all failing generators and the given errors.

USAGE

Running from clojure code

(require '[nl.jomco.proof-specs :refer [proof-specs]])

(proof-specs
  :num-vals 10
  :verbose true
  :limit-ms 500
  :include [#"nl.jomco.*"])

Produces something like:

(proof-specs {:limit-ms 1000
                                             :num-vals 10
                                             :include #{#"nl\.jomco\.proof-specs-test"}})
{:problems
 #:nl.jomco.proof-specs-test{:non-spec
                             {:nl.jomco.proof-specs/message
                              "Unable to construct gen at: [] for: :nl.jomco.proof-specs-test/non-spec",
                              :nl.jomco.proof-specs/type
                              clojure.lang.ExceptionInfo,
                              :clojure.spec.alpha/path [],
                              :clojure.spec.alpha/form
                              :nl.jomco.proof-specs-test/non-spec,
                              :clojure.spec.alpha/failure :no-gen},
                             :missing-deps
                             #:nl.jomco.proof-specs{:message
                                                    "Unable to resolve spec: :nl.jomco.proof-specs-test/missing",
                                                    :type
                                                    java.lang.Exception},
                             :non-spec-dep
                             {:nl.jomco.proof-specs/message
                              "Unable to construct gen at: [:nl.jomco.proof-specs-test/non-spec] for: :nl.jomco.proof-specs-test/non-spec",
                              :nl.jomco.proof-specs/type
                              clojure.lang.ExceptionInfo,
                              :clojure.spec.alpha/path
                              [:nl.jomco.proof-specs-test/non-spec],
                              :clojure.spec.alpha/form
                              :nl.jomco.proof-specs-test/non-spec,
                              :clojure.spec.alpha/failure :no-gen}},
 :specs
 #{:nl.jomco.proof-specs-test/non-spec
   :nl.jomco.proof-specs-test/missing-deps
   :nl.jomco.proof-specs-test/non-spec-dep}}

The return value will not contain a :problems key when no problems were found.

Running as a leiningen alias

Add a "proof-specs" alias to your project.clj - also make sure you include proof-specs as a development dependency.

  ...
  profiles {:dev {:dependencies [[nl.jomco/proof-specs "<VERSION>"]]}}
  ...
  :aliases {"proof-specs" ["run" "-m" "nl.jomco.proof-specs"
                           "--include" ".*jomco.*"
                           "--require" "nl.jomco.proof-specs,nl.jomco.blerk"]}
  ...

And run using:

lein proof-specs

Produces something like:

Problems generating data for 1 out of 12 specs:
:nl.jomco.blerk/yelp

#:nl.jomco.blerk{:yelp #error {
 :cause "Couldn't satisfy such-that predicate after 100 tries."
 :data {...}, :max-tries 100}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message "Couldn't satisfy such-that predicate after 100 tries."
   :data {...}, :max-tries 100}
   :at [...]}]
 :trace
 [...]}}

The exit status will be 0 when no problems were found, otherwise 1.

Interpreting output

When there are multiple failing generators, a good strategy is to focus on and fix the simplest generators first, since that also fixes complex generators that only fail because of failing components.