Liking cljdoc? Tell your friends :D

Cruler

ci Clojars Project

Cruler is a framework of file format validation.

What is it

There are many cases you want to implement validation of file format. But creating validator is time-consuming because you need to consider input-file format, human-readable error message, result format etc.

Cruler is a validation framework. Cruler reduces time to create validators and make validator's error more human-readable.

Features

  1. Provide a uniform result format
  2. Provide human readable error message
  3. Redefine clojure.spec error message
  4. Make validators reusable

Quick Start

Cruler has sample validators. You can run the validators.

$ clojure -M:validate "dev-resources/sample-validator" -v

Loading config: dev-resources/sample-validator/cruler.edn

Validating :sample-validator.sort/sort

Validating :cruler.validators/start-of-file

Validating :sample-validator.csv-blank/csv-blank

Validating :cruler.validators/blank-line

Validating :cruler.validators/trailing-whitespace

Validating :sample-validator.spec/approval

Validating :sample-validator.reference/approval<->drug

Validating :cruler.validators/end-of-file

Validating :sample-validator.duplication/duplication

Ran 9 validations.
9 passes, 0 failures.

Then you should modify a file to be validated. For example, you add new line to description/test.txt as follows.

Abilify
Bacitracin

Cabergoline
Dabigatran

You will get a validation fail message with error preview.

ERROR at :cruler.validators/blank-line validator
Blank line is found at line

dev-resources/sample-validator/description/test.txt
  line: 3
  preview:
-----
Abilify
Bacitracin

Cabergoline
Dabigatran
-----

Usage

Pre Requirements

  • You already have project which has validator according to the rules of cruler
  • You created cruler.edn in the project

In detail, see ruler of validator and cruler.edn section.

Run Validations

Cruler's basic usage is simple.

$ clojure -M:validate /path/to/validator/project

For example, in case you have sample-validate-project as follows,

├── cruler
└── sample-validate-project
    ├── cruler.edn
    ├── resources
    │   ├── test.txt
    │   └── test.csv
    └── validator
        ├── validator1.clj
        └── validator2.clj

then you can run validator/*.clj.

$ clojure -M:validate "../sample-validate-project"
...
...

Ran XXX validations.
XXX passes, XXX failures.

Use with Docker

You can use the docker image on DockerHub/cruler.

# Specify image tag
$ TAG=1.2.2
$ docker pull xcoo/cruler:${TAG}

# Run validation
$ docker run --rm -v /path/to/validator/project:/cruler -it xcoo/cruler:${TAG}

Use as a library

You can use Cruler as a library.

(ns sample
  (:require [cruler.core :as cc]))

(defn run [_]
  (let [[config-file-path config] (cc/setup-config "dev-resources/sample-validator" "cruler.edn")]
    (println config-file-path)

    ;; If you want to validate a single file, you can use "run-validators-single-file" method.
    (println (cc/run-validators-single-file (:validators config) "dev-resources/sample-validator" "description/test.txt"))

    ;; If you want to validate a directory as same as CLI, you can use "run-validators" method.
    (println (cc/run-validators (:validators config) "dev-resources/sample-validator"))))

Options

$ clojure -M:validate -h

Usage: cruler [<options>] [<directory>]

Options:
  -c, --config CONFIG  Specify a configuration file (default: cruler.edn)
  -v, --verbose        Make cruler verbose during the operation

Rule of validator

A simple validator is as follows.

(defmethod validate ::validate-key
  [_ data]
  (let [files (remove #(re-find #"(\A|[^\n]+\n)\z" (:raw-content %)) data)]
    {:errors (map #(select-keys % [:file-path]) files)
     :message "Error message"}))

Cruler requires some specifications for validators. As sample, you can refer sample-validators.

defmethod

Cruler defines (defmulti validate (fn [key _] key)), and expects validators to be dispatched. So you should declare validators as defmulti validate ::key.

Args

First arg is not used in validators. Second arg is data. data is a array of Map, and the Map conatins key-value as follows.

keytypevaluedescription
:file-pathstring(.getPath file)Relative path from the project's root dir
:file-typekeyword:csv, :text, or :yamlFile type
:raw-contentstring(slurp file)Raw string of file contents
:parsed-contentanySee cruler.parserThe parsed data of :raw-content. The structure of the data depends on :file-type.

Return value

Cruler expects validators to return Map. The Map should be as follow.

keytypedescription
:errorssequenceSee errors
:messagestringIf :errors is not empty, the message is shown

:errors

:errors is array of Map. The Map should be as follows.

keytypedescription
:file-pathstringRelative path from the project's root dir
:error-valuestring or hash-mapWhen it is string, it should be invalid value. And when it is hash-map, it has at least :path :pred and :val keys. describing the predicate and the value that failed at that path. See explain-data
:error-blockstringA block in :parsed-content containing the error location. See spec.clj as sample
:error-keyssequenceKeys in :error-block for indicating the specific error location.

cruler.edn

cruler.edn is a configuration file of cruler.

{:validators {
   :validator.namespace/validator-key ["regex of resource file"]}
 :paths ["validator"]
 :colorize true
 :format {:error-value :pprint}
 :deps [[library version]]}

cruler.edn requires some keys.

keyrequiredefaultdescription
:validatorstrue Define the validator and resources to be validated
:pathsfalse["validator"]Define the classpaths including validator source codes
:colorizefalsetrueDefine whether to color the output result
:formatfalse{:error-value :pprint}Define an error-value print function. You can use :pprint or :print
:depsfalsenilDefine the dependencies which the project require as library

See cruler.edn.sample and sample-validator/cruler.edn as samples.

clojure.spec error messages

cruler can make clojure.spec error messages human-readable.

In case, you have this yml and define spec.

- drug: foo
  types:
    - category: A
      # serial: 1234
  comment: A1
(s/def ::type
  (s/keys :req-un [:sample/serial]
          :opt-un [:sample/category]))

You will get this error message.

{:category "A"} - failed: (contains? % :serial) in: [0 :types 0] at: [:types] spec: :sample-validator.spec/type

cruler will replace this clojure.spec error message with readable message like bellow.

error: Missing key: serial
  {:category "A"}

Redefine error message

You can also manually redefine clojure.spec error message using defmsg.

(ns sample
  (:require [cruler.spec-parser :refer [defmsg]]))

(defmsg ::group "Should be A, B or C")
(s/def ::group #{"A" "B" "C"})

cruler can show error as "Should be A, B or C" if you not satisfy the predicate ::group. As sample, you can refer sample_validator/spec.clj.

Unit Test & Lint

using clojure.test and cljfmt, you can run unit test and lint.

$ clojure -M:test
$ clojure -M:lint

License

Copyright 2020 Xcoo, Inc.

Licensed under the Apache License, Version 2.0.

Can you improve this documentation? These fine people already did:
Toshiki Takeuchi, kbaba1001, Taiju Aoki & Takashi AOKI
Edit on GitHub

cljdoc is a website building & hosting documentation for Clojure/Script libraries

× close