Liking cljdoc? Tell your friends :D
diction
Diction is a Clojure library similar to spec but with added functionality and some batteries included.
Status of Diction
Existing Clojure libraries/frameworks for data specifications, validation, and more:
spec: https://clojure.org/guides/specprismatic/schema: https://github.com/plumatic/schema
Diction is meant to provide everything out-of-the-box for data dictionary, schema definition, schema validation, schema generation, custom diction schema types, custom/dynamic validation rules, custom/dynamic decoration rules, import/export of diction schemas, data metadata definition/management/query, and so much more.
Diction is also meant to change and improve to handle the evolving and dynamic nature of enterprise data element management.
Features
- data dictionary
- batteries-included baked-in default types
- primitives: string, int, long, float, double
- collections: enum, vector
- rich types: uuid, joda
- composite: documents/entities
- declarative and/or rich definition
- basic validation
- descriptive explanation of validation failures
- generation w/ optional
sensible-values(human-readable) - grooming (recursive document sieve to only only declared fields through)
- metadata
- audit
- personal identifiable information (PII) tracking at element level
- labels and descriptions
- custom
- query
- batteries-included baked-in default types
- flexibility
- leverages functions, not macros, to allow for runtime diction processing
- diction elements may be defined completely declaratively (data/declarative vs. code/programmatic)
- allows for off-line storage of diction definitions in files
- exports diction definitions to files
- imports diction definitions from files
- custom diction types
- supports custom diction elements
- create custom diction validation functions
- create custom diction value generator functions
- extensible data rules (function-based)
- custom rich functional/programmatic rules registered to elements
- fires upon validation
- extensible decoration rules (function-based)
- custom annotation/decoration function/programmatic rules registered to elements
- fires all decoration rules associated with element using the element value as start
- multiple (or none) decorator rules may be defined per diction element id
- generative function testing
- register function, argument diction(s), and result diction
- generative function testing using the data generated by the diction elements
- guard functions (similar to
:preannotationed functions) - HTTP payload / parameter validation hooks
- generated hyperlinked data dictionary documentation
- markdown
- hiccup
- HTML
Planned Features
- integration with document-oriented NoSQL stores (MongoDB, AWS DocumentDB, Codax, ?)
- predicate logic in declarative diction element definitions (simliar to
spec) - rich meta data handling for compliance and data element awareness
Quick Start
Data elements may be defined/registered declaratively (data) or programmatically (code).
Declarative Element Registration
The diction library allows for the registration of data elements via data
for data elements that use the built-in data diction types.
Please refer to the diction data elements generated documentation in the
diction wiki.
(def dictionary
[
{:id :first_name
:type :string
:min 1
:max 50
:meta {:description "First or given name."
:sensible-values ["John" "Jane" "Juan" "Maria" "Abhul"]}}
{:id :last_name
:type :string
:min 1
:max 50
:meta {:description "Last name or surname."
:sensible-values ["Smith" "Lopez" "Nguyen" "Chang"]}}
{:id :address
:type :string
:min 1
:max 100
:meta {:description "Address line."
:sensible-values ["123 Main St." "34 Rue De Fleurs"]}}
{:id :address2
:type :string
:min 0
:max 100
:meta {:description "Address line #2 for units, suites, etc."
:sensible-values ["Unit 42" "#10" "Apt. D" "Suite 4100"]}}
{:id :city
:type :string
:min 1
:max 50
:meta {:description "City or town."
:sensible-values ["Las Vegas" "Paris" "London" "Beijing" "Tokyo"]}}
{:id :province
:type :string
:min 1
:max 50
:meta {:description "Province or state."
:sensible-values ["NV" "ID" "WY"]}}
{:id :postal_code
:type :string
:meta {:description "Postal or zip code."
:sensible-values ["89521" "NR14 7PZ"]}}
{:id :country
:type :string
:meta {:description "Country code."
:sensible-values ["US" "CH" "FR"]}}
{:id :phone
:meta {:description "Phone number"
:sensible-values ["+1 (415) 622-1233" "+42 34 2344 234"]}
:type :string
:min 1
:max 30}
{:id :cell_phone
:clone :phone
:meta {:description "Cell phone"}}
{:id :home_phone
:clone :phone
:meta {:description "Home phone"}}
{:id :work_phone
:clone :phone
:meta {:description "Work phone"}}
{:id :email
:meta {:description "Email address"
:sensible-values ["jane@acme.org" "mulan@cater.io"]}
:type :string
:regex-pattern ".+@.+..{2,20}"}
{:id :contact_email
:clone :email
:meta {:description "Contact email address"
:sensible-values ["jane@acme.org" "mulan@kemper.io"]}}
{:id :notification_email
:clone :email
:meta {:description "Automated notification address"
:sensible-values ["orders@acme.org" "fulfilment@sunshine.io"]}}
{:id :notes
:type :string
:meta {:description "Notes"
:sensible-values ["Some notes" "Notes are here" "And more notes"]}}
{:id :active
:meta {:description "Active flag (true/false)"}
:type :boolean}
{:id :latitude
:meta {:description "Latitude geo"}
:type :double
:min -90.0
:max 90.0}
{:id :longitude
:meta {:description "Longitude geo (-180 - 180)"}
:type :double
:min -180.0
:max 180.0}
{:id :long_lat
:meta {:description "Longitude and latitude pair"}
:type :tuple
:tuple [:longitude :latitude]}
{:id :long_lats
:meta {:description "List of longitude/latitude pairs."
:sensible-min 1
:sensible-max 1}
:type :vector
:vector-of :long-lat
:min 1
:max 10}
{:id :tag
:meta {:description "Smart tag"
:sensible-values ["tag1" "tag2" "tag3"
"tag4" "tag5" "tag6"
"tag99"]}
:type :string
:min 2
:max 32}
{:id :tags
:meta {:description "Smart tags"
:sensible-min 1
:sensible-max 4}
:type :set
:set-of :tag
:min 1
:max 12}
{:id :id
:type :string
:meta {:description "Unique identifier."
:sensible-values ["id1" "id2" "abcdef.id"]}}
{:id :person
:type :document
:meta {:description "Person"}
:required-un [:id :first_name :last_name :address :province :city :country :province]
:optional-un [:active :address2 :email :cell_phone :work_phone :home_phone :long_lat :tags]}
])
And then register with diction via diction.core/import! for a single data element
map:
(diction/import! {:id :label :type :string :min 1 :max 50})
Or diction.core/imports! for a vector of data element maps.
(diction/imports! dictionary)
(diction/generate :person)
;=>
{:address "123 Main St.",
:city "London",
:country "CH",
:first_name "Juan",
:id "id2",
:last_name "Nguyen",
:province "WY"}
Programmatic Element Registration
;; defines label as a string with a minimum length of 0 and max length of 8
;; w/ built-in validation and generation
(string! ::label {:min 0 :max 8})
;; defines unit-count as a long (default whole numbers in Clojure) with a min of 0 and max of 999.
;; w/ built-in validation and generation
(long! ::unit-count {:min 0 :max 999})
;; defines unit-cost as a double (default real numbers in Clojure) with a min of 0.01 and max of 99.99.
;; w/ built-in validation and generation
(double! ::unit-cost {:min 0.01 :max 99.99})
(string! ::tag {:min 2 :max 12})
;; defines tags as a vector of diction element `::tag` with min item count of 0 and max of 64.
;; w/ built-in validation and generation
(vector! ::tags ::tag {:min 0 :max 8})
;; defines badge as a keyword diction element with max length of 8.
;; w/ built-in validation and generation
(keyword! ::badge {:max 8})
;; defines created as a joda datetime element
;; w/ built-in validation and generation
(joda! ::created)
;; defines id as a specialized string type uuid
;; w/ built-in validation and generation
(uuid! ::id)
;; defines cost-bases as an enum of keywords
;; w/ built-in validation and generation
(enum! ::cost-basis #{:flat :per-unit})
(string! ::description {:min 0 :max 128})
(double! ::percent-upcharge {:min 0.0 :max 1.0})
(double! ::fee-upcharge {:min 0.0 :max 999.99})
;; defines additional-charges-fee as a document
;; convenience function `document!`
;; takes the diction element id,
;; then the unqualified required diction element id(s),
;; then the unqualified optional diction element id(s)
(document! ::additional-charges-fee
[::label ::fee-upcharge ::cost-basis]
[::description])
(document! ::additional-charges-percent
[::label ::percent-upcharge ::cost-basis]
[::description])
(vector! ::fees-template ::additional-charges-fee)
(vector! ::percents ::additional-charges-percent)
;; clones diction element fees-template to new diction element fees
;; w/ built-in validation and generation
(clone! ::fees-template ::fees)
(document! ::additional-charges
[]
[::fees ::percents])
;; defintes an item diction element with required un-namespaced
;; ::id ... ::unit-cost, and optional un-namespaced ::badge ... ::additional-charges
;; note: the document allows for nest documents (like ::additional-charges)
(document! ::item
[::id ::label ::unit-count ::unit-cost]
[::badge ::tags ::description ::additional-charges])
Working with Diction
Generation
(diction/generate :person)
;=>
{:address "123 Main St.",
:city "London",
:country "CH",
:first_name "Juan",
:id "id2",
:last_name "Nguyen",
:province "WY"}
Validation / Explanation
The valid? and valid-all? predicate functions determine if the value is compliant
against the diction element id provided.
If the return is true then the value is valid/compliant.
If the return is 'false' then the value is not valid/compliant.
(valid? ::element-id value) ; validates only against the diction element validation function
(valid-all? ::element-id value) ; validates not only against the diction element validation
; function but with all associated validation rules for the element id
The explain? and explain-all? functions determine if the value is compliant
against the diction element id provided.
(explain ::element-id value) ; validates and explains any failures only against the diction
; element validation function
(explain-all ::element-id value) ; validates and explains any failures not only against the diction element
; validation function but with all associated validation rules for the element id
If the return is nil then the value is valid/compliant.
Otherwise, the return is a vector of failure maps with the following shape:
[
{:id :id-of-the-diction-element-tested-against
:v :value-tested
:msg "Failure message that should be descriptive to aid in troubleshooting."}
{:id :id-of-the-diction-element-tested-against
:v :value-tested
:msg "Failure message that should be descriptive to aid in troubleshooting."}
]
(explain ::tag "t")
;=>
[{:id :diction.example/tag,
:entry {:id :diction.example/tag,
:element {:id :diction.example/tag,
:type :string,
:min 2,
:max 12,
:gen-f #object[diction.core$wrap_gen_f$fn__4572
0x37e5f92c
"diction.core$wrap_gen_f$fn__4572@37e5f92c"],
:valid-f #object[clojure.core$partial$fn__5565 0x424f7cf2 "clojure.core$partial$fn__5565@424f7cf2"],
:parent-id :diction/string}},
:v "t",
:msg "Failed ':diction.example/tag': value 't' has too few characters. (min=2)"}]
(explain ::item {:id "da97d0b6-9c1d-495b-5367-c23da019dc01",
:label "EXJIJS",
:unit-count 33,
:unit-cost 75.0581280630676,
:badge :VHObB,
:tags ["8dsdfasdfs" "y6n" "q4oH" "+ bb" "ny7LZO6" "ru" "F4vi8nNSo5o"],
:description "19Ti7Ekh6wassjon3RJ=jmBhsX-bLygGAy6pcGl3F6KM3",
:additional-charges {:fees [{:label "Ut",
:fee-upcharge 193.5841860568779,
:cost-basis :flat,
:description "HwGpoD7o96pGsM7N2mqq7To2fHfq7=ekjHxO+"}
{:label "fD7AXP",
:fee-upcharge 350.57367256651224,
:cost-basis :per-unit-bad-xxx, ;; bad cost-basis enum
:description "PswklPNDFbJS0l9QrBCnpT6I=PEM41Hq5B9"}
{:label "",
:fee-upcharge 286.83015039087127,
:cost-basis :flat}]}})
;=>
[{:id :diction.example/cost-basis,
:entry {:id :diction.example/cost-basis,
:element {:id :diction.example/cost-basis,
:enum #{:flat :per-unit},
:gen-f #object[diction.core$wrap_gen_f$fn__5476
0x6d164c9e
"diction.core$wrap_gen_f$fn__5476@6d164c9e"],
:valid-f #object[clojure.core$partial$fn__5561 0x5be3e4b7 "clojure.core$partial$fn__5561@5be3e4b7"]}},
:msg "Failed :diction.example/cost-basis: value ':per-unit-bad-xxx' not in enum '#{:flat :per-unit}'.",
:parent-element-id [:diction.example/additional-charges-fee :diction.example/additional-charges :diction.example/item]}]
Custom Data Element Types
To create a custom data element type, you need the following:
- type generator function
- type validation function
- type normalization function
- add type normalizer to diction
- partial type element registration function (optional)
Custom Type Generation Function
(defn generate-odd-pos-int
"Generate a simple odd, positive integer."
[]
(-> (rand)
(* (/ Integer/MAX_VALUE 2))
int
(* 2)
inc))
Custom Type Validation Function
(defn validate-odd-pos-int
"Validate a simple odd, positive integer."
[v id entry]
(when (not (and (int? v) (odd? v)))
(let [fail {:id id :v v :entry entry}]
(filter #(not (nil? %))
[(when-not (int? v)
(assoc fail
:msg (str "Value '" v "' for field " id " needs to be an integer.")))
(when-not (if (number? v)
(odd? (long v))
true)
(assoc fail
:msg (str "Value '" v "' for field " id " needs to be odd.")))]))))
Custom Normalization Function
(defn normalize-odd-pos-int
"Normalizes the odd positive int number entry type elements (if necessary) given element map `m`. If not a int number type, passhtru
the element map `m`."
[m]
(if (= :odd-pos-int (:type m))
(assoc m :gen-f (diction/wrap-gen-f generate-odd-pos-int)
:valid-f validate-odd-pos-int)
m))
Add Custom Type Normalizer to Diction
(diction/type-normalizer! normalize-odd-pos-int)
Create a Custom Data Element Registration Function
(def odd-pos-int! (partial diction/custom-element! :odd-pos-int))
Register Custom Data Type Field
(odd-pos-int! :field-of-odd-pos-int)
Exercising Custom Data Type
(diction/generate :field-of-odd-pos-int)
; => 2115533555
(diction/valid? :field-of-odd-pos-int 33)
; => true
(diction/valid? :field-of-odd-pos-int 34)
; => false
(diction/explain :field-of-odd-pos-int 33)
; => nil
(diction/explain :field-of-odd-pos-int 34)
; =>
[{:id :field-of-odd-pos-int,
:v 34,
:entry {:id :field-of-odd-pos-int,
:element {:id :field-of-odd-pos-int,
:type :odd-pos-int,
:gen-f #object[diction.core$wrap_gen_f$fn__13635
0x1cde4602
"diction.core$wrap_gen_f$fn__13635@1cde4602"],
:valid-f #object[diction.demo$validate_odd_pos_int
0x640ed2e7
"diction.demo$validate_odd_pos_int@640ed2e7"]}},
:msg "Value '34' for field :field-of-odd-pos-int needs to be odd."}]
(diction/explain :field-of-odd-pos-int 34.3)
;=>
[{:id :field-of-odd-pos-int,
:v 34.3,
:entry {:id :field-of-odd-pos-int,
:element {:id :field-of-odd-pos-int,
:type :odd-pos-int,
:gen-f #object[diction.core$wrap_gen_f$fn__13635
0x1cde4602
"diction.core$wrap_gen_f$fn__13635@1cde4602"],
:valid-f #object[diction.demo$validate_odd_pos_int
0x640ed2e7
"diction.demo$validate_odd_pos_int@640ed2e7"]}},
:msg "Value '34.3' for field :field-of-odd-pos-int needs to be an integer."}
{:id :field-of-odd-pos-int,
:v 34.3,
:entry {:id :field-of-odd-pos-int,
:element {:id :field-of-odd-pos-int,
:type :odd-pos-int,
:gen-f #object[diction.core$wrap_gen_f$fn__13635
0x1cde4602
"diction.core$wrap_gen_f$fn__13635@1cde4602"],
:valid-f #object[diction.demo$validate_odd_pos_int
0x640ed2e7
"diction.demo$validate_odd_pos_int@640ed2e7"]}},
:msg "Value '34.3' for field :field-of-odd-pos-int needs to be odd."}]
(diction/explain :field-of-odd-pos-int "meh")
;=>
[{:id :field-of-odd-pos-int,
:v "meh",
:entry {:id :field-of-odd-pos-int,
:element {:id :field-of-odd-pos-int,
:type :odd-pos-int,
:gen-f #object[diction.core$wrap_gen_f$fn__13635
0x1cde4602
"diction.core$wrap_gen_f$fn__13635@1cde4602"],
:valid-f #object[diction.demo$validate_odd_pos_int
0x640ed2e7
"diction.demo$validate_odd_pos_int@640ed2e7"]}},
:msg "Value 'meh' for field :field-of-odd-pos-int needs to be an integer."}]
(diction/lookup :field-of-odd-pos-int)
; =>
{:id :field-of-odd-pos-int,
:element {:id :field-of-odd-pos-int,
:type :odd-pos-int,
:gen-f #object[diction.core$wrap_gen_f$fn__13635
0x1cde4602
"diction.core$wrap_gen_f$fn__13635@1cde4602"],
:valid-f #object[diction.demo$validate_odd_pos_int
0x640ed2e7
"diction.demo$validate_odd_pos_int@640ed2e7"]}}
Validation Rules
;;;; Rules ================================================================
;; defines a rule function that validates that if an item has a :tags field
;; then if also must have a :badge field
;; NOTE: dicton element generators are NOT compliant with external validation rule
;; functions
(defn rule-item-if-tags-then-badge-required
[value entry validation-rule context]
(when (:tags value)
(when-not (:badge value)
[{:id (:element-id validation-rule)
:rule-id (:id validation-rule)
:msg (str "Item must have a :badge field if the :tags field is present.")}])))
(validation-rule! ::item ::rule-item-if-tags-then-badge rule-item-if-tags-then-badge-required)
Validation rules are only applied in the explain-all and valid-all? function calls.
(explain ::item {:id "da97d0b6-9c1d-495b-5367-c23da019dc01",
:label "EXJIJS",
:unit-count 33,
:unit-cost 75.0581280630676,
:badgex :VHObB,
:tags ["8dsdfasdfs" "y6n" "q4oH" "+ bb" "ny7LZO6" "ru" "F4vi8nNSo5o"],
:description "19Ti7Ekh6wassjon3RJ=jmBhsX-bLygGAy6pcGl3F6KM3",
:additional-charges {:fees [{:label "Ut",
:fee-upcharge 193.5841860568779,
:cost-basis :flat,
:description "HwGpoD7o96pGsM7N2mqq7To2fHfq7=ekjHxO+"}
{:label "fD7AXP",
:fee-upcharge 350.57367256651224,
:cost-basis :per-unit, ;; bad cost-basis enum
:description "PswklPNDFbJS0l9QrBCnpT6I=PEM41Hq5B9lua"}
{:label "",
:fee-upcharge 286.83015039087127,
:cost-basis :flat}]}})
;=>
nil
(explain-all ::item {:id "da97d0b6-9c1d-495b-5367-c23da019dc01",
:label "EXJIJS",
:unit-count 33,
:unit-cost 75.0581280630676,
:badgex :VHObB,
:tags ["8dsdfasdfs" "y6n" "q4oH" "+ bb" "ny7LZO6" "ru" "F4vi8nNSo5o"],
:description "19Ti7Ekh6wassjon3RJ=jmBhsX-bLygGAy6pcGl3F6KM3",
:additional-charges {:fees [{:label "Ut",
:fee-upcharge 193.5841860568779,
:cost-basis :flat,
:description "HwGpoD7o96pGsM7N2mqq7To2fHfq7=ekjHxO+"}
{:label "fD7AXP",
:fee-upcharge 350.57367256651224,
:cost-basis :per-unit, ;; bad cost-basis enum
:description "PswklPNDFbJS0l9QrBCnpT6I=PEM41Hq5B9lua"}
{:label "",
:fee-upcharge 286.83015039087127,
:cost-basis :flat}]}})
;=>
[{:id :diction.example/item,
:rule-id :diction.example/rule-item-if-tags-then-badge,
:msg "Item must have a :badge field if the :tags field is present."}]
Generative Function Tests
Generative function testing uses function! to register functions to
test generatively leveraging the diction data elements.
(function! :function-id
function
[:diction-element-id-arg1 :id-arg2 :id-argn]
:diction-element-id-result)
The test-function and test-all-functions are used to run generatively function testing.
If the test-function or test-all-functions all pass, then nil is returned.
Otherwise, a vector of failed message maps.
;;;; Functions ===========================================================
(defn sum-long-and-double
[l d]
(str (+ l d)))
(long! ::arg-long)
(double! ::arg-double)
(string! ::result-string)
;; registers function for generative testing
;; (testing if generated element parameters result in expected diction
;; element type)
(function! :sum-long-and-double
sum-long-and-double
[::arg-long ::arg-double]
::result-string)
;; tests function :sum-long-double generatively 999 times
(test-function :sum-long-double 999)
;=>
nil
;; tests function :sum-long-double generatively default
;; number of times (100)
(test-function :sum-long-double)
;=>
nil
Grooming
Grooming allows for the removal of all unregistered fields/elements from diction elements.
Although the grooming is most useful for maps/documents/entities, grooming a values of a registered diction element should return the diction element value.
If the groom returns nil, an error might have occurred and a full explain
should be done on the value.
If the groom returns a non-nil value, that is the groomed value for that element.
(groom ::item {:id "a5965cda-b465-448f-45fa-c90779c32508",
:label "SaJ",
:unit-count 393,
:unit-cost 52.968335308020826,
:badge :OEEbI,
:foo :bar ; should be groomed away
:tags ["t1"],
:description "MO6FX5CIIeadb_6 xNCSKBRSsQ82obEi",
:additional-charges {:fees [{:label "_5LkD1",
:boo :meh ; should be groomed away
:fee-upcharge 498.3408668249411, :cost-basis :per-unit}
{:label "MLLcqP", :fee-upcharge 65.00989414713105, :cost-basis :flat}
{:label "_UL",
:foo :baz ; should be groomed away
:fee-upcharge 778.3347611327353, :cost-basis :flat}
{:label "",
:fee-upcharge 300.638425007598,
:cost-basis :flat,
:description "tj7YXJq_4qg_k4ceKdfKi_mE9FSvDsdhVSXBUXQTMY dcsqZ2OD6Ilti-J s7FezsRVSYsqAiAuQVFp=1Ily1G_eEB2Vb6yBEOkeL1gjH37QqfX0j=s3F657qaN9hDZ"}],
:percents [{:label "orl",
:percent-upcharge 0.5618874139197575,
:meh :bah ; should be groomed away
:cost-basis :per-unit,
:description "Ep2Csq757fOPsr1vV-pD21663Aj2tH j58x0izg6Y25"}
{:label "+7+t", :percent-upcharge 0.0734059122348183, :cost-basis :per-unit}]}}
)
;=>
{:description "MO6FX5CIIeadb_6 xNCSKBRSsQ82obEi",
:tags ["t1"],
:label "SaJ",
:id "a5965cda-b465-448f-45fa-c90779c32508",
:unit-cost 52.968335308020826,
:unit-count 393,
:badge :OEEbI,
:additional-charges {:fees [{:fee-upcharge 498.3408668249411, :cost-basis :per-unit, :label "_5LkD1"}
{:fee-upcharge 65.00989414713105, :cost-basis :flat, :label "MLLcqP"}
{:fee-upcharge 778.3347611327353, :cost-basis :flat, :label "_UL"}
{:description "tj7YXJq_4qg_k4ceKdfKi_mE9FSvDsdhVSXBUXQTMY dcsqZ2OD6Ilti-J s7FezsRVSYsqAiAuQVFp=1Ily1G_eEB2Vb6yBEOkeL1gjH37QqfX0j=s3F657qaN9hDZ",
:fee-upcharge 300.638425007598,
:cost-basis :flat,
:label ""}],
:percents [{:description "Ep2Csq757fOPsr1vV-pD21663Aj2tH j58x0izg6Y25",
:percent-upcharge 0.5618874139197575,
:cost-basis :per-unit,
:label "orl"}
{:percent-upcharge 0.0734059122348183, :cost-basis :per-unit, :label "+7+t"}]}}
Decoration Rules
Decoration rules may be defined and leveraged to centralize decoration/normalization of data shapes.
Decoration is not aware of the validation and/or generation of the diction elements, and validation/generation is not aware of decorated element values.
This means that if you decorate an element value and then try to validate and your validation does not take into account the decorations, then the validation might fail.
Note that a diction element may have zero or many decoration rules.
(defn decoration-rule-calc-item-inventory-retail-worth
[v entry rule ctx]
(assoc v
:inventory-retail-worth
(* (get v :unit-count 0) (get v :unit-cost 0.0))))
(defn decoration-rule-inventory-str
[v entry rule ctx]
(assoc v :inventory-str (str "There are "
(get v :unit-count "unk")
" "
(get v :label "unk")
" item(s) [" (get v :id "-") "] "
"with an inventory retail value of "
(* (get v :unit-count 0) (get v :unit-cost 0.0))
" USD.")))
;; first argument is the diction element id for the decoration rule
;; second argument is the decoration rule id
;; third argument is the decoration rule function
;; optional fourth function is the context of the decoration rule call
(diction/decoration-rule! ::item
:calculate-item-inventory-retail-worth
decoration-rule-calc-item-inventory-retail-worth)
(diction/decoration-rule! ::item
:item-inventory-str
decoration-rule-inventory-str)
;; decorate call is simply the diction element id and the diction element value
(diction/decorate ::item {:id "ba2b6e0c-3091-4ff2-34b2-91483a4aaabf",
:label "4ntL0R",
:unit-count 80,
:unit-cost 25.00,
:badge :GmSh})
;=>
{:id "ba2b6e0c-3091-4ff2-34b2-91483a4aaabf",
:label "4ntL0R",
:unit-count 80,
:unit-cost 25.0,
:badge :GmSh,
:inventory-retail-worth 2000.0,
:inventory-str "There are 80 4ntL0R item(s) [ba2b6e0c-3091-4ff2-34b2-91483a4aaabf] with an inventory retail value of 2000.0 USD."}
Metadata Query
Metadata queries allow for the querying/searching of diction elements with certain meta data, such as protected personal identifiable identification (PII) elements, protected HIPAA/medical, sensitive elements, auditable elements, and any other type of metadata domains need to include to work intelligently with their data.
(int! ::ans {:meta {:pii true :rank 3 :label "answer" :foo "bars"}})
(string! ::mercy {:meta {:pii true :rank 2 :label "merciful" :desc "what?"}})
(string! ::wonk {:meta {:pii false :rank 3 :label "wonky"}})
(double! ::tau {:meta {:label "tau vs. pi" :rank 4 :pii true :foo false}})
(meta-query {:query {:pii true}})
;=>
[{:id :diction.example/tau,
:element {:id :diction.example/tau,
:type :double,
:gen-f #object[diction.core$wrap_gen_f$fn__10837 0x23ed19f7 "diction.core$wrap_gen_f$fn__10837@23ed19f7"],
:valid-f #object[clojure.core$partial$fn__5563 0x5fe95d87 "clojure.core$partial$fn__5563@5fe95d87"],
:parent-id :diction/double,
:meta {:label "tau vs. pi", :rank 4, :pii true, :foo false}}}
{:id :diction.example/mercy,
:element {:id :diction.example/mercy,
:type :string,
:min 0,
:max 64,
:gen-f #object[diction.core$wrap_gen_f$fn__10837 0x5cd8f6e0 "diction.core$wrap_gen_f$fn__10837@5cd8f6e0"],
:valid-f #object[clojure.core$partial$fn__5565 0x9c15f50 "clojure.core$partial$fn__5565@9c15f50"],
:parent-id :diction/string,
:meta {:pii true, :rank 2, :label "merciful", :desc "what?"}}}
{:id :diction.example/ans,
:element {:id :diction.example/ans,
:type :int,
:gen-f #object[diction.core$wrap_gen_f$fn__10837 0x4fa60fee "diction.core$wrap_gen_f$fn__10837@4fa60fee"],
:valid-f #object[clojure.core$partial$fn__5563 0x76c7641a "clojure.core$partial$fn__5563@76c7641a"],
:parent-id :diction/int,
:meta {:pii true, :rank 3, :label "answer", :foo "bars"}}}]
;; by default, meta-query returns the matching diction entries
;; but a mask of meta fields may be return instead using `:mask` in the query-map
(meta-query {:query {:pii true} :mask [:pii :label]})
;=>
[{:id :diction.example/tau, :pii true, :label "tau vs. pi"}
{:id :diction.example/mercy, :pii true, :label "merciful"}
{:id :diction.example/ans, :pii true, :label "answer"}]
;; :query fields may be literals, or functions that take a single
;; argument (the value of the field/key) and returns truthy/false
(meta-query {:mask [:label :pii :rank] :query {:pii true :rank #(> % 2)}})
;=>
[{:id :diction.example/tau, :label "tau vs. pi", :pii true, :rank 4}
{:id :diction.example/ans, :label "answer", :pii true, :rank 3}]
;; :query-f allows for a function that takes the `meta` of
;; the candidate entry and returns truthy/falsey
(meta-query {:mask [:label :pii :rank :foo] :query-f #(some? (:foo %))})
;=>
[{:id :diction.example/tau, :label "tau vs. pi", :pii true, :rank 4, :foo false}
{:id :diction.example/ans, :label "answer", :pii true, :rank 3, :foo "bars"}]
;; and all three (3) query mechanism may be used at once
(meta-query {:mask [:label :pii :rank :foo] :query {:pii true :rank #(> % 3)} :query-f #(some? (:foo %))})
;=>
[{:id :diction.example/tau, :label "tau vs. pi", :pii true, :rank 4, :foo false}]
HTTP Request Validations
; (compojure.core/wrap-routes validate-payload) ; (compojure.core/wrap-routes validate-parameters)
Wrapping Handlers
In the handler ns of your application, wrap the routes with diction.http
functions validate-payload and validate-parameters.
(ns something.handler
(:require [compojure.core :refer [wrap-routes]]
[diction.http :as diction-http]))
;...
(wrap-routes diction-http/validate-payload)
(wrap-routes diction-http/validate-parameters)
;...
Payload
(payload-validation-routes!
"/item" {:post :item-payload-document-element-id}
"/customer" {:post :customer-payload-element-id})
Parameters
(parameter-validation-routes!
"/item" {:get :item-parameters-document-element-id}
"/customer" {:post :customer-parameters-element-id})
Validation Failed Handler Function
The @diction.http/bad-request-f is the function with a single body
argument for handling validation failure.
To reset the bad-request-f atom:
(bad-request-f! (fn [body] [:bad body {:message "Bad validation stuff"}]))
The default bad request function simple returns the body with a 400
status.
Sample Validation Failure
{:status 400,
:body {:error "Payload validation failed for element ':diction/foobar'. [failure count=1]",
:body {:foo "this", :ans true},
:element :diction/foobar,
:failures [{:id :diction/ans,
:entry {:id :diction/ans,
:element {:id :diction/ans,
:type :long,
:gen-f #object[diction.core$wrap_gen_f$fn__5023
0x4728a22b
"diction.core$wrap_gen_f$fn__5023@4728a22b"],
:valid-f #object[clojure.core$partial$fn__5826
0x39e086dc
"clojure.core$partial$fn__5826@39e086dc"],
:parent-id :diction/long,
:meta {:sensible-values [41 42 43 99]}}},
:v true,
:msg "Failed ':diction/ans': value 'true' is not a long number.",
:parent-element-id [:diction/foobar]}]}}
Guard Functions
Guard functions may wrap other functions, like storing to the db or calling other services or functions, to validate one of the arguments in the target wrapped function against a diction element.
[diction.guard :refer [guard guard-fail-f!]]
(guard element-id wrapped-function-f)
(guard element-id extract-from-args-list-f wrapped-function-f )
The guard/guard is a higher-order function that returns a wrapper function
with validation against any single argument (defaults to the first argument
of the wrapped function call).
Example
(diction/string! :meh)
(defn meh!
[meh]
(println "success:" meh))
(guard-fail-f! (fn [eid wrapped-f v value-extract-f failures & args]
(println :failed-validation :eid eid :v v :failures failures :args args)))
(def guarded-meh! (guard :meh meh!))
(guarded-meh! "this is a string meh so good")
;=> success: this is a string meh so good
(guarded-meh! 42)
;=> :failed-validation :eid :meh :v 42 :failures [{:id :meh, :entry {:id :meh, :element {:id :meh, :type :string, :min 0, :max 64, :gen-f #object[diction.core$wrap_gen_f$fn__4362 0x57c94c2d diction.core$wrap_gen_f$fn__4362@57c94c2d], :valid-f #object[clojure.core$partial$fn__5828 0x5d4c4bd3 clojure.core$partial$fn__5828@5d4c4bd3], :parent-id :diction/string}}, :v 42, :msg Failed ':meh': value '42' is not a string.}] :args (42)
(defn meh2!
[id meh]
(println "success2: " :id id :meh meh))
(def guarded-meh2! (guard :meh second meh2!)) ; note the `second` args extract function
(guarded-meh2! 42 "string")
;=> success2: :id 42 :meh string
(guarded-meh2! 42 34)
;=> :failed-validation :eid :meh :v 34 :failures [{:id :meh, :entry {:id :meh, :element {:id :meh, :type :string, :min 0, :max 64, :gen-f #object[diction.core$wrap_gen_f$fn__4362 0x57c94c2d diction.core$wrap_gen_f$fn__4362@57c94c2d], :valid-f #object[clojure.core$partial$fn__5828 0x5d4c4bd3 clojure.core$partial$fn__5828@5d4c4bd3], :parent-id :diction/string}}, :v 34, :msg Failed ':meh': value '34' is not a string.}] :args (42 34)
Documentation
Markdown
Diction provides a diction.documentation namespace that can convert the current
Diction data dictionary into markdown.
(diction.documentation/->markdown)
The markdown may be spit into a text file:
(spit "data-dictionary.md" (diction.documentation/->markdown))
When added to a Github wiki page, the generated data dictionary markdown is
navigatable, meaning that a user may click on related data elements
to view the information for those related data elements (such as a data
element referenced by a document will have a link to the referencing
document data element).
HTML
Diction provides a diction.documentation namespace that can convert the current
Diction data dictionary into a navigatable HTML page.
(diction.documentation/->html)
(diction.documentation/->html {:title "Acme Documentation"})
The HTML may be spit into a text file:
(spit "data-dictionary.html" (diction.documentation/->html {:title "Acme Documentation"))
The (diction.documentation/->html ctx) function call allows for some optional settings.
ctx map keys:
header: HTML string with header tags; freeform HTMLtitle: Title string of the generated HTML page HEAD and top header of HTMLstylesheet: CSS linkstyle: raw CSS textsuppress-style: iftrue, will suppress default CSSstart-body: HTML string with tags at the top of the BODY tag; freeform HTMLend-body: HTML string with tags at the bottom of the BODY tag; freeform HTML
Hiccup
Diction provides a diction.documentation namespace that can convert the current
Diction data dictionary into an HTML hiccup vector.
(diction.documentation/->hiccup)
(diction.documentation/->hiccup {:title "Acme Documentation"})
The (diction.documentation/->hiccup ctx) function call allows for some optional settings.
ctx map keys:
header: HTML string with header tags; freeform HTMLtitle: Title string of the generated HTML page HEAD and top header of HTMLstylesheet: CSS linkstyle: raw CSS textsuppress-style: iftrue, will suppress default CSSstart-body: HTML string with tags at the top of the BODY tag; freeform HTMLend-body: HTML string with tags at the bottom of the BODY tag; freeform HTML
License
Copyright © 2020 SierraLogic LLC
Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.
Can you improve this documentation? These fine people already did:
Greg Seaton & gseatonEdit on GitHub
cljdoc is a website building & hosting documentation for Clojure/Script libraries
× close