Metazoa

The Metazoa library provides an extensible API for viewing, testing, searching, and querying Clojure metadata and includes several metadata providers that take full advantage of those tools: executable examples, function tables, interactive tutorials at the REPL, and structured documents that can contain other metadata provider values as nodes.

;; Git dep
{dev.glossa/metazoa
 {:git/url "https://gitlab.com/glossa/metazoa.git"
  :git/tag "v0.1.281"
  :git/sha "c525e20ae51639cd8492a3021d704e9cf5ccff82"}}

;; Maven dep at https://clojars.org/dev.glossa/metazoa
{dev.glossa/metazoa {:mvn/version "0.1.281"}}

;; Exclusions of optional deps that are included by default:
:exclusions [cljfmt/cljfmt datascript/datascript metosin/malli org.apache.lucene/lucene-core org.apache.lucene/lucene-queryparser]

Metazoa is best learned at the REPL:

(require '[glossa.metazoa :as meta])
(meta/help)

Introductory video on YouTube

Article on Motivation and Background

Introduction

• meta/view View metadata in rich, interesting ways.
• meta/check Check that your metadata is still valid via testing.
• meta/search Search your project's metadata.
• meta/query Query your project's metadata with Datalog queries.

These functions rely on a set of multimethods defined in [glossa.metazoa.api :as meta.api] that form the Metadata Provider API. A metadata provider implements one or more of these multimethods and can be identified by its dispatch value.

The built-in metadata providers are:

• :glossa.metazoa/doc
• :glossa.metazoa/example
• :glossa.metazoa/fn-table
• :glossa.metazoa/tutorial

If you have a REPL ready, try running (meta/help) to get started—there's an interactive tutorial waiting for you.

In all the sections that follow:

• IMeta refers to an instance of clojure.lang.IMeta, which is a value that can store Clojure metadata.
• ;; [out] indicates characters printed via *out*
• ;; [err] indicates characters printed via *err*
• #_=> indicates the evaluated value of a Clojure expression.

Viewing

The meta/view function has been designed for REPL use, printing to *out* with leading semicolons and a narrow column width. Review the following examples to gain an intuition of how meta/view works:

;; What metadata providers are available on this namespace?
(meta/view 'glossa.metazoa)
#_=> (:glossa.metazoa/doc :glossa.metazoa/tutorial)

;; View a var's example
(meta/view #'clojure.core/name ::meta/example)
;; [out] 
;; [out] ;; The `name` function generously converts unqualified symbols, keywords, and
;; [out] ;; strings to strings.
;; [out] (= (name 'alpha) (name :alpha) (name "alpha"))
;; [out] #_=> true
;; [out] 
;; [out] ;; If a qualified symbol or keyword, `name` returns the second part, without the
;; [out] ;; namespace included.
;; [out] (= "bar" (name 'foo/bar) (name :foo/bar))
;; [out] #_=> true
#_=> #'clojure.core/name

;; View a function table
(meta/view #'clojure.core/max ::meta/fn-table)
;; [out] 
;; [out]   OR   0   1  
;; [out] ----- --- ----
;; [out]    0   0   1  
;; [out]    1   1   1  
#_=> #'clojure.core/max

;; View a standalone metadata provider value, useful while developing it
(meta/view
 (meta/example {:ns *ns*, :code '(map max [0 0 1 1] [0 1 0 1])}))
;; [out] 
;; [out] (map max [0 0 1 1] [0 1 0 1])
;; [out] #_=> (0 1 1 1)
#_=> []

Themeta/view function will attempt to resolve a symbol to a namespace or var (as seen in the first example).

While printing returns nil in Clojure, the meta/view function returns the given IMeta, so that you can thread calls to meta/viewand other Metazoa functions that take an IMeta instance.

There are two multimethods that underlie meta/view:

• meta.api/render-metadata Return a value that can be trivially printed.
• meta.api/view-metadata Provide a custom view experience beyond simple printing.

Most metadata providers need only implement meta.api/render-metadata, because meta/view prints its return value if a separate meta/view-metadata implementation is not found. The :glossa.metazoa/tutorial provider implements its own meta/view-metadata to provide an interactive, in-REPL tutorial player.

Checking/Testing

If you've taken the time to adorn your codebase with rich metadata, you should take the time to ensure it remains up-to-date. Metazoa supports this through its checking and testing story.

;; Add `:expected` to your ::meta/example metadata:
(::meta/example (meta #'clojure.core/max))
#_=> {:code (max 5 -5 10 0),
#_=>  :expected 10,
#_=>  :ns #object[clojure.lang.Namespace 0x41058886 "glossa.metazoa-meta"]}

;; And you'll get meaningful check output:
(meta/check #'clojure.core/max ::meta/example)
#_=> [{:code (max 5 -5 10 0),
#_=>   :expected 10,
#_=>   :actual-out "",
#_=>   :actual-err "",
#_=>   :actual 10}]

Evaluate (meta/check imeta k) to see a data representation expressing your metadata's validity; run (meta/test-imeta imeta k) to assert the same validity using clojure.test. The separation of functional checking and side-effecting testing allows users to wire up Metazoa with testing libraries other than clojure.test if desired.

Both meta/check and meta/test-imeta will accept just an IMeta, in which case all of the metadata entries that have a meta.api/check-metadata implementation will be exercised.

Run (meta/test-imetas) inside a clojure.test/deftest form to test all metadata on all IMetas on the classpath.

Searching

Optional Dependencies, included by default:

• Apache Lucene, latest tested artifacts:
  • org.apache.lucene/lucene-core {:mvn/version "8.9.0"}
  • org.apache.lucene/lucene-queryparser {:mvn/version "8.9.0"}

Metazoa's meta/search function allows you to search your code base's metadata using Lucene queries:

;; Search for 'source'
(take 5 (meta/search "source"))
;; [out] Indexing metadata for full-text search...
#_=> (#'datascript.query/*implicit-source*
#_=>  #'datascript.parser/with-source
#_=>  #'clojure.tools.reader.reader-types/log-source
#_=>  #'datascript.query/source?
#_=>  #'datascript.parser/source)

;; How many results was that?
(count (meta/search "source"))
#_=> 30

;; If 30, that's the default limit. How many really?
(:total-hits (meta (meta/search "source")))
#_=> 49

;; You can specify `:num-hits` or `:limit`
(meta/search {:query "source", :num-hits 3})
#_=> [#'datascript.query/*implicit-source*
#_=>  #'datascript.parser/with-source
#_=>  #'clojure.tools.reader.reader-types/log-source]

;; Exclude certain namespace patterns
(count (meta/search "name:source AND -ns:cider.*"))
#_=> 11

;; Limit results to macros
(meta/search "name:source AND -ns:cider.* AND macro:true")
#_=> [#'clojure.tools.reader.reader-types/log-source]

;; How many public forms are in namespaces prefixed with 'clojure.' ?
(-> (meta/search "ns:clojure.*") meta :total-hits)
#_=> 1001

;; Which namespaces does that include?
(meta/search "id:clojure.* AND imeta-type:clojure.lang.Namespace")
#_=> [#object[clojure.lang.Namespace 0x70713e9d "clojure.tools.reader.impl.utils"]
#_=>  #object[clojure.lang.Namespace 0x77aa566a "clojure.stacktrace"]
#_=>  #object[clojure.lang.Namespace 0x1ec2caf2 "clojure.tools.cli"]
#_=>  #object[clojure.lang.Namespace 0x2d30d676 "clojure.test.check.results"]
#_=>  #object[clojure.lang.Namespace 0x5063fdd "clojure.test"]
#_=>  #object[clojure.lang.Namespace 0x20ba2011 "clojure.core.server"]
#_=>  #object[clojure.lang.Namespace 0x2827fbae "clojure.core.specs.alpha"]
#_=>  #object[clojure.lang.Namespace 0x38f0bba "clojure.spec.alpha"]
#_=>  #object[clojure.lang.Namespace 0x374334c1 "clojure.set"]
#_=>  #object[clojure.lang.Namespace 0x46883090 "clojure.string"]
#_=>  #object[clojure.lang.Namespace 0x51e55964 "clojure.tools.reader.default-data-readers"]
#_=>  #object[clojure.lang.Namespace 0x2acc5a39 "clojure.template"]
#_=>  #object[clojure.lang.Namespace 0x487969d "clojure.core"]
#_=>  #object[clojure.lang.Namespace 0x30c2a2d "clojure.tools.reader.reader-types"]
#_=>  #object[clojure.lang.Namespace 0x78596d51 "clojure.walk"]
#_=>  #object[clojure.lang.Namespace 0x308cfdf1 "clojure.main"]
#_=>  #object[clojure.lang.Namespace 0x887b603 "clojure.data"]
#_=>  #object[clojure.lang.Namespace 0x6a38a87c "clojure.tools.reader.edn"]
#_=>  #object[clojure.lang.Namespace 0x5938638f "clojure.edn"]
#_=>  #object[clojure.lang.Namespace 0xe0c6e2c "clojure.java.io"]
#_=>  #object[clojure.lang.Namespace 0x3b0ee640 "clojure.test.check.random"]
#_=>  #object[clojure.lang.Namespace 0x145ac215 "clojure.pprint"]
#_=>  #object[clojure.lang.Namespace 0xe64ef0a "clojure.java.classpath"]
#_=>  #object[clojure.lang.Namespace 0x63f48de4 "clojure.test.check.rose-tree"]
#_=>  #object[clojure.lang.Namespace 0x2f045b3c "clojure.tools.reader"]
#_=>  #object[clojure.lang.Namespace 0x6d0c6ea3 "clojure.zip"]]

Note: Results are limited to 30 hits by default. Use the map-based query and specify :num-hits or :limit (see examples above) to adjust this.

In the first example, how did it match the namespace glossa.metazoa? All of your code base's metadata is indexed, not just names of vars and namespaces. For Clojure vars, Metazoa's search indexing includes only public ones by default, but you have the option to provide a collection of IMetas to be indexed yourself as follows:

;; This indexes _all_ vars, not just public ones:
(meta/reset-search
 (meta.api/find-imetas (fn [ns] (conj ((comp vals ns-interns) ns) ns))))

The metadata map of each IMeta in your code is indexed as a separate Lucene document. Each metadata entry of each IMeta is indexed as a separate field in those Lucene documents. By default, string metadata values are indexed as full text fields and most others are indexed as simple text fields (using str of the value). Every Lucene document has a imeta-symbol and imeta-type fields, which are the fully-qualified identifier and the type of the source IMeta, as well as a imeta-value-type field which is the type of the underlying value contained by an IMeta that is a Clojure var.

All fully-qualified idents have their / character replaced with _ to be acceptable for Lucene query syntax. If a document fails to index, a message is printed to *err* but the indexing process will proceed to index as many IMetas as possible.

Metadata providers can implement custom search indexing behavior. To customize how your metadata is indexed, implement the meta.api/index-for-search multimethod and return a map as follows:

;; Option A: Supply a function expecting a Lucene Document, use Lucene API to
;; index your field.
{:lucene
 {:index-fn
  (fn
   [doc]
   (.add doc (TextField. "your-field" "your-value" Field$Store/NO)))}}

;; Option B: Supply a map specifying how to index your metadata value.         
;; Currently limited in expressivity.
{:lucene
 {:field :text-field,
  :stored? false,
  :value "Some custom stringification of your metadata value."}}

Querying

Optional Dependencies, included by default:

• DataScript, latest tested artifacts:
  • datascript/datascript {:mvn/version "1.2.8"}

Metazoa's meta/query function allows you to query your code base's metadata using Datalog queries:

;; What was added to Clojure core in version 1.4?
(sort
 (meta/query
  '[:find [?name ...]
    :in $ ?ns ?added
    :where
    [?e :ns ?ns]
    [?e :name ?name]
    [?e :added ?added]]
  (the-ns 'clojure.core)
  "1.4"))
#_=> (*compiler-options*
#_=>  *data-readers*
#_=>  default-data-readers
#_=>  ex-data
#_=>  ex-info
#_=>  filterv
#_=>  mapv
#_=>  reduce-kv)

;; How many public vars lack a :doc string?
(meta/query
 '[:find [(count ?e)]
   :where
   [?e :ns]
   (not [?e :doc])])
#_=> [956]

;; How many of those are functions?
(meta/query
 '[:find [(count ?e)]
   :where
   [?e :ns]
   [?e ::meta/imeta-value ?value]
   [(clojure.core/fn? ?value)]
   (not [?e :doc])])
#_=> [777]

;; What are the important magic numbers in my code base?
(meta/query
 '[:find ?imeta ?value
   :in $ package
   :where
   [?e :ns ?ns]
   [(package ?ns)]
   [?e ::meta/imeta-value ?value]
   [(clojure.core/number? ?value)]
   [?e ::meta/imeta ?imeta]]
 (fn package [ns] (str/starts-with? (str (ns-name ns)) "glossa.")))
#_=> #{[#'glossa.metazoa.query.datascript/temp-id -3037]
#_=>   [#'glossa.metazoa.search/default-num-hits 30]
#_=>   [#'glossa.metazoa.fmt/default-display-width 80]
#_=>   [#'glossa.weave/default-display-width 80]}

The metadata map of each IMeta in your code is transacted to the DataScript database as an entity. The Metazoa library transacts :glossa.metazoa/imeta, :glossa.metazoa/imeta-symbol, and :glossa.metazoa/imeta-type attributes which store the IMeta itself, the qualified symbol identifier of the IMeta, and the type of the IMeta, respectively. In addition, if the IMeta is a Clojure var, the underlying value and its type are transacted as :glossa.metazoa/imeta-value and :glossa.metazoa/imeta-value-type.

Built-in Providers

For now, I suggest reading through the glossa.metazoa-meta namespace to see Metazoa's metadata providers in action, including the source for the main tutorial and the source of this README document.

You can find Malli schemas under the glossa.metazoa.provider namespaces that show what each provider expects.

The ::meta/doc provider expects a Weave document value, extended to support nodes of ::meta/example and ::meta/fn-table, so that a single ::meta/doc can serve as a cohesive document with prose and testable code examples interwoven.

Optional Dependencies

Here are all of Metazoa's optional-but-included-by-default dependencies:

• cljfmt/cljfmt {:mvn/version "0.8.0"}
• datascript/datascript {:mvn/version "1.2.8"}
• metosin/malli {:mvn/version "0.6.1"}
• org.apache.lucene/lucene-core {:mvn/version "8.9.0"}
• org.apache.lucene/lucene-queryparser {:mvn/version "8.9.0"}

You can exclude these via :exclusions in your project dependencies if you do not want to use the Metazoa features that rely on them.

Metazoa's searching and querying functions—since they are totally reliant on the third-party dependencies and are intended as interactive tools—will throw an exception if you attempt to use them without their dependencies. Code formatting and schema validation functionality supplied by cljfmt and malli respectively will simply be skipped without throwing exceptions.

External Metadata Providers

Official:

• (Planned) :glossa.metazoa/plantuml In its own words: "PlantUML is used to draw...diagrams, using a simple and human readable text description."
• (Planned) :glossa.metazoa/vega-lite Create powerful visualizations of your data or the behavior of your functions using Vega-Lite as your "high-level grammar of interactive graphics"

Community:

• Please open an issue on this project and tag it with community-metadata-provider so it can be considered for inclusion here.

License

Copyright © Daniel Gregoire, 2021

THE ACCOMPANYING PROGRAM IS PROVIDED UNDER THE TERMS OF THE ECLIPSE PUBLIC LICENSE ("AGREEMENT"). ANY USE, REPRODUCTION OR DISTRIBUTION OF THE PROGRAM CONSTITUTES RECIPIENT'S ACCEPTANCE OF THIS AGREEMENT.