Glossa: Metazoa

The Metazoa library provides functions for viewing, searching, querying, and testing Clojure metadata and includes several metadata providers that take full advantage of those tools.

;; Git dep
{dev.glossa/metazoa
 {:git/url "https://gitlab.com/glossa/metazoa.git"
  :git/tag "v0.1.264"
  :git/sha "df2a1df5b093465160426bfcb60cb6365fc1cd47"}}

;; Maven dep at https://clojars.org/dev.glossa/metazoa
{dev.glossa/metazoa {:mvn/version "0.1.264"}}

;; Exclusions of optional deps that are included by default:
:exclusions [cljfmt/cljfmt datascript/datascript metosin/malli org.apache.lucene/lucene-core org.apache.lucene/lucene-queryparser]

Metazoa is best learned at the REPL:

(require '[glossa.metazoa :as meta])
(meta/help)

Introductory video on YouTube

Introduction

• meta/view View metadata in rich, interesting ways.
• meta/search Search your project's metadata.
• meta/query Query your project's metadata with Datalog queries.
• meta/check Check that your metadata is still valid via testing.

These functions rely on a set of multimethods that form the Metadata Provider API. A metadata provider implements one or more of these multimethods (described in detail later in this document) and can be identified by its dispatch value.

The built-in metadata providers are:

• :glossa.metazoa/doc
• :glossa.metazoa/example
• :glossa.metazoa/fn-table
• :glossa.metazoa/tutorial

If you have a REPL ready, try running (meta/help) to get started—there's an interactive tutorial waiting for you.

In this document, first we'll cover how to use the core functions. Then we'll take a look at the built-in metadata providers. Finally, we'll review the Metadata Provider API itself, so that you can implement your own custom providers.

In all the sections that follow:

• IMeta refers to an instance of clojure.lang.IMeta, which is a value that can store Clojure metadata.
• ;; [out] indicates characters printed via *out*
• ;; [err] indicates characters printed via *err*
• #_=> indicates the evaluated value of a Clojure expression.

Viewing

The meta/view function has been designed for REPL use, printing to *out* with leading semicolons and a narrow column width. Review the following examples to gain an intuition of how meta/view works:

;; What metadata providers are available on this namespace?
(meta/view 'glossa.metazoa)
#_=> (:glossa.metazoa/doc :glossa.metazoa/tutorial)

;; View an example
(meta/view #'clojure.core/name ::meta/example)
;; [out] 
;; [out] ;; The `name` function generously converts unqualified symbols, keywords, and
;; [out] ;; strings to strings.
;; [out] (= (name 'alpha) (name :alpha) (name "alpha"))
;; [out] #_=> true
;; [out] 
;; [out] ;; If a qualified symbol or keyword, `name` returns the second part
;; [out] ;; (the...name).
;; [out] (= "bar" (name 'foo/bar) (name :foo/bar))
;; [out] #_=> true
#_=> #'clojure.core/name

;; View a function table
(meta/view #'clojure.core/max ::meta/fn-table)
;; [out] 
;; [out]   OR   0   1  
;; [out] ----- --- ----
;; [out]    0   0   1  
;; [out]    1   1   1  
#_=> #'clojure.core/max

Themeta/view function will attempt to resolve a symbol to a namespace or var (as seen in the first example).

While printing returns nil in Clojure, the meta/view function returns the given IMeta, so that you can thread calls to meta/viewand other Metazoa functions that take an IMeta instance.

There are two multimethods that underlie meta/view:

• meta.api/render-metadata Return a value that can be trivially printed.
• meta.api/view-metadata Provide a custom view experience beyond simple printing.

Most metadata providers need only implement meta.api/render-metadata, because meta/view prints its return value if a separate meta/view-metadata implementation is not found. The :glossa.metazoa/tutorial provider implements its own meta/view-metadata to provide an interactive, in-REPL tutorial player.

Searching

Optional Dependencies, included by default.

• Apache Lucene, latest tested artifacts:
  • org.apache.lucene/lucene-core {:mvn/version "8.9.0"}
  • org.apache.lucene/lucene-queryparser {:mvn/version "8.9.0"}

Metazoa's meta/search function allows you to search your code base's metadata using Lucene queries:

;; Search for 'source'
(take 5 (meta/search "source"))
;; [out] Indexing metadata for full-text search...
#_=> (#'datascript.query/*implicit-source*
#_=>  #'datascript.parser/with-source
#_=>  #'clojure.tools.reader.reader-types/log-source
#_=>  #'datascript.query/source?
#_=>  #'datascript.parser/source)

;; How many results was that?
(count (meta/search "source"))
#_=> 30

;; If 30, that's the default limit. How many really?
(:total-hits (meta (meta/search "source")))
#_=> 49

;; You can specify `:num-hits` or `:limit`
(meta/search {:query "source", :num-hits 3})
#_=> [#'datascript.query/*implicit-source*
#_=>  #'datascript.parser/with-source
#_=>  #'clojure.tools.reader.reader-types/log-source]

;; Exclude certain namespace patterns
(count (meta/search "name:source AND -ns:cider.*"))
#_=> 11

;; Limit results to macros
(meta/search "name:source AND -ns:cider.* AND macro:true")
#_=> [#'clojure.tools.reader.reader-types/log-source]

;; How many public forms are in namespaces prefixed with 'clojure.' ?
(-> (meta/search "ns:clojure.*") meta :total-hits)
#_=> 1001

;; Which namespaces does that include?
(meta/search "id:clojure.* AND imeta-type:clojure.lang.Namespace")
#_=> [#object[clojure.lang.Namespace 0x2346dfad "clojure.tools.reader.impl.utils"]
#_=>  #object[clojure.lang.Namespace 0x5162bede "clojure.stacktrace"]
#_=>  #object[clojure.lang.Namespace 0x2b79945f "clojure.tools.cli"]
#_=>  #object[clojure.lang.Namespace 0x596eebdb "clojure.test.check.results"]
#_=>  #object[clojure.lang.Namespace 0x4ed507b0 "clojure.test"]
#_=>  #object[clojure.lang.Namespace 0xa05c4fa "clojure.core.server"]
#_=>  #object[clojure.lang.Namespace 0x6db96289 "clojure.core.specs.alpha"]
#_=>  #object[clojure.lang.Namespace 0x43a77cd6 "clojure.spec.alpha"]
#_=>  #object[clojure.lang.Namespace 0x25290b95 "clojure.set"]
#_=>  #object[clojure.lang.Namespace 0x1af29de "clojure.string"]
#_=>  #object[clojure.lang.Namespace 0x2724ae48 "clojure.tools.reader.default-data-readers"]
#_=>  #object[clojure.lang.Namespace 0x5f9d27b6 "clojure.template"]
#_=>  #object[clojure.lang.Namespace 0x3205ea73 "clojure.core"]
#_=>  #object[clojure.lang.Namespace 0x4a9cd434 "clojure.tools.reader.reader-types"]
#_=>  #object[clojure.lang.Namespace 0x48c865c1 "clojure.walk"]
#_=>  #object[clojure.lang.Namespace 0x7e5e5bf9 "clojure.main"]
#_=>  #object[clojure.lang.Namespace 0x63157033 "clojure.data"]
#_=>  #object[clojure.lang.Namespace 0x4f843511 "clojure.tools.reader.edn"]
#_=>  #object[clojure.lang.Namespace 0x4469a74d "clojure.edn"]
#_=>  #object[clojure.lang.Namespace 0x4864adaa "clojure.java.io"]
#_=>  #object[clojure.lang.Namespace 0x38d87d "clojure.test.check.random"]
#_=>  #object[clojure.lang.Namespace 0x6efa59d6 "clojure.pprint"]
#_=>  #object[clojure.lang.Namespace 0x6720c88b "clojure.java.classpath"]
#_=>  #object[clojure.lang.Namespace 0xa442038 "clojure.test.check.rose-tree"]
#_=>  #object[clojure.lang.Namespace 0x4cf9fa3d "clojure.tools.reader"]
#_=>  #object[clojure.lang.Namespace 0x70713e9d "clojure.zip"]]

Note: Results are limited to 30 hits by default. Use the map-based query and specify :num-hits or :limit (see examples above) to adjust this.

In the first example, how did it match the namespace glossa.metazoa? All of your code base's metadata is indexed, not just names of vars and namespaces.

The metadata map of each IMeta in your code is indexed as a separate Lucene document. Each metadata entry of each IMeta is indexed as a separate field in those Lucene documents. By default, string metadata values are indexed as full text fields and most others are indexed as simple text fields (using str of the value). Every Lucene document has a imeta-symbol and imeta-type fields, which are the fully-qualified identifier and the type of the source IMeta, as well as a imeta-value-type field which is the type of the underlying value contained by an IMeta that is a Clojure var.

All fully-qualified idents have their / character replaced with _ to be acceptable for Lucene query syntax.

Metadata providers, however, can implement custom search indexing behavior. To customize how your metadata is indexed, implement the meta.api/index-for-search multimethod and return a map as follows:

;; Option A: Supply a function expecting a Lucene Document, use Lucene API to
;; index your field.
{:lucene
 {:index-fn
  (fn
   [doc]
   (.add doc (TextField. "your-field" "your-value" Field$Store/NO)))}}

;; Option B: Supply a map specifying how to index your metadata value.         
;; Currently limited in expressivity.
{:lucene
 {:field :text-field,
  :stored? false,
  :value "Some custom stringification of your metadata value."}}

Querying

Optional Dependencies, included by default.

• DataScript, latest tested artifacts:
  • datascript/datascript {:mvn/version "1.2.8"}

Metazoa's meta/query function allows you to query your code base's metadata using Datalog queries:

;; What was added to Clojure core in version 1.4?
(sort
 (meta/query
  '[:find [?name ...]
    :in $ ?ns ?added
    :where
    [?e :ns ?ns]
    [?e :name ?name]
    [?e :added ?added]]
  (the-ns 'clojure.core)
  "1.4"))
#_=> (*compiler-options*
#_=>  *data-readers*
#_=>  default-data-readers
#_=>  ex-data
#_=>  ex-info
#_=>  filterv
#_=>  mapv
#_=>  reduce-kv)

;; How many public vars lack a :doc string?
(meta/query
 '[:find [(count ?e)]
   :where
   [?e :ns]
   (not [?e :doc])])
#_=> [956]

;; How many of those are functions?
(meta/query
 '[:find [(count ?e)]
   :where
   [?e :ns]
   [?e ::meta/imeta-value ?value]
   [(clojure.core/fn? ?value)]
   (not [?e :doc])])
#_=> [776]

;; What are the important magic numbers in my code base?
(meta/query
 '[:find ?imeta ?value
   :in $ package
   :where
   [?e :ns ?ns]
   [(package ?ns)]
   [?e ::meta/imeta-value ?value]
   [(clojure.core/number? ?value)]
   [?e ::meta/imeta ?imeta]]
 (fn package [ns] (str/starts-with? (str (ns-name ns)) "glossa.")))
#_=> #{[#'glossa.metazoa.query.datascript/temp-id -3038]
#_=>   [#'glossa.metazoa.search/default-num-hits 30]
#_=>   [#'glossa.weave/default-display-width 80]
#_=>   [#'glossa.metazoa.fmt/default-display-width 80]}

The metadata map of each IMeta in your code is transacted to the DataScript database as an entity. The Metazoa library transacts :glossa.metazoa/imeta, :glossa.metazoa/imeta-symbol, and :glossa.metazoa/imeta-type attributes which store the IMeta itself, the qualified symbol identifier of the IMeta, and the type of the IMeta, respectively. In addition, if the IMeta is a Clojure var, the underlying value and its type are transacted as :glossa.metazoa/imeta-value and :glossa.metazoa/imeta-value-type.

Checking/Testing

If you've taken the time to adorn your codebase with rich metadata, you should take the time to ensure it remains up-to-date. Metazoa supports this through its checking and testing story.

;; Add `:expected` to your ::meta/example metadata:
(::meta/example (meta #'clojure.core/max))
#_=> {:code (max 5 -5 10 0),
#_=>  :expected 10,
#_=>  :ns #object[clojure.lang.Namespace 0x19c4c24d "glossa.metazoa-meta"]}

;; And you'll get meaningful check output:
(meta/check #'clojure.core/max :glossa.metazoa/example)
#_=> [{:code (max 5 -5 10 0),
#_=>   :expected 10,
#_=>   :actual-out "",
#_=>   :actual-err "",
#_=>   :actual 10}]

Evaluate (meta/check imeta k) to see a data representation expressing your metadata's validity; run (meta/test-imeta imeta k) to assert the same validity using clojure.test. The separation of functional checking and side-effecting testing allows users to wire up Metazoa with testing libraries other than clojure.test if desired.

Custom Checks

Metadata providers that can be meaningfully tested should implement the meta/check-metadata method, which takes an imeta and a provider keyword k like the other metadata multimethods in this library. When called, it should not use any testing forms like clojure.test/is, but instead should be a pure function that returns a "check report" map (or a sequence of such maps). See meta.api/schema-check for details.

This is an open schema, so include additional entries that would help a user debug a test failure involving your metadata provider.

Metazoa provides built-in support for clojure.test via the various meta/test-* functions. At the time of this writing, the entire output of meta/check is stringified to provide the msg argument to clojure.test/is that is used to render an explanation upon test failure.

Built-in Providers

Run (meta/help) at the REPL and work through the Metazoa tutorial to see its built-in metadata providers in action. After that, read through the glossa.metazoa-meta namespace for multiple examples, including the :glossa.metazoa/doc source of this README document.

Optional Dependencies

Here are all of Metazoa's optional-but-included-by-default dependencies:

• cljfmt/cljfmt {:mvn/version "0.8.0"}
• datascript/datascript {:mvn/version "1.2.8"}
• metosin/malli {:mvn/version "0.6.1"}
• org.apache.lucene/lucene-core {:mvn/version "8.9.0"}
• org.apache.lucene/lucene-queryparser {:mvn/version "8.9.0"}

You can exclude these via :exclusions in your project dependencies if you do not want to use the Metazoa features that rely on them.

Metazoa's searching and querying functions—since they are totally reliant on the third-party dependencies and are intended as interactive tools—will throw an exception if you attempt to use them without their dependencies. Code formatting and schema validation functionality supplied by cljfmt and malli respectively will simply be skipped without throwing exceptions.

External Metadata Providers

Official:

• (Planned) :glossa.metazoa/plantuml In its own words: "PlantUML is used to draw...diagrams, using a simple and human readable text description."
• (Planned) :glossa.metazoa/vega-lite Create powerful visualizations of your data or the behavior of your functions using Vega-Lite as your "high-level grammar of interactive graphics"

Community:

• Please open an issue on this project and tag it with community-metadata-provider so it can be considered for inclusion here.

License

Copyright © Daniel Gregoire, 2021

THE ACCOMPANYING PROGRAM IS PROVIDED UNDER THE TERMS OF THE ECLIPSE PUBLIC LICENSE ("AGREEMENT"). ANY USE, REPRODUCTION OR DISTRIBUTION OF THE PROGRAM CONSTITUTES RECIPIENT'S ACCEPTANCE OF THIS AGREEMENT.