What is this

See this ClojureVerse thread about a central documentation hub for Clojure similar to Elixir's https://hexdocs.pm/.

If you want to know more, open an issue or contact me elsewhere.

Contents

• Design (wip)
• Progress
  • Hosting
  • Service + Isolation
  • API docs
  • Grimoire
  • GitHub + Non-API docs
  • Long Shots
  • Bot
• Notes
  • UI Ideas
  • Versioning Thoughts
  • Change Viewer
  • Rendering Extensibility

Design (wip)

• build an ecosystem-encompassing Grimoire store
• build various tools around that store, ideas:
  • static html API docs + guides
  • single page app to browse documentation
  • API
  • machine readable doc-bundles
  • Docsets for documentation browsing apps
• build tools to raise overall quality of documentation (Github bots, Doc templates, etc)

Progress

HOSTING

• [x] setup s3 bucket / static website using Confetti
• [x] sync files generated by build-docs to S3
• [ ] attach some metadata to S3 objects to identify ones that have been built with old versions of cljdoc

SERVICE + ISOLATION

• [ ] create Docker setup to run build-docs
  • starting point sudo docker run -it adzerk/boot-clj repl
• [ ] investigate AWS lambda for doing analysis work
• [ ] only the grimoire analysis would need to run isolated, everything else operates on top of the grimoire store, which is safe to read
• [ ] create an API server that runs the Docker setup
• [ ] copy created files to host, upload to S3 (so that AWS keys are not exposed in Docker env)
• [ ] How to get notified when new stuff is released on Clojars?
• [ ] Split codox analysis into it's own module/library (this is the only thing that really needs the isolation)

API DOCS

• Linking to source files/lines on GitHub
  • Potential Approaches
    • probably needs parsing of project.clj or build.boot or
    • perhaps we can derive source locations by overlaying jar contents
    • if the repo is tagged we can try finding the file with the same SHA as the file in the jar and derive it's repo location this way
      • this information should go into the namespace-meta I believe
  • Grimoire data contains source so perhaps source URI becomes less important
  • still would be nice to jump to the entire source file on Github
• [x] give grimoire and related tools a go
• [x] Build Codox style index page based on grimoire info
  • :bulb: Generating pages straight from grimoire info results in pretty tight coupling
  • :bulb: We should probably generate an intermediate format and cache it in Transit
  • :bulb: This cache could be used to generate static and SPA pages alike
  • :bulb: Spec could be used to validate the cache format / generate sample data
• [x] think about how different platforms can be combined in API docs
  • def list should only show one name with appropriate indication
  • clj-only, cljs-only, both — maybe CLJ, CLJS, CLJ/S
• [ ] build static site generator or SPA that runs on top of grimoire data

GRIMOIRE

• :white_check_mark: Grimoire currently does not support cljs or cljc
  • lib-grimoire #29 & lib-grimoire #30
  • :tada: Mixing Codox and Grimoire has proven productive
  • [x] codox has support for clojurescript - could this code be shared?
  • [x] not necessary maybe put this into a shared library that grimoire and codox can use
  • lib-grimoire doesn't actually analyse namespaces, it's done in lein-grim
  • [x] look into replacing the code I copied from lein-grim with something similar to codox' implementation
    • [x] depend on codox, make sure it's reader implementation can work with grimoire
    • [x] not necessary fork codox, turn into codox-reader export api including this stuff in codox.reader namespace
• [ ] throw an error or something if grimoire does not find anything
• [x] could grimoire data include information whether a var is a protocol, record etc?
• Building docs from jar vs src — what are the tradeoffs?
  • Not really a grimoire thing
• Are there any fundamental issues with grimoire that could become problematic later?
• How do you build docs for a [prj v] when another version is already on the classpath? (e.g. grimoire itself)
  • This will also apply to other libraries on the classpath for doc generation (codox, tools.namespace, etc)
  • One way could be to "rename" namespaces of the to be documented library and maintain the mapping, although that might show up in the docs in some way?
  • https://github.com/benedekfazekas/mranderson
  • :clock10: While this will be needed eventually it's not necessary for a first version
• :construction: Getting source of defs
  • @arrdem mentioned source-fn might have issues with anything that uses ::foo or ::foo/bar keywords
    • [ ] check if this still applies
  • [ ] Check how whatever mechanism works for .cljc forms — what forms are emitted, if any?
• :sparkles: lib-grimoire #31 Spec integration
• What to put in metadata store of:
  • group
  • artifact
  • version: scm-url, git-tag, git-sha
  • platform

GITHUB + NON-API DOCS

• [x] read Github URL from pom.xml or Clojars
• [x] clone repo, copy doc directory, provide to codox
• [x] try to check out Git repo at specific tag for doc/ stuff; warn if no tags
• [ ] figure out what other metadata should be imported
• [ ] Think about how stuff like https://shadow-cljs.github.io/docs/UsersGuide.html could be accommodated
• [ ] Figure out how to deal with changes between tagged releases
  • We probably don't want to pick up API changes since people commonly push development progress straight to master
  • Picking up changes to doc/ would be nice and the above problem probably applies a little less
  • When updating doc/ triggered by Github commits we need to derive the correct version
    • Take latest "RELEASE" version on Clojars?
    • Take version specified by most recent tag?

LONG SHOTS

• [ ] think about discovery of projects with same group-id
• [ ] think about how something like dynadoc (interactive docs) could be integrated
• [ ] think about how stack style REPL examples could be integrated
  • https://github.com/arrdem/stacks/issues/7

BOT

• [ ] notify people that there are api docs available for a jar they just published
• [ ] suggest to add some plain text documentation/guides + provide templates

Notes

UI IDEAS

• Highlight items with long docstrings as "well-documented" (needs testing)

VERSIONING THOUGHTS

• Show version in UI when viewing versioned information (i.e. anything in grimoire) and show "unversioned" with red warning sign when showing unversioned documentation (i.e. from repo's that have not been tagged)
• How to version article documentation:
  • [x] :speech_balloon: Discuss this with @arrdem
  • Problem: Articles may receive useful updates between tagged releases and thus updates could be useful. Also article documentation seems more stable intuitively.
  • Solution: Show last tagged version with a +N indicator where N is the number of doc/ revisions since the tagged release
    • doc/ revisions are counted as the number of different shasums for the contents of the doc/ directory across all commits since the last tagged release

CHANGE VIEWER

• Using caches of different versions a diff could be calculated, it could be used to
  • show newly added functions
  • show functions whose arglists have changed
  • show functions whose source has changed
  • deduce at which version a function was added (perhaps even at arity level)
• :wrench: Users may want to configure some priority for changes accross namespaces or maybe even at the function level.
• Git blame could be used to identify commits that affected a given function's source
• :small_orange_diamond: Changes to a functions source may affect other upstream functions' API/return values so just knowing changed functions may not be as useful to learn about API breakage.
  • It could help with other things as well (e.g. understanding how a function evolved)

RENDERING EXTENSIBILITY

(defprotocol ICacheRenderer
  (render [cache output-config]
    "Render contents of cache to :file or :dir specified in output-config"))

Intended implementations:

• Single Transit file
• Multiple HTML files
• Single HTML file
• Single Page Apps
  • Single HTML file (CLJS app only working with existing data)
  • Global app (CLJS app with ways out of the current context: other projects, versions etc.)
  • Both of them could share a lot of code but have slightly different entry points
• Docsets for Dash and similar apps
  • Based on HTML, perhaps using dashing