Fulcro Datomic Docs — fulcrologic/fulcro-datomic 2.0.0-alpha1

Fulcro Datomic Docs

The Fulcro Datomic library includes a few useful additions that make it easier to integrate one or more Datomic databases as persistent storage. It includes:

A configurable database component for use with Fulcro Server
Schema and migration support (based on Yuppiechef Datomic Schema)
Experimental schema rule enforcement for additional referential integrity checks (experimental)

1. Injecting and configuring database(s)

Fulcro Datomic includes a database component that can be easily added to an Fulcro Server. The steps are quite simple:

Create a configuration (TODO: LINK TO configuration section)
Create a database using fulcro.datomic.core/build-database and assign that database a name
Inject the database into the parser (so you can access it for reads/mutations)
Pull the database from the environment when parsing a request

1.1. Database Configuration

The component looks for the configuration to come from the Fulcro Server’s built-in configuration component. You can configure as many databases as you need. The key-path for a database’s config is [:datomic :dbs :NAME]. For example, the TODO MVC implementation sets up a development mode in-memory Datomic database that will auto-migrate and auto-drop on stop/start. The keys of interest are:

:url The Datomic db URL
:schema the namespace in your source where schema definitions live
:auto-migrate automatically detect and run missing migrations (recommended OFF in production. You can run migrations manually from command line)
:auto-drop automatically drop the database on shutdown. Useful in development mode where you’re still chaning schema in incompatible ways.

{ :datomic {:dbs {:NAME {:url          "datomic:mem://todo"
                        :schema       "todomvc.migrations"
                        :auto-migrate true
                        :auto-drop    true
                        }}}}

1.2. Creating the database component

Create a database component is as simple as (fulcro.datomic.core/build-database :NAME). The :NAME keyword must match the name used in the config. Typically you will embed this into your server build. You can see this in action in TODO MVC at initial setup:

(:require
  [fulcro.server.core :as core]
  [fulcro.datomic.core :refer [build-database])
  [om.next.server :as om])

(defn make-system []
  (let [config-path "/usr/local/etc/server.edn"]
    (core/make-fulcro-server
      :config-path config-path
      :parser (om/parser {:read my-api-read :mutate my-api-mutate})
      :parser-injections #{:todo-database}
      :components {:todo-database (build-database :todo)
                   :logger        {}})))

Note that the build-database is given the name to configure, and you can use an alternate name for the actual component (in this case :todo-database). The component can be made available simply by including it in the injection set (:parser-injections).

1.3. Accessing the Database in your Parser

The server reads/mutates will automatically include any components that you name in the :parser-injections set. So, you read/mutate need only pull them out:

Many samples exist in the code of TODO MVC. For example:

(:require
  [fulcro.datomic.protocols :as udb]
  [datomic.api :as d])

(defmethod apimutate 'todo/delete-item [{:keys [todo-database]} _ {:keys [id]}]
  {:action #(let [connection (udb/get-connection todo-database)
                  tx [[:db.fn/retractEntity id]]]
             @(d/transact connection tx)
             true)})

The database component API includes the following (protocol) methods:

(ns fulcro.datomic.protocols)

(defprotocol Database
  (get-connection [this] "Get a connection to this database" )
  (get-info [this]
    "Get descriptive information about this database. This will be a map that includes general information, such as the
    name and uri of the database. Useful for logging and debugging.")
  (get-db-config [this]
    "Get the config for this database, usually based on the database's name"))

2. Schema Migrations

The goals of the schema migration system are:

Ensure that you cannot start a production server on a database whose schema does not conform to the desired schema.
Ensure that you have the means to prevent accidental premature migration of a production database.
Ensure that migrations are applied once and only once.
Easily track what changes have been made to the database over time.

The schema support in Fulcro Datomic is a fork of Yuppiechef’s Datomic Schema with additions for experimental additional schema enforcements. The Database Configuration section talks about the :schema key. The namespace you use there will be scanned for files that are not named template.clj. The migration system scans the names for an ISO date. We recommend naming them description_yyyyMMDD.clj. The description will not affect the sorting.

The migrations (once sorted) are then checked via conformity. If auto-migrate is true, then the migrations are applied to the database (and recorded as applied). Otherwise the startup will throw an exception if the database does not conform.

2.1. Running Migrations in Development

You can easily configure you development machine with a file that sets auto-migrate to true. Optionally setting auto-drop to true can make life easier as you rapidly evolve and edit your initial schema.

2.2. Running Migrations in Production

Once you’re in production, you will be wanting to be very careful with migrations. You will want to ensure that your migrations will apply in order, and that there is a controlled procedure for doing them. It is an ERROR to edit a migration that has been applied in production. You should only ever add new migration files to be applied in production.

To meet the overall goals (stated at the top of this section), we recommend that you set auto-migrate to false in your production configuration. The following sample entry point can serve as a template for starting your server but also allowing an admin to run your migrations from the CLI in a controlled fashion:

(ns core
  (:require
    [com.stuartsierra.component :as component]
    [YOUR.SYSTEM :as sys]
    [fulcro.server.core :as c]
    [fulcro.server.impl.components.config :refer [load-config]]
    [fulcro.datomic.schema :as schema]
    [fulcro.datomic.core :as dc]
    [taoensso.timbre :as timbre])
  (:gen-class))

(def console (System/console))
(defn exit [exit-code]
  (System/exit exit-code))

(defn exit-if-headly
  "Exits with specified unix-y exit code, if the program is being run from a command line."
  [exit-code]
  (if console (exit exit-code)))

(def config-path "/usr/local/etc/production.edn")
(def production-config-component (c/new-config config-path))

(defn -main
  "Main entrypoint"
  [& args]
  (let [system (c/make-fulcro-server .....) ; your fulcro server
        stop (fn [] (component/stop system))
        cli-config (load-config production-config-component)
        db-config (:dbs cli-config)]
    ; main-handler is a pre-written CLI hook for dealing with args
    (if args (do (dc/main-handler db-config args) (exit-if-headly 0))
             (if (or (:auto-migrate cli-config) (empty? (schema/migration-status-all db-config false)))
               (do (.addShutdownHook (Runtime/getRuntime) (Thread. stop))
                   (component/start system))
               (do (timbre/fatal "System startup failed! Database does not conform to all migrations")
                   (exit-if-headly 1))))))

The above main:

Will fail to start the server if the migrations are out of date
Can be used to check/run migrations
Will start the server if migrations are already properly applied

Now (assuming you have a config file in place) you should be able to:

java -jar myserver.jar : Start the server (fails if migrations are not applied)
java -jar myserver.jar --help : Get help on running migrations
java -jar myserver.jar --status all : Get the migration status of all databases
java -jar myserver.jar --migrate all : Run migrations on all databases
java -jar myserver.jar --migrate todo : Run migrations on the todo database
java -jar myserver.jar -l : List all of the configured databases

3. Schema Extensions

Fulcro Datomic includes some additional schema support that is integrated into the database as metadata. This information can be useful as simple documentation, and can even be validated with an included (experimental) function vtransact (for validated transact). You may wish to simply use datomic-schema directly instead, but these extensions are under experimental evaluation in our own work. It was impossible for us to integrate our extensions as part of datomic-schema, and as such we ended up with a fork.

A full sample schema looks like this in a migration file:

(ns sample-migrations.migrations.users-20150609
  (:require [fulcro.datomic.schema :as s]
            [datomic.api :as d]))

(defn transactions
  "Returns a vector of transactions (each of which is a vector of operations to transact)"
  []
  [(s/generate-schema
     [(s/schema user
               (s/fields
                  [user-id :uuid :unique-identity :definitive "Unique User ID for the user."]
                  [email :string :unique-value "Email for the user. Must be unique across all users"]
                  [password :string :unpublished "Hash encoded password"]))

       (s/schema component (s/fields [name :string :unique-identity]))

       (s/schema application
                 (s/fields
                   [application-id :uuid :unique-identity :definitive]
                   [name :string]
                   [component :ref :many :component {:references :component/name}]))]
     {:index-all? true})
   (s/entity-extensions :user "A User" [])
   (s/entity-extensions :component "A component" [:user/email])
   (s/entity-extensions :application "An application" [])])

3.1. Typed Entities

The schema extensions have the concept of loose types for entities. The idea is that the namespace of attributes indicates (confers) a type IFF that attribute is marked :definitive. For example, a Datomic entity with:

{ :user/user-id 44 }

is a user because :user/user-id is marked definitive. From this point forward, ONLY attributes that have namespaces of the conferred type (and foriegn attributes) are allowed on that entity (enforce only if you use vtransact).

An entity with:

{ :user/user-id 22 :application/application-id 44}

would be both a user and an application (validation is done at the end of creation, and both of those attributes confer a type.)

3.2. Foreign Attributes

In some cases it makes sense for an attribute to appear on an entity even though the inferred type is wrong. These are known as foreign attributes, and are specified via a separate schema transaction using the entity-extensions function. This function both adds a doc string for the overall entity, and allows you to list attributes that are legally allowed to appear on an entity. In the schema above we’ve indicated that a component can have a :user/email (even though it isn’t a user). Perhaps for the admin of that component.

If an attribute is :definitive and foreign, then it will confer its type when added (e.g. :user/email can be added at any time to a component, but it does not make the component also become a user. If :user/email was marked :definitive, then adding it to a component would add that type to that entity (which would then be both a component and user). Again, all of this is enforced by vtransact.

3.3. Additional Schema Markup

In additional to the normal schema support from datomic-schema, you may append the following:

:required - Indicates that an entity of the attributes namespace type MUST include this attribute. E.g. if the required attribute’s namespace is user, then user entities MUST have that attribute.
{ :references :attr/name } - A foreign-key constraint. Indicates that you may use the specified (which must be a ref) to point to entities that contain the given target :attr/name attribute. This, combined with typed entities and :required can be used to place explicity limits on the entity graph.
:definitive - Indicates that the given attribute confers its namespace as an additional type on an entity.
:unpublished - Indicates that the attribute contains secure information and should not be shown (advisory marker…you interpret the meaning)

3.4. Foreign Key Integrity

The map {:references :attr/name} indicates that a given reference attribute may only point to an entity (or entities) have the given attribute. Since entities have no real type, you specify an attribute that you expect to be on all attributes of the referenced type (e.g. :user/user-id). This, combined with the :required marker can be used to create a system of foreign-key integrity. Again, this is enforced when using vtransact.

3.5. Validated transactions (EXPERIMENTAL)

The library comes with a function called vtransact. It works just like Datomic’s transact, but enforces the above additional schema constraints. This function should be considered experimental. We’d be interested in feedback.

The API is identical to Datomic’s transact function.