Liking cljdoc? Tell your friends :D

Building your first API

Welcome

This is a guide to get you started with Vase - the data-driven microservice container. It will help you write your first descriptor file.

What You Will Learn

After reading this guide, you will be able to:

Write a Vase API descriptor
Exchange data with Vase services

Getting Started

It may help to understand the design of the system and the file formats used before you begin. The most important pieces:

Vase descriptors are usually stored in Extensible Data Notation.
Vase descriptors are just Clojure data in memory, so the EDN format is a convenience not a requirement.
Vase services use JSON as their data exchange format.

It also important to note that the terms service, server, container are all used interchangeably.

First, be sure you have a Datomic transactor running. See the Datomic docs to get set up. Note that the template project uses Datomic Pro. (You can change the project.clj dependency to use Datomic Free. Be sure to make the corresponding change in the :datomic-uri later.)

Create a new project from the Vase leiningen template:

lein new vase your-first-api

Up and running

The exact sequence varies a little bit depending on whether you prefer Leiningen or Boot.

With Leiningen

Start the service by running the following command at a terminal:

lein repl

Once the REPL is running, start a dev mode server like this:

(def srv (run-dev))

That means your server is up and running, listening on port 8080.

With Boot

Start the service by running the following command at a terminal:

boot repl

Once the REPL is running, start a dev mode server like this:

(require 'your-first-api.server)
(in-ns 'your-first-api.server)
(def srv (your-first-api.server/run-dev))

Either way

There will be a lot of logging, but it should end with something like this:

INFO  o.e.jetty.server.AbstractConnector - Started ServerConnector@5d04c566{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:8080}
INFO  org.eclipse.jetty.server.Server - Started @85814ms

A Personal Inventory System

Let's build a system to handle basic user accounts and can track items they own.

With this simple goal statement, we can already envision some pieces of the data model and even some URLs that might appear in the API.

Our data model needs to handle:

Items (with item IDs, names, and descriptions)
Users (with user IDs, emails, and a list of items they own)

And we might have URLs that:

Fetch all items in the system
Fetch all users in the system
Fetch a user given their user ID
Fetch a user's owned-item list given their user ID
etc.

We put the description into a .edn file that our service loads. (EDN is specified by https://github.com/edn-format/edn.) This description tells Vase what API a service offers, what schema it defines, and what specifications the data must follow.

The Vase template created resources/your-first-api_service.edn with a lot of examples. For the moment, it has too much. Just delete that file and start a fresh one with these contents:

{:activated-apis []
 :datomic-uri "datomic:mem://example"
 :descriptor

 {:vase/norms {}
  :vase/specs {}
  :vase/apis  {}}}

This says we have no schema, no APIs, no specs, and nothing will run. Not very exciting, is it? It gets better in a bit.

Here are the meanings of the top-level keys:

Key	Type	Meaning
`:activated-apis`	Vector of API "names."	Routes for these APIs will be created. Schema used by these APIs will be applied.
`:datomic-uri`	String with a Datomic URI	This is the database that Vase will use. If it doesn't exist, Vase will create it.
`:descriptor`	Map with specific keys expected	Holds the schemas (`:vase/norms`), specs (`:vase/specs`), and APIs (`:vase/apis`

Building the data model

We often start with a model of the entities a service must manipulate.

Data models are defined with a schema (called "norms" for historical reasons.) You may split your schema up into logical pieces that map directly to your application domain. In our example, we'll define one part of the schema for Items, and another part for Users.

The :vase/norms key is where we put schema definitions. It holds a map of "schema name" to schema definition. The schema name is a keyword that you choose. We recommend using a namespaced keyword. It's best to avoid using vase as part of your namespace. It's not a reserved name, but you may confuse parts of your application with parts that Vase requires.

To start with, let's make space for our two schemas:

 :vase/norms
  {:accounts/item {}
   :accounts/user {}
  }

(This goes inside the map attached to :descriptor.)

This says we're building one schema named :accounts/item and another named :accounts/user. The first component of a schema is defining its normalized, master form. That's done using the :vase/norms entry. In this map, we'll define attributes for Items and Users. You'll notice that the Users schema requires the Items schema, so that we can describe a user's owned items. You'll also notice that attributes of a given entity are defined in terms of database transactions (txes) that describe them.

Vase offers a reader literal shorthand called #vase/schema-tx for defining Datomic schema transactions within :vase.norms/txes. Its body is a vector of vectors. Each subvector defines an attribute name, its cardinality (:one or :many), its type, some optional flags, and a doc string.

The allowed flags are translated into parts of the attribute definition for Datomic:

Flag	Meaning	Equates to
`:unique`	Only one entity in the DB can have this value for this attribute	`:db/unique :db.unique/value`
`:identity`	An entity is uniquely identified by this value. Upsert is enabled	`:db/unique :db.unique/identity`
`:index`	This attribute should be indexed	`:db/index true`
`:fulltext`	This attribute should be searchable as text. (Mildly deprecated.)	`:db/fulltext true`
`:component`	The entity referenced by this attribute should be retracted when this one is. (Cascading delete)	`:db/isComponent true`
`:no-history`	Do not preserve old values of this attribute	`:db/noHistory true`

Even though the short-schema version covers most of the use cases you'll need, you're always free to use full Datomic schema definitions like:

:vase.norm/txes [[{:db/id          #db/id[:db.part/db]
                   :db/ident       :item/name
                   :db/valueType   :db.type/string
                   :db/cardinality :db.cardinality/one
                   :db/doc         "The name of an item"}]]

A common convention for attribute names is that the namespace identifies the "entity type" that the attribute belongs to. This is strictly for human consumption, though. Datomic doesn't have any notion of entity type and doesn't care how you mix and match attributes.

Fill in the schema parts like this:

 :vase/norms
 {:accounts/item
  {:vase.norm/txes [#vase/schema-tx
                     [[:item/itemId      :one :long   :identity "The unique identifier for an item"]
                      [:item/name        :one :string           "The name of an item"]
                      [:item/description :one :string           "A short description of the item"]]]}

  :accounts/user
  {:vase.norm/requires [:accounts/item]
   :vase.norm/txes [#vase/schema-tx
                     [[:user/userId      :one  :long   :identity "The unique identifier for a user"]
                      [:user/email       :one  :string :unique   "The email address for a given user"]
                      [:user/items       :many :ref              "The collection of items a user owns"]]]}}

This says the :accounts/item schema has three attributes: :item/itemId, :item/name, and :item/description. The :accounts/user schema has three more attributes: :user/userId, :user/email, and :user/items. Furthermore, the :accounts/user schema requires that the :accounts/item schema must be in place first. (It doesn't really... there are no definitions in :accounts/user that depend on :accounts/item. This is just for illustration in this tutorial.)

Making It Act

With the data model in place, we can build our API.

Vase defines routes defined with URL strings, HTTP verbs (get, post, etc), and action literals. The foundation of Vase's routing is based on Pedestal's capabilities, but with care given to represent routes in an external data file.

Fill in :vase/apis with this route that gives some information about our API:

 :vase/apis
 {:accounts/v1
  {:vase.api/routes
   {"/about" {:get #vase/respond {:name :accounts.v1/about-response
                                  :body "General User and Item Information"}}}}}

Routes are defined as nested maps. Each map defines a single route, and then a map of HTTP verbs to action literals---in this case #vase/respond. Every action literal requires a unique :name. Other keys and their interpretations are defined per literal. Here we see that GET /api/accounts/v1/about will respond with a string.

These are the action literals:

Literal	Purpose
`#vase/respond`	Respond with static data, optionally formatted as JSON
`#vase/redirect`	Redirect an incoming request to a different URL
`#vase/query`	Respond with the results of a database query
`#vase/transact`	Add or update data in the database
`#vase/validate`	Validate data against specs
`#vase/intercept`	Apply a hand-crafted, artisinal interceptor

An API usually depends on some amount of schema. In this example, we've added a dependency from the :accounts/v1 API to the User schema. Since the User schema depends on the Item schema, both parts of schema will be applied to our database.

{:vase/apis
 {:accounts/v1
  {:vase.api/routes  ,,, ;; skipping the routes for space
   :vase.api/schemas [:accounts/user]}}}

An API can depend on any number of schemas. You should feel free to grow and evolve your schema by adding new "norms."

One note: Vase tracks which schemas it has already applied to a database. Think of each schema like a migration in other database frameworks: once it's applied to the database you don't change it. Just add new schemas under :vase/schemas and add them to your APIs' :vase.api/schemas dependencies.

Activating the API

You can use curl to test the new URL, but it won't work yet.

curl http://localhost:8080/api/accounts/v1/about

If you've been following along in this guide, you got a 404 response just now. That's because Vase has one more concept about APIs: activation.

A single EDN file can define many APIs and many schema fragments. It is up to a service instance to determine which of these to activate using the top-level key :activated-apis. This is in the EDN file for ease, but could be supplied separately as part of your service config. (For example, as EC2 instance data.)

To activate the accounts API, modify the top of your EDN file like this:

{:activated-apis [:accounts/v1]
 ,,,

Now re-run curl and you'll get back a 200 status code with the body string from our #vase/respond literal.

Forwarding Headers

Some Javascript and Clojurescript clients provide request IDs that they use to correlate requests and responses. We can tell Vase to forward headers from the request through to the client. This is at the level of the whole API:

{:vase/apis
 {:accounts/v1
  {:vase.api/routes          ,,, ;; skipping the routes for space
   :vase.api/schemas         [:accounts/user]
   :vase.api/forward-headers ["vaserequest-id"]}}}

This says to forward the vaserequest-id header from every request to every response. This HTTP header is used to trace and debug requests to the service (and is automatically added if it's not sent in).

Handling Parameters

Vase does some code generation to make HTTP parameters available as Clojure bindings for a route's action literal(s). These are conveyed from the route to Vase by Pedestal, using the :params keyword in the request map.

Params arrive in various forms, extracted from:

EDN payload for POSTs
JSON payload for POSTs
Form data for POSTs
Query string arguments
URL (i.e., path) parameters.

Parameters are resolved with an order of precedence, as listed below:

EDN POST payloads override
JSON POST payloads override
POST form data payloads override
Query string args override
URL parameters

See Pedestal's docs for a complete reference on parameters.

Let's look at them from the bottom up.

Path Parameters

A path parameter binds a single value from a URL to a symbol name in your action literal.

Add a route with a path parameter:

 :vase/apis
 {:accounts/v1
  {:vase/routes
   {"/about"            {:get #vase/respond {:name   :accounts.v1/about-response
                                             :body   "General User and Item Information"}}
    "/about/:your-name" {:get #vase/respond {:name   :accounts.v1/about-yourname
                                             :params [your-name]
                                             :body   your-name}}}
   :vase.api/schemas         [:accounts/user]
   :vase.api/forward-headers ["vaserequest-id"]}}

In this trivial example we bind part of the URL path to the symbol your-name and return it as the body of our response. A path parameter value is always a string.

Try it out with curl:

curl http://127.0.0.1:8080/api/accounts/v1/about/paul

Let's see how we might take multiple parameters for a given route.

Query Parameters

Query string args are typically used for filtering the result of returned data. Here's an example were we'll return a JSON response for all of our expected query args

 :vase/apis
 {:accounts/v1
  {:vase/routes
   {"/about"            {:get #vase/respond {:name   :accounts.v1/about-response
                                             :body   "General User and Item Information"}}
    "/about/:your-name" {:get #vase/respond {:name   :accounts.v1/about-yourname
                                             :params [your-name]
                                             :body   your-name}}
    "/aboutquery"       {:get #vase/respond {:name   :accounts.v1/about-query
                                             :params [one-thing another-thing]
                                             :body   {:first-param  one-thing
                                                      :second-param another-thing}}}}
   :vase.api/schemas         [:accounts/user]
   :vase.api/forward-headers ["vaserequest-id"]}}

With curl:

curl 'http://127.0.0.1:8080/api/accounts/v1/aboutquery?one-thing=hello&another-thing=world'

(Make sure you quote the whole string properly... the '?' and '&' mean something entirely different to the shell.)

Notice that the response that comes back is JSON, not text like the other #vase/respond action we specified. That's because the new action returned a Clojure data structure instead of a string. When the body of a response is not a string, Vase converts it to JSON.

Parameter Defaults

You can provide a default value for any :params binding. For example, :params [your-name [age 42]]. Of course, if a path parameter is nil, then the route didn't even match. But query and body parameters can be defaulted this way.

A Dangerous Truth

So far, we've lead you to believe that action literals are purely data. That's not entirely true.

Many parts of the action literals are evaluated as code, in an environment where the parameter names are bound. Everything from clojure.core is available. That is, unless you shadow something from clojure.core with a parameter name!

This means we can add a function call directly to our #vase/respond actions. Let's update our url-param route to print a more interesting string using clojure.core/str.

Modify the action for "/about/:your-name" like this:

 :vase/apis
 {:accounts/v1
  {:vase.api/routes
   {"/about"            {:get #vase/respond {:name   :accounts.v1/about-response
                                             :body   "General User and Item Information"}}
    "/about/:your-name" {:get #vase/respond {:name   :accounts.v1/about-yourname
                                             :params [your-name]
                                             :body   (str "You said your name was: " your-name)}}
    "/aboutquery"       {:get #vase/respond {:name   :accounts.v1/about-query
                                             :params [one-thing another-thing]
                                             :body   {:first-param  one-thing
                                                      :second-param another-thing}}}}
   :vase.api/schemas         [:accounts/user]
   :vase.api/forward-headers ["vaserequest-id"]}}

With great power comes great responsibility - bending this ability is dangerous and can cause the container to crash, but in a pinch it can allow you to shape an API to meet your needs.

Obviously, this means anyone who can alter your descriptor can run arbitrary code in your server. Don't accept user inputs as descriptors!

Getting data in with `transact`

In addition to rendering content, the Vase system also provides a #vase/transact action allowing the storage of incomming POST data.

Vase expects transaction data to arrive as a JSON entity body. The top level of the body is an object with the single key payload. The payload should be a collection of entity bodies (i.e., maps) to transact.

Add the "/user" route shown here to your descriptor:

  :vase/apis
  {:accounts/v1
   {:vase.api/routes
     {"/about"            {:get #vase/respond    {:name       :accounts.v1/about-response
                                                  :body       "General User and Item Information"}}
      "/about/:your-name" {:get #vase/respond    {:name       :accounts.v1/about-yourname
                                                  :params     [your-name]
                                                  :body       (str "You said your name was: " your-name)}}
      "/aboutquery"       {:get #vase/respond    {:name       :accounts.v1/about-query
                                                  :params     [one-thing another-thing]
                                                  :body       {:first-param  one-thing
                                                              :second-param another-thing}}}
       "/user"             {:post #vase/transact {:name       :accounts.v1/user-create
                                                  :properties [:db/id
                                                               :user/userId
                                                               :user/email]}}}
    :vase.api/schemas         [:accounts/user]
    :vase.api/forward-headers ["vaserequest-id"]}}

The new route has a single #vase/transact literal with a name and properties. The :properties key holds a whitelist of attribute names that Vase will accept in the incoming POST data. In this case, the payload should have a sequence of maps that each have :user/userId and :user/email keys and values. These will be asserted in Datomic.

Try a Transaction

We want to POST the following JSON

{"payload":
  [{
    "user/userId" : 42,
    "user/email" : "user@example.com"
  }]
}

This is how you would use cURL to POST such a payload:

curl -H "Content-Type: application/json" -X POST -d '{"payload": [{"user/userId": 42, "user/email": "user@example.com"}]}' http://localhost:8080/api/accounts/v1/user

The quoting is a little different on Windows:

curl -H "Content-Type: application/json" -X POST -d "{\"payload\":[{\"user/userId\":42,\"user/email\":\"user@example.com\"}]}" http://localhost:8080/api/accounts/v1/user

Give it a try!

Insert, Assert, Upsert

The properties :user/userId and :user/userEmail are fairly self-explanatory, but the :db/id property is handled specially. That is, the :db/id key signifies if the incoming data refers to existing entities in the Vase database, or to new entities.

In the JSON packet above, the entity did not contain a :db/id field. Vase treats it as a new entity and attaches a tempid before asserting it.

At this point, one of three things will happen:

The entity has an :identity attribute whose value matches an existing entity in the database. This becomes an "upsert" on that entity: the new attribute values are merged with the existing entity.
The entity has a :unique value that already exists in the database. Because it has a tempid, Datomic will reject the transaction and Vase will return an error to the client.
Neither of the above are true, and Datomic will create a new entity.

It is possible to supply a db/id directly, like this:

{"payload":
  [{
    "db/id" : 100,
    "user/userId" : 9,
    "user/email" : "user9@example.com"
  }]
}

Because the :db/id field is set to a value, Vase will try to resolve the entity in the database before transacting the data. Obviously, if no such entity exists then a failure will occur, thus notifying the calling client.

One final way to refer to existing entities is to set the value at the :db/id field to correspond to a unique value for the entity in question. For example:

{"payload":
  [{
    "db/id" : ["user/userId", 9],
    "user/email" : "user9@example.com"
  }]
}

We declared :user/userId as an :identity attribute. That means we can use it in lookup refs as well as doing upsert with it.

See Datomic's lookup refs for more details.

Getting data out with `query`

The #vase/query action provides a way to define service routes that return data based on Datalog queries.

Go ahead and add a :get action to the "/user" route like this:

  :vase/apis
  {:accounts/v1
   {:vase.api/routes
     {"/about"            {:get #vase/respond    {:name       :accounts.v1/about-response
                                                  :body       "General User and Item Information"}}
      "/about/:your-name" {:get #vase/respond    {:name       :accounts.v1/about-yourname
                                                  :params     [your-name]
                                                  :body       (str "You said your name was: " your-name)}}
      "/aboutquery"       {:get #vase/respond    {:name       :accounts.v1/about-query
                                                  :params     [one-thing another-thing]
                                                  :body       {:first-param  one-thing
                                                              :second-param another-thing}}}
       "/user"             {:post #vase/transact {:name       :accounts.v1/user-create
                                                  :properties [:db/id
                                                               :user/userId
                                                               :user/email]}
                            :get #vase/query     {:name       :accounts.v1/user-page
                                                  :params     [email]
                                                  :query      [:find ?e
                                                               :in $ ?email
                                                               :where [?e :user/email ?email]]}}}
    :vase.api/schemas         [:accounts/user]
    :vase.api/forward-headers ["vaserequest-id"]}}

The "/user" route now supports both POST and GET requests. The POST request hits the #vase/transact the same as before. Now a GET runs the #vase/query action. The query looks up an entity based on a query string parameter. The two main keys of interest in the #vase/query action are :params and :query. (We'll discuss an optional third property a bit later.)

The :params property defines the accepted keyed data names that are used as external arguments to the query to resolve those listed parameters to the incoming values. The #vase/query action passes these as extra arguments to datomic.api/q, following the database value itself.

If the :params field is empty or missing, they query doesn't accept any arguments. In that case all parameters in the URL, query string, form, etc. will be ignored.

The :query property contains a Datomic datalog query.

One limitation of providing query parameters as URL arguments or path parameters is that only string types are shuttled across to the server. Often it's useful to refer to arguments that are other useful types, numbers are a common case. Here's a new route to illustrate that conversion:

     "/user/:id" {:get #vase/query {:name :accounts-v1/user-id-page
                                    :params [id]
                                    :edn-coerce [id]
                                    :query
                                    [:find ?e
                                     :in $ ?id
                                     :where [?e :user/userId ?id]]}}

(We're going to stop repeating all the routes in the interest of saving space and helping focus on the new stuff. It should be clear by now where this goes in the EDN file.)

The new route /user/:id allows the a similar information lookup, but with the argument as a path parameter (e.g. http://example.com/user/id). By default the :id parameter would be a string value, but by using the :edn-coerce property of the #query action we tell Vase to attempt to parse the string as a valid EDN data type. Therefore, when clients hit the URL bound to that query the proper types will match (i.e. the DB expects integer IDs, not string IDs).

Querying `or` and other constants

It's often useful to model a query that match against anything within a given data set, for example, "Give me all users whose email is in ["jane@domain.com", "bill@domain.com"]"

In Datomic this is called a "parameterized in/or query," and it's achieved by binding the names with the :in clause. To do this, we supply additional constant data to the Vase query with the :constants option. This can also be used with parameter binding (covered above), in which case parameters are passed in to the query before the constants.

An example of simple constants follows - observe our last change to the schema below:

     "/special-users" {:get #vase/query {:name :accounts.v1/special-users
                                         :params []
                                         :constants [["jane@domain.com" "bill@domain.com"]]
                                         :query
                                         [:find ?e
                                          :in $ [?email ...]
                                          :where
                                          [?e :user/email ?email]]}}

Be Persistent

So far, we've used an in-memory URI for Datomic. That means just what it sounds like: values are only stored in memory. To make it persistent, you need to pick a storage engine and update the :datomic-uri value.

Wrapping Up

We've covered most of the action literals. The examples in this guide created a real, if somewhat quirky, API for an accounts system.

The next step is to read the Action Literals reference.

❮Design Documentation and Vision

Can you improve this documentation? These fine people already did:
Michael Nygard, Michael T. Nygard, David Goldfarb, Donald Ball, Paul deGrandis, Malchevskiy Misha & Jimmy MillerEdit on GitHub

cljdoc is a website building & hosting documentation for Clojure/Script libraries

Keyboard shortcuts Report a problem cljdoc on GitHub

× close