Liking cljdoc? Tell your friends :D

Installation

Using Leiningen / Clojars:

About

Lancaster is an Apache Avro library for Clojure and ClojureScript. It aims to be fully compliant with the Avro Specification. It is assumed that the reader of this documentation is familiar with Avro and Avro terminology. If this is your first exposure to Avro, please read the Avro Overview and the Avro Specification before proceeding.

Lancaster provides for:

Easy schema creation
Serialization of arbitrarily-complex data structures to a byte array
Deserialization from a byte array, including schema resolution
Conversion from Lancaster schemas to Plumatic schemas (spec support is planned).

Lancaster does not support:

Avro protocols
Avro container files (may be supported in the future).

Performance

Lancaster aims to be fast. Microbenchmarks show that it is generally faster than JSON serialization / deserialization while producing output that is much more compact. The output of an unscientific run of the microbenchmark in deercreeklabs.perf-test is pasted in below. Your mileage may vary.

Clojure 1.9 on JVM 1.8

Parameter	Lancaster	JSON	JSON+Deflate
Encode ops/s	`235,849`	`138,696`	`29,240`
Decode ops/s	`364,964`	`233,645`	`86,207`
Encoded size (bytes)	`14`	`142`	`105`

ClojureScript 1.10.339 on Node.js 8.10

Parameter	Lancaster	JSON	JSON+Deflate
Encode ops/s	`32,468`	`30,769`	`2,865`
Decode ops/s	`76,923`	`10,428`	`3,597`
Encoded size (bytes)	`14`	`162`	`109`

Project Name

The Avro Lancaster was an airplane manufactured by Avro Aircraft.

Examples

Here is an introductory example of using Lancaster to define a schema, serialize data, and then deserialize it.

(require '[deercreeklabs.lancaster :as l])

(l/def-record-schema person-schema
  [:name l/string-schema]
  [:age l/int-schema])

(def alice
  {:name "Alice"
   :age 40})

(def encoded (l/serialize person-schema alice))

(l/deserialize person-schema person-schema encoded)
;; {:name "Alice" :age 40}

Here is a more complex example, using nested schemas and the maybe function to make fields nillable:

(require '[deercreeklabs.lancaster :as l])

(l/def-enum-schema hand-schema
  :left :right)

(l/def-record-schema person-schema
  [:name l/string-schema]
  [:age l/int-schema]
  [:dominant-hand hand-schema]
  [:favorite-integers (l/array-schema l/int-schema)]
  [:favorite-color (l/maybe l/string-schema)]) ;; Field is nillable

(def alice
  {:name "Alice"
   :age 40
   :favorite-integers [12 59]
   :dominant-hand :left})

(def encoded (l/serialize person-schema alice))

(l/deserialize person-schema person-schema encoded)
;; {:name "Alice", :age 40, :dominant-hand :left, :favorite-integers [12 59], :favorite-color nil}

Creating Schema Objects

Lancaster schema objects are required for serialization and deserialization. These can be created in two ways:

Using an existing Avro schema in JSON format. To do this, use the json->schema function. This is best if you are working with externally defined schemas from another system or language.
Using Lancaster schema functions and/or macros. This is best if you want to define Avro schemas using Clojure/ClojureScript. Lancaster lets you concisely create and combine schemas in arbitrarily complex ways, as explained below.

Primitive Schemas

Lancaster provides predefined schema objects for all the Avro primitives. The following vars are defined in the deercreeklabs.lancaster namespace:

null-schema: Represents an Avro null
boolean-schema: Represents an Avro boolean
int-schema: Represents an Avro int
long-schema: Represents an Avro long
float-schema: Represents an Avro float
double-schema: Represents an Avro double
bytes-schema: Represents an Avro bytes
string-schema: Represents an Avro string

These schema objects can be used directly or combined into complex schemas.

Complex Schemas

Most non-trivial Lancaster use cases will involve complex Avro schemas. The easiest and most concise way to create complex schemas is by using the Schema Creation Macros. For situations where macros do not work well, the Schema Creation Functions are also available.

Schema Creation Macros

def-array-schema: Defines a var w/ an array schema.
def-enum-schema: Defines a var w/ an enum schema.
def-fixed-schema: Defines a var w/ a fixed schema.
def-map-schema: Defines a var w/ a map schema. Keys must be strings.
def-flex-map-schema: Defines a var w/ a flex-map schema. Flex map keys may be of any schema type.
def-record-schema: Defines a var w/ a record schema.
def-merged-record-schema: Defines a var w/ a record schema which contains all the fields of all record schemas passed in.
def-union-schema: Defines a var w/ a union schema.
def-maybe-schema: Defines a var w/ a nillable schema.

Schema Creation Functions

array-schema: Creates an array schema.
enum-schema: Creates an enum schema.
fixed-schema: Creates a fixed schema.
map-schema: Creates a map schema. Keys must be strings.
flex-map-schema: Creates a flex-map schema. Flex map keys may be of any schema type.
record-schema: Creates a record schema.
merged-record-schema: Creates a record schema which contains all the fields of all record schemas passed in.
union-schema: Creates a union schema.
maybe: Creates a nillable schema.

Operations on Schema Objects

All of these functions take a Lancaster schema object as the first argument:

serialize: Serializes data to a byte array.
deserialize: Deserializes data from a byte array, using separate reader and writer schemas. This is the recommended deserialization function.
deserialize-same: Deserializes data from a byte array, using the same reader and writer schema. This is not recommended, as it does not allow for schema resolution / evolution.
wrap: Wraps data for use in an ambiguous union. See Notes About Union Data Types below.
edn: Returns the EDN representation of the schema.
json: Returns the JSON representation of the schema.
pcf: Returns a JSON string containing the Parsing Canonical Form of the schema.
fingerprint64: Returns the 64-bit Rabin fingerprint of the Parsing Canonical Form of the schema.
schema?: Is the argument a Lancaster schema?
plumatic-schema: Returns a Plumatic schema for the schema.
default-data: Returns default data that conforms to the schema.

Data Types

Serialization

When serializing data, Lancaster accepts the following Clojure(Script) types for the given Avro type:

Avro Type	Acceptable Clojure / ClojureScript Types
`null`	`nil`
`boolean`	`boolean`
`int`	`int`, `java.lang.Integer`, `long (if in integer range)`, `java.lang.Long (if in integer range)`, `js/Number (if in integer range)`
`long`	`long`, `java.lang.Long`
`float`	`float`, `java.lang.Float`, `double (if in float range)`, `java.lang.Double (if in float range)`, `js/Number (if in float range)`
`double`	`double`, `java.lang.Double`, `js/Number`
`bytes`	`byte-array`, `java.lang.String`, `js/Int8Array`, `js/String`
`string`	`byte-array`, `java.lang.String`, `js/Int8Array`, `js/String`
`fixed`	`byte-array`, `js/Int8Array`. Byte array length must equal the size declared in the creation of the Lancaster `fixed` schema.
`enum`	`keyword`
`array`	Any data that passes `(sequential? data)`
`map`	Any data that passes `(map? data)`, if all keys are strings. Clojure(Script) records DO NOT qualify, since their keys are keywords.
`record`	Any data that passes `(map? data)`, if all keys are Clojure(Script) keywords. Clojure(Script) records DO qualify, since their keys are keywords.
`union`	Any data that matches one of the member schemas declared in the creation of the Lancaster `union` schema. Note that some unions require wrapping, as explained in Notes About Union Data Types below.

Deserialization

When deserializing data, Lancaster returns the following Clojure or ClojureScript types for the given Avro type:

Avro Type	Clojure Type	ClojureScript Type
`null`	`nil`	`nil`
`boolean`	`boolean`	`boolean`
`int`	`java.lang.Integer`	`js/Number`
`long`	`java.lang.Long`	`js/Number`
`float`	`java.lang.Float`	`js/Number`
`double`	`java.lang.Double`	`js/Number`
`bytes`	`byte-array`	`js/Int8Array`
`string`	`java.lang.String`	`js/String`
`fixed`	`byte-array`	`js/Int8Array`
`enum`	`keyword`	`keyword`
`array`	`vector`	`vector`
`map`	`hash-map`	`hash-map`
`record`	`hash-map`	`hash-map`
`union`	Data that matches one of the member schemas declared in the creation of the Lancaster `union` schema. If the union schema requires wrapping, the returned data will be wrapped. See Notes About Union Data Types below.

Notes About Union Data Types

To quote the Avro spec:

Unions may not contain more than one schema with the same type, except for the named types record, fixed and enum. For example, unions containing two array types or two map types are not permitted, but two types with different names are permitted.

In Lancaster, the data for both records and maps can be Clojure hash-maps. Also, multiple record schemas can be members in a union schema. This makes it impossible (or at least difficult) to determine which member schema of a union to use at serialization time.

A Lancaster union schema is ambiguous if it contains:

More than one record or map
More than one numeric type: int, long, float, or double
More than one bytes type: bytes, string, or fixed.

Given a Clojure hash-map, and an ambiguous union schema, which schema should be used to serialize it? This is resolved via wrapping.

Wrapping indicates the schema of the given data. Wrapped data is a two-element vector. The first element is the EDN name of the schema, and the second element is the data itself. For example:

[:user/dog {:name "Fido", :owner "Roger"}]

Wrapping is most easily accomplished using the wrap function.

When deserializing data using an ambiguous union, wrapped data is returned. This allows you to take different actions based on the schema of the data. To access the schema name of the wrapped data, use the schema-name function. To access the data inside wrapped data, use the data function.

Functions For Wrapping and Unwrapping Data

wrap: Wraps the given data for serializing with an ambigous union.
schema-name: Returns the schema-name portion of wrapped data.
data: Returns the data portion of wrapped data.

Example

;; Define two record schemas
(l/def-record-schema person-schema
  [:name l/string-schema "No name"]
  [:age l/int-schema 0])

(l/def-record-schema dog-schema
  [:name l/string-schema]
  [:owner l/string-schema])

;; Define an ambiguous union schema, using the two records
(l/def-union-schema person-or-dog-schema
  person-schema dog-schema)

(def fido {:name "Fido" :owner "Roger"})

;; Serializing without wrapping fails because the union is ambiguous:
(l/serialize person-or-dog-schema fido)
;; ExceptionInfo Union requires wrapping, but data is not wrapped.

;; Wrapping the data before serialization tells the union which type
;; to use when serializing.
(def wrapped-fido (l/wrap dog-schema fido))
;; {:dog {:name "Fido" :owner "Roger"}}

;; This works now
(def encoded (l/serialize person-or-dog-schema wrapped-fido))
encoded
;; #object["[B" 0x2cc2072e "[B@2cc2072e"]

(def decoded (l/deserialize person-or-dog-schema person-or-dog-schema encoded))

;; Note that deserialized data from an ambiguous union is returned in wrapped form
decoded
;; [:user/dog {:name "Fido", :owner "Roger"}]

;; Get the schema name portion
(l/schema-name decoded)
;; :user/dog

;; Get the data portion
(l/data decoded)
;; {:name "Fido", :owner "Roger"}

Names and Namespaces

Named Avro schemas (records, enums, fixeds, and flex-maps) contain a name part and, optionally, a namespace part. The Names section of the Avro spec describes this in detail. Lancaster fully supports the spec, allowing both names and namespaces. These are combined into a single fullname, including both the namespace (if any) and the name.

Lancaster schema names and namespaces must start with a letter and subsequently only contain letters, numbers, or hyphens.

When using the Schema Creation Macros, the name used in the schema is derived from the name of the symbol passed to the def-*-schema macro. It the symbol ends with -schema (as is common), the -schema portion is dropped from the name. The namespace is taken from the Clojure(Script) namespace where the schema is defined.

When using the Schema Creation Functions, the name and namespace are taken from the name-kw parameter passed to the *-schema function. If the keyword is namespaced, the keyword's namespace is used as the schema's namespace. If the keyword does not have a namespace, the schema will not have a namespace. Only the functions for creating named schemas (enum-schema, fixed-schema, record-schema, flex-map-schema, and merged-record-schema) have a name-kw parameter.

In the EDN representation of a named schema, the :name attribute contains the name of the schema, including the namespace, if any. In the JSON and PCF representations, the name portion is converted from kebab-case to PascalCase, and any namespace is converted from kebab-case to snake_case. This matches the Avro spec (which does not allow hyphenated names) and provides for easy interop with other languages (Java, JS, C++, etc.)

For example, using the def-enum-schema macro:

(l/def-enum-schema suite-schema
  :clubs :diamonds :hearts :spades)

(l/edn suite-schema)
;; {:name :user/suite, :type :enum, :symbols [:clubs :diamonds :hearts :spades]}
;; Note that the :name includes the namespace (:user in this case)
;; and that the name is 'suite', not 'suite-schema'

(l/json suite-schema)
;; "{\"name\":\"user.Suite\",\"type\":\"enum\",\"symbols\":[\"CLUBS\",\"DIAMONDS\",\"HEARTS\",\"SPADES\"]}"
;; Note that the name has been converted to user.Suite

Or using the enum-schema function:

(def suite-schema
  (l/enum-schema :a-random-ns/suite [:clubs :diamonds :hearts :spades]))

(l/edn suite-schema)
;; {:name :a-random-ns/suite, :type :enum, :symbols [:clubs :diamonds :hearts :spades]}
;; Note that the namespace is not :user, but is :a-random-ns

(l/json suite-schema)
;; "{\"name\":\"a_random_ns.Suite\",\"type\":\"enum\",\"symbols\":[\"CLUBS\",\"DIAMONDS\",\"HEARTS\",\"SPADES\"]}"
;; Note that the name has been converted to a_random_ns.Suite

API Documentation

All public vars, functions, and macros are in the deercreeklabs.lancaster namespace. All other namespaces should be considered private implementation details that may change.

def-record-schema

(def-record-schema name-symbol & fields)

Defines a var whose value is a Lancaster schema object representing an Avro record. For cases where a macro is not appropriate, use the record-schema function instead.