Liking cljdoc? Tell your friends :D

edd.nippy.core

A custom nippy-based binary encoder and decoder, also called as 'nippy-stream' in PLC3.

In PLC3, we had troubles when serializing vast Clojure collections. Apparently, we faced an issue when nippy tries to allocate more than Integer/MAX_VALUE bytes for an array, which ends up with an exception.

Another issue that nippy suffers from, is that items cannot be read lazily one by one (technically they can but there is no such an API). This is important when processing items sequentially without reading all them at once.

The encoder below is quite simple. It accepts an arbitrary collection (lazy as well) and a destination which gets coerced into a DataOutputStream. Then, each item gets binary-encoded and stored into that stream. The binary payload forms a pattern:

[oid][data][oid][data] [oid][data] |---------||---------| ... |---------| item1 item2 itemN

Where:

  • oid is a byte indicating a type in nippy;
  • data is a byte array whose length and content depends on a value type.

Since nippy relies on DataOutputStream to store primitives, most of them can be read back by calling .readInt, .readFloat, and other methods from a DataInputStream instance.

The decoding function returns a lazy sequence of items decoded back. Since it requires a stream, always place it into the (with-open ...) macro and ensure you don't process items after you have exited from it.

See examples in the development section below.

A custom nippy-based binary encoder and decoder, also
called as 'nippy-stream' in PLC3.

In PLC3, we had troubles when serializing vast Clojure
collections. Apparently, we faced an issue when nippy
tries to allocate more than `Integer/MAX_VALUE`
bytes for an array, which ends up with an exception.

Another issue that nippy suffers from, is that items
cannot be read lazily one by one (technically they can
but there is no such an API). This is important when
processing items sequentially without reading all them
at once.

The encoder below is quite simple. It accepts an arbitrary
collection (lazy as well) and a destination which gets
coerced into a `DataOutputStream`. Then, each item gets
binary-encoded and stored into that stream. The binary
payload forms a pattern:

[oid][data][oid][data]     [oid][data]
|---------||---------| ... |---------|
   item1      item2           itemN

Where:
- oid is a byte indicating a type in nippy;
- data is a byte array whose length and content depends
  on a value type.

Since nippy relies on `DataOutputStream` to store primitives,
most of them can be read back by calling `.readInt`, `.readFloat`,
and other methods from a `DataInputStream` instance.

The decoding function returns a lazy sequence of items decoded
back. Since it requires a stream, always place it into the
`(with-open ...)` macro and ensure you don't process items
after you have exited from it.

See examples in the development section below.
raw docstring

edd.nippy.vectorz

VectorZ extensions for Nippy. They have been copy-pasted across many projects and have finally centralized here.

VectorZ extensions for Nippy. They have been copy-pasted
across many projects and have finally centralized here.
raw docstring

cljdoc builds & hosts documentation for Clojure/Script libraries

Keyboard shortcuts
Ctrl+kJump to recent docs
Move to previous article
Move to next article
Ctrl+/Jump to the search field
× close