A custom nippy-based binary encoder and decoder, also called as 'nippy-stream' in PLC3.
In PLC3, we had troubles when serializing vast Clojure
collections. Apparently, we faced an issue when nippy
tries to allocate more than Integer/MAX_VALUE
bytes for an array, which ends up with an exception.
Another issue that nippy suffers from, is that items cannot be read lazily one by one (technically they can but there is no such an API). This is important when processing items sequentially without reading all them at once.
The encoder below is quite simple. It accepts an arbitrary
collection (lazy as well) and a destination which gets
coerced into a DataOutputStream. Then, each item gets
binary-encoded and stored into that stream. The binary
payload forms a pattern:
[oid][data][oid][data] [oid][data] |---------||---------| ... |---------| item1 item2 itemN
Where:
Since nippy relies on DataOutputStream to store primitives,
most of them can be read back by calling .readInt, .readFloat,
and other methods from a DataInputStream instance.
The decoding function returns a lazy sequence of items decoded
back. Since it requires a stream, always place it into the
(with-open ...) macro and ensure you don't process items
after you have exited from it.
See examples in the development section below.
A custom nippy-based binary encoder and decoder, also called as 'nippy-stream' in PLC3. In PLC3, we had troubles when serializing vast Clojure collections. Apparently, we faced an issue when nippy tries to allocate more than `Integer/MAX_VALUE` bytes for an array, which ends up with an exception. Another issue that nippy suffers from, is that items cannot be read lazily one by one (technically they can but there is no such an API). This is important when processing items sequentially without reading all them at once. The encoder below is quite simple. It accepts an arbitrary collection (lazy as well) and a destination which gets coerced into a `DataOutputStream`. Then, each item gets binary-encoded and stored into that stream. The binary payload forms a pattern: [oid][data][oid][data] [oid][data] |---------||---------| ... |---------| item1 item2 itemN Where: - oid is a byte indicating a type in nippy; - data is a byte array whose length and content depends on a value type. Since nippy relies on `DataOutputStream` to store primitives, most of them can be read back by calling `.readInt`, `.readFloat`, and other methods from a `DataInputStream` instance. The decoding function returns a lazy sequence of items decoded back. Since it requires a stream, always place it into the `(with-open ...)` macro and ensure you don't process items after you have exited from it. See examples in the development section below.
VectorZ extensions for Nippy. They have been copy-pasted across many projects and have finally centralized here.
VectorZ extensions for Nippy. They have been copy-pasted across many projects and have finally centralized here.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |