A custom nippy-based binary encoder and decoder, also called as 'nippy-stream' in PLC3.
In PLC3, we had troubles when serializing vast Clojure
collections. Apparently, we faced an issue when nippy
tries to allocate more than Integer/MAX_VALUE
bytes for an array, which ends up with an exception.
Another issue that nippy suffers from, is that items cannot be read lazily one by one (technically they can but there is no such an API). This is important when processing items sequentially without reading all them at once.
The encoder below is quite simple. It accepts an arbitrary
collection (lazy as well) and a destination which gets
coerced into a DataOutputStream. Then, each item gets
binary-encoded and stored into that stream. The binary
payload forms a pattern:
[oid][data][oid][data] [oid][data] |---------||---------| ... |---------| item1 item2 itemN
Where:
Since nippy relies on DataOutputStream to store primitives,
most of them can be read back by calling .readInt, .readFloat,
and other methods from a DataInputStream instance.
The decoding function returns a lazy sequence of items decoded
back. Since it requires a stream, always place it into the
(with-open ...) macro and ensure you don't process items
after you have exited from it.
See examples in the development section below.
A custom nippy-based binary encoder and decoder, also called as 'nippy-stream' in PLC3. In PLC3, we had troubles when serializing vast Clojure collections. Apparently, we faced an issue when nippy tries to allocate more than `Integer/MAX_VALUE` bytes for an array, which ends up with an exception. Another issue that nippy suffers from, is that items cannot be read lazily one by one (technically they can but there is no such an API). This is important when processing items sequentially without reading all them at once. The encoder below is quite simple. It accepts an arbitrary collection (lazy as well) and a destination which gets coerced into a `DataOutputStream`. Then, each item gets binary-encoded and stored into that stream. The binary payload forms a pattern: [oid][data][oid][data] [oid][data] |---------||---------| ... |---------| item1 item2 itemN Where: - oid is a byte indicating a type in nippy; - data is a byte array whose length and content depends on a value type. Since nippy relies on `DataOutputStream` to store primitives, most of them can be read back by calling `.readInt`, `.readFloat`, and other methods from a `DataInputStream` instance. The decoding function returns a lazy sequence of items decoded back. Since it requires a stream, always place it into the `(with-open ...)` macro and ensure you don't process items after you have exited from it. See examples in the development section below.
(decode input-stream)For a input stream produced by the encode function,
return a lazy sequence of decoded items. Use it under
the (with-open ...) macro.
For a input stream produced by the `encode` function, return a *lazy* sequence of decoded items. Use it under the `(with-open ...)` macro.
(decode-gzip input-stream)Like decode but wraps the input with GZIPInputStream.
Like `decode` but wraps the input with GZIPInputStream.
(encode dest x)A general encode function that accepts any value end uses a corresponding encoding method depending whether it's sequential or not.
A general encode function that accepts any value end uses a corresponding encoding method depending whether it's sequential or not.
(encode-gzip dest x)Like encode but with GZIP compression.
Like `encode` but with GZIP compression.
(encode-seq dest coll)Encode a (lazy) sequence of items into a destination which can be a file, a string path, a stream, etc. Return a number of items processed.
Encode a (lazy) sequence of items into a destination which can be a file, a string path, a stream, etc. Return a number of items processed.
(encode-val dest x)Encode a single value into a destination (a file, a byte array, an output stream, etc).
Encode a single value into a destination (a file, a byte array, an output stream, etc).
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |