High-performance serialization library for Clojure
High-performance serialization library for Clojure
(fn [byte-array])->compressor used by `(freeze <x> {:compressor :auto}), nil => default
(fn [byte-array])->compressor used by `(freeze <x> {:compressor :auto}),
nil => default{<hash-or-byte-id> (fn [data-input])->read}
{<hash-or-byte-id> (fn [data-input])->read}
DEPRECATED: prefer *freeze-fallback.
DEPRECATED: prefer `*freeze-fallback`.
(fn [data-output x])->freeze, nil => default
(fn [data-output x])->freeze, nil => default
Used when attempting to freeze or thaw an object that:
In this case, Java's Serializable interface will be permitted iff the predicate (serializable-whitelist <class-name>) returns true.
I.e. this is a predicate (fn allow-class? [class-name]) that specifies whether Nippy may use a given class's Serializable implementation as fallback when its own protocol is unfamiliar with the type.
If thaw encounters an unwhitelisted Serialized class:
thaw will throw if it's not possible to safely quarantine.{:nippy/unthawable {:class-name _ :content <quarantined-ba> ...}}.This is a security measure to prevent Remote Code Execution (RCE).
Default value for v2.14.2 is: (constantly true).
Default value for v2.15.x is: #{}.
Value may be overridden with swap-serializable-whitelist! or with:
taoensso.nippy.serializable-whitelist-base JVM property
taoensso.nippy.serializable-whitelist-add JVM property
TAOENSSO_NIPPY_SERIALIZABLE_WHITELIST_BASE env var
TAOENSSO_NIPPY_SERIALIZABLE_WHITELIST_ADD env var
If present, these will be read as comma-separated lists of class names and formed into sets. Initial whitelist value will then be: (into (or <?base> <default>) <?additions>).
I.e. you can use: - The "base" property/var to override Nippy's default whitelist. - The "add" property/var to add to Nippy's default whitelist.
Strings in sets may contain "*" wildcards.
See also taoensso.encore/compile-str-filter, a util to help
easily build more advanced predicate functions.
================ Further context:
Reading arbitrary Serializable classes can be dangerous if they come from an untrusted source.
Specifically: if your classpath contains a vulnerable ("gadget")[2] class - it is possible for an attacker to produce an object that can run arbitrary code when read via Serializable.
Note that Clojure <= 1.8 itself contains such a class [1].
What to use as a whitelist?
If you DO NOT wish to support Serializable: #{} is safest,
and just entirely disallows its use.
If you DO with to support Serializable:
2a. If you might serialize data from an untrusted source, or
if you'll only be serializing a limited number of known
classes: enumerate those class names, e.g.:
`#{"java.lang.Throwable", ...}`.
2b. If you're CERTAIN to NEVER serialize data from an untrusted
source, you can use `(constantly true)` as predicate. This
will whitelist everything, allowing Serializable for ANY class.
Upgrading from an older version of Nippy and not sure whether you've been using Nippy's Serializable support? Here's a code snippet that will allow AND RECORD any class using Nippy's Serializable fallback:
;; Deref for set of all class names that made use of Nippy's Serializable support: (defonce observed-serializables_ (atom #{}))
(swap-serializable-whitelist! (fn [_] (fn allow-class? [class-name] (swap! observed-serializables_ conj class-name) ; Record class name true ; Allow any class )))
Thanks to Timo Mihaljov (@solita-timo-mihaljov) for an excellent report identifying this vulnerability.
[1] https://groups.google.com/forum/#!msg/clojure/WaL3hHzsevI/7zHU-L7LBQAJ [2] Jackson maintains a list of common gadget classes at https://github.com/FasterXML/jackson-databind/blob/master/src/main/java/com/fasterxml/jackson/databind/jsontype/impl/SubTypeValidator.java
Used when attempting to freeze or thaw an object that:
- Does not implement Nippy's Freezable protocol.
- Does implement Java's Serializable interface.
In this case, Java's Serializable interface will be permitted iff
the predicate (*serializable-whitelist* <class-name>) returns true.
I.e. this is a predicate (fn allow-class? [class-name]) that specifies
whether Nippy may use a given class's Serializable implementation as
fallback when its own protocol is unfamiliar with the type.
If `thaw` encounters an unwhitelisted Serialized class:
- `thaw` will throw if it's not possible to safely quarantine.
- Otherwise the object will be thawed as:
`{:nippy/unthawable {:class-name _ :content <quarantined-ba> ...}}`.
This is a security measure to prevent Remote Code Execution (RCE).
Default value for v2.14.2 is: `(constantly true)`.
Default value for v2.15.x is: `#{}`.
Value may be overridden with `swap-serializable-whitelist!` or with:
- `taoensso.nippy.serializable-whitelist-base` JVM property
- `taoensso.nippy.serializable-whitelist-add` JVM property
- `TAOENSSO_NIPPY_SERIALIZABLE_WHITELIST_BASE` env var
- `TAOENSSO_NIPPY_SERIALIZABLE_WHITELIST_ADD` env var
If present, these will be read as comma-separated lists of class
names and formed into sets. Initial whitelist value will then be:
(into (or <?base> <default>) <?additions>).
I.e. you can use:
- The "base" property/var to override Nippy's default whitelist.
- The "add" property/var to add to Nippy's default whitelist.
Strings in sets may contain "*" wildcards.
See also `taoensso.encore/compile-str-filter`, a util to help
easily build more advanced predicate functions.
================
Further context:
Reading arbitrary Serializable classes can be dangerous if they
come from an untrusted source.
Specifically: if your classpath contains a vulnerable ("gadget")[2]
class - it is possible for an attacker to produce an object that
can run arbitrary code when read via Serializable.
Note that Clojure <= 1.8 itself contains such a class [1].
What to use as a whitelist?
1. If you DO NOT wish to support Serializable: `#{}` is safest,
and just entirely disallows its use.
2. If you DO with to support Serializable:
2a. If you might serialize data from an untrusted source, or
if you'll only be serializing a limited number of known
classes: enumerate those class names, e.g.:
`#{"java.lang.Throwable", ...}`.
2b. If you're CERTAIN to NEVER serialize data from an untrusted
source, you can use `(constantly true)` as predicate. This
will whitelist everything, allowing Serializable for ANY class.
Upgrading from an older version of Nippy and not sure whether you've
been using Nippy's Serializable support? Here's a code snippet that
will allow AND RECORD any class using Nippy's Serializable fallback:
;; Deref for set of all class names that made use of Nippy's Serializable support:
(defonce observed-serializables_ (atom #{}))
(swap-serializable-whitelist!
(fn [_]
(fn allow-class? [class-name]
(swap! observed-serializables_ conj class-name) ; Record class name
true ; Allow any class
)))
Thanks to Timo Mihaljov (@solita-timo-mihaljov) for an excellent report
identifying this vulnerability.
[1] https://groups.google.com/forum/#!msg/clojure/WaL3hHzsevI/7zHU-L7LBQAJ
[2] Jackson maintains a list of common gadget classes at
https://github.com/FasterXML/jackson-databind/blob/master/src/main/java/com/fasterxml/jackson/databind/jsontype/impl/SubTypeValidator.java{[<x> <meta>] <idx>} for freezing, {<idx> <x-with-meta>} for thawing.
{[<x> <meta>] <idx>} for freezing, {<idx> <x-with-meta>} for thawing.
Default 128bit AES-CBC encryptor with many-round SHA-512 key-gen.
See also aes-128-cbc-encryptor.
Default 128bit AES-CBC encryptor with many-round SHA-512 key-gen. See also `aes-128-cbc-encryptor`.
Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen.
USE CASE: You want more than a small, finite number of passwords (e.g. each item encrypted will use a unique user-provided password).
IMPLEMENTATION: Uses a relatively cheap key hash, but automatically salts every key.
PROS: Each key is independent so would need to be attacked independently. CONS: Key caching impossible, so there's an inherent trade-off between encryption/decryption speed and the difficulty of attacking any particular key.
Slower than aes128-cached, and easier to attack any particular key - but
keys are independent.
USE CASE: You want only a small, finite number of passwords (e.g. a limited number of staff/admins, or you'll be using a single password to encrypt many items).
IMPLEMENTATION: Uses a very expensive (but cached) key hash, and no salt.
PROS: Great amortized encryption/decryption speed. Expensive key hash makes attacking any particular key very difficult. CONS: Using a small number of keys for many encrypted items means that if any key is somehow compromised, all items encrypted with that key are compromised.
Faster than aes128-salted, and harder to attack any particular key - but
increased danger if a key is somehow compromised.
Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen.
Password form [:salted "my-password"]
---------------------------------------
USE CASE: You want more than a small, finite number of passwords (e.g. each
item encrypted will use a unique user-provided password).
IMPLEMENTATION: Uses a relatively cheap key hash, but automatically salts
every key.
PROS: Each key is independent so would need to be attacked independently.
CONS: Key caching impossible, so there's an inherent trade-off between
encryption/decryption speed and the difficulty of attacking any
particular key.
Slower than `aes128-cached`, and easier to attack any particular key - but
keys are independent.
Password form [:cached "my-password"]
---------------------------------------
USE CASE: You want only a small, finite number of passwords (e.g. a limited
number of staff/admins, or you'll be using a single password to
encrypt many items).
IMPLEMENTATION: Uses a _very_ expensive (but cached) key hash, and no salt.
PROS: Great amortized encryption/decryption speed. Expensive key hash makes
attacking any particular key very difficult.
CONS: Using a small number of keys for many encrypted items means that if any
key _is_ somehow compromised, _all_ items encrypted with that key are
compromised.
Faster than `aes128-salted`, and harder to attack any particular key - but
increased danger if a key is somehow compromised.Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen.
USE CASE: You want more than a small, finite number of passwords (e.g. each item encrypted will use a unique user-provided password).
IMPLEMENTATION: Uses a relatively cheap key hash, but automatically salts every key.
PROS: Each key is independent so would need to be attacked independently. CONS: Key caching impossible, so there's an inherent trade-off between encryption/decryption speed and the difficulty of attacking any particular key.
Slower than aes128-cached, and easier to attack any particular key - but
keys are independent.
USE CASE: You want only a small, finite number of passwords (e.g. a limited number of staff/admins, or you'll be using a single password to encrypt many items).
IMPLEMENTATION: Uses a very expensive (but cached) key hash, and no salt.
PROS: Great amortized encryption/decryption speed. Expensive key hash makes attacking any particular key very difficult. CONS: Using a small number of keys for many encrypted items means that if any key is somehow compromised, all items encrypted with that key are compromised.
Faster than aes128-salted, and harder to attack any particular key - but
increased danger if a key is somehow compromised.
Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen.
Password form [:salted "my-password"]
---------------------------------------
USE CASE: You want more than a small, finite number of passwords (e.g. each
item encrypted will use a unique user-provided password).
IMPLEMENTATION: Uses a relatively cheap key hash, but automatically salts
every key.
PROS: Each key is independent so would need to be attacked independently.
CONS: Key caching impossible, so there's an inherent trade-off between
encryption/decryption speed and the difficulty of attacking any
particular key.
Slower than `aes128-cached`, and easier to attack any particular key - but
keys are independent.
Password form [:cached "my-password"]
---------------------------------------
USE CASE: You want only a small, finite number of passwords (e.g. a limited
number of staff/admins, or you'll be using a single password to
encrypt many items).
IMPLEMENTATION: Uses a _very_ expensive (but cached) key hash, and no salt.
PROS: Great amortized encryption/decryption speed. Expensive key hash makes
attacking any particular key very difficult.
CONS: Using a small number of keys for many encrypted items means that if any
key _is_ somehow compromised, _all_ items encrypted with that key are
compromised.
Faster than `aes128-salted`, and harder to attack any particular key - but
increased danger if a key is somehow compromised.(cache x)Experimental, subject to change.
Wraps value so that future writes of the same wrapped value with same metadata will be efficiently encoded as references to this one.
(freeze [(cache "foo") (cache "foo") (cache "foo")]) will incl. a single "foo", plus 2x single-byte references to "foo".
Experimental, subject to change. Wraps value so that future writes of the same wrapped value with same metadata will be efficiently encoded as references to this one. (freeze [(cache "foo") (cache "foo") (cache "foo")]) will incl. a single "foo", plus 2x single-byte references to "foo".
(compress compressor ba)(decompress compressor ba)(decrypt encryptor pwd ba)(encrypt encryptor pwd ba)(extend-freeze type custom-type-id [x out] & body)Extends Nippy to support freezing of a custom type (ideally concrete) with given id of form:
NB: be careful about extending to interfaces, Ref. http://goo.gl/6gGRlU.
(defrecord MyRec [data]) (extend-freeze MyRec :foo/my-type [x data-output] ; Keyword id (.writeUTF [data-output] (:data x))) ;; or (extend-freeze MyRec 1 [x data-output] ; Byte id (.writeUTF [data-output] (:data x)))
Extends Nippy to support freezing of a custom type (ideally concrete) with given id of form: * Keyword - 2 byte overhead, keywords hashed to 16 bit id * ℕ∈[1, 128] - 0 byte overhead NB: be careful about extending to interfaces, Ref. http://goo.gl/6gGRlU. (defrecord MyRec [data]) (extend-freeze MyRec :foo/my-type [x data-output] ; Keyword id (.writeUTF [data-output] (:data x))) ;; or (extend-freeze MyRec 1 [x data-output] ; Byte id (.writeUTF [data-output] (:data x)))
(extend-thaw custom-type-id [in] & body)Extends Nippy to support thawing of a custom type with given id: (extend-thaw :foo/my-type [data-input] ; Keyword id (MyRec. (.readUTF data-input))) ;; or (extend-thaw 1 [data-input] ; Byte id (MyRec. (.readUTF data-input)))
Extends Nippy to support thawing of a custom type with given id: (extend-thaw :foo/my-type [data-input] ; Keyword id (MyRec. (.readUTF data-input))) ;; or (extend-thaw 1 [data-input] ; Byte id (MyRec. (.readUTF data-input)))
(fast-freeze x)Like freeze but:
fast-thawEquivalent to (but a little faster than): `(freeze x {:compressor nil :encryptor nil :no-header? true})
Like `freeze` but:
- Writes data without a Nippy header
- Drops all support for compression and encryption
- Must be thawed with `fast-thaw`
Equivalent to (but a little faster than):
`(freeze x {:compressor nil :encryptor nil :no-header? true})(fast-thaw ba)Like thaw but:
fast-freezeEquivalent to (but a little faster than): `(thaw x {:compressor nil :encryptor nil :no-header? true})
Like `thaw` but:
- Drops all support for compression and encryption
- Supports only data frozen with `fast-freeze`
Equivalent to (but a little faster than):
`(thaw x {:compressor nil :encryptor nil :no-header? true})(freezable? x)(freezable? x {:keys [allow-clojure-reader? allow-java-serializable?]})Alpha - subject to change. Returns truthy iff Nippy appears to support freezing the given argument.
:allow-clojure-reader? and :allow-java-serializable? options may be
used to enable the relevant roundtrip fallback test(s). These tests are
only moderately reliable since they're cached by arg type and don't
test for pre/post serialization value equality (there's no good general
way of doing so).
Alpha - subject to change. Returns truthy iff Nippy *appears* to support freezing the given argument. `:allow-clojure-reader?` and `:allow-java-serializable?` options may be used to enable the relevant roundtrip fallback test(s). These tests are only **moderately reliable** since they're cached by arg type and don't test for pre/post serialization value equality (there's no good general way of doing so).
(freeze x)(freeze x
{:as opts
:keys [compressor encryptor password]
:or {compressor :auto encryptor aes128-gcm-encryptor}})Serializes arg (any Clojure data type) to a byte array. To freeze custom
types, extend the Clojure reader or see extend-freeze.
Serializes arg (any Clojure data type) to a byte array. To freeze custom types, extend the Clojure reader or see `extend-freeze`.
(freeze-to-file file x)(freeze-to-file file x freeze-opts)Convenience util: like freeze, but writes to (clojure.java.io/file <file>)
and returns the byte array written.
See also thaw-from-file.
Convenience util: like `freeze`, but writes to `(clojure.java.io/file <file>)` and returns the byte array written. See also `thaw-from-file`.
(freeze-to-out! data-output x)Serializes arg (any Clojure data type) to a DataOutput.
This is a low-level util: in most cases you'll want freeze instead.
Serializes arg (any Clojure data type) to a DataOutput. This is a low-level util: in most cases you'll want `freeze` instead.
(freeze-to-string x)(freeze-to-string x freeze-opts)Convenience util: like freeze, but returns a Base64-encoded string.
See also thaw-from-string.
Convenience util: like `freeze`, but returns a Base64-encoded string. See also `thaw-from-string`.
(-freeze-without-meta! x data-output)(-freeze-with-meta! x data-output)(inspect-ba ba)(inspect-ba ba thaw-opts)Alpha - subject to change
Alpha - subject to change
Default net.jpountz.lz4 compressor: Ratio: low. Write speed: very high. Read speed: very high.
A good general-purpose compressor, competitive with Snappy.
Thanks to Max Penet (@mpenet) for our first implementation, Ref. https://github.com/mpenet/nippy-lz4
Default net.jpountz.lz4 compressor:
Ratio: low.
Write speed: very high.
Read speed: very high.
A good general-purpose compressor, competitive with Snappy.
Thanks to Max Penet (@mpenet) for our first implementation,
Ref. https://github.com/mpenet/nippy-lz4Like lz4-compressor but trades some write speed for ratio.
Like `lz4-compressor` but trades some write speed for ratio.
Default org.tukaani.xz.LZMA2 compressor: Ratio: high. Write speed: very slow (also currently single-threaded). Read speed: slow.
A specialized compressor for large, low-write data in space-sensitive environments.
Default org.tukaani.xz.LZMA2 compressor:
Ratio: high.
Write speed: _very_ slow (also currently single-threaded).
Read speed: slow.
A specialized compressor for large, low-write data in space-sensitive
environments.(set-auto-freeze-compressor! x)(set-freeze-fallback! x)Default org.iq80.snappy.Snappy compressor: Ratio: low. Write speed: very high. Read speed: very high.
A good general-purpose compressor.
Default org.iq80.snappy.Snappy compressor:
Ratio: low.
Write speed: very high.
Read speed: very high.
A good general-purpose compressor.Reference data used for tests & benchmarks
Reference data used for tests & benchmarks
Reference data with stuff removed that breaks reader or other utils we'll be benching against
Reference data with stuff removed that breaks reader or other utils we'll be benching against
Reference data with stuff removed that breaks roundtrip equality
Reference data with stuff removed that breaks roundtrip equality
(swap-custom-readers! f)(swap-serializable-whitelist! f)Changes root *serializable-whitelist* value to (f old-val).
Example f arguments:
Strings in sets may contain "*" wildcards.
See also `serializable-whitelist.
Changes root `*serializable-whitelist*` value to (f old-val).
Example `f` arguments:
- (fn [_old] true) ; Whitelist everything (allow all classes)
- (fn [_old] #{}) ; Whitelist nothing (disallow all classes)
- (fn [_old] #{"java.lang.Throwable"}) ; Reset class whitelist set
- (fn [ old] (conj old "java.lang.Throwable"))) ; Add class to whitelist set
- (fn [ old] (conj old "java.lang.*")) ; Add classes to whitelist set (note wildcard)
Strings in sets may contain "*" wildcards.
See also `*serializable-whitelist*.(thaw ba)(thaw ba
{:as opts
:keys [v1-compatibility? compressor encryptor password]
:or {compressor :auto encryptor :auto}})Deserializes a frozen Nippy byte array to its original Clojure data type.
To thaw custom types, extend the Clojure reader or see extend-thaw.
** By default, supports data frozen with Nippy v2+ ONLY **
Add {:v1-compatibility? true} option to support thawing of data frozen with
legacy versions of Nippy.
Options include: :v1-compatibility? - support data frozen by legacy versions of Nippy? :compressor - :auto (checks header, default) an ICompressor, or nil :encryptor - :auto (checks header, default), an IEncryptor, or nil
Deserializes a frozen Nippy byte array to its original Clojure data type.
To thaw custom types, extend the Clojure reader or see `extend-thaw`.
** By default, supports data frozen with Nippy v2+ ONLY **
Add `{:v1-compatibility? true}` option to support thawing of data frozen with
legacy versions of Nippy.
Options include:
:v1-compatibility? - support data frozen by legacy versions of Nippy?
:compressor - :auto (checks header, default) an ICompressor, or nil
:encryptor - :auto (checks header, default), an IEncryptor, or nil(thaw-from-file file)(thaw-from-file file thaw-opts)Convenience util: like thaw, but reads from (clojure.java.io/file <file>).
To thaw from a resource on classpath (e.g in Leiningen resources dir):
(thaw-from-file (clojure.java.io/resource "my-resource-name.npy"))
See also freeze-to-file.
Convenience util: like `thaw`, but reads from `(clojure.java.io/file <file>)`. To thaw from a resource on classpath (e.g in Leiningen `resources` dir): (thaw-from-file (clojure.java.io/resource "my-resource-name.npy")) See also `freeze-to-file`.
(thaw-from-in! data-input)Deserializes a frozen object from given DataInput to its original Clojure data type.
This is a low-level util: in most cases you'll want thaw instead.
Deserializes a frozen object from given DataInput to its original Clojure data type. This is a low-level util: in most cases you'll want `thaw` instead.
(thaw-from-string s)(thaw-from-string s thaw-opts)Convenience util: like thaw, but takes a Base64-encoded string.
See also freeze-to-string.
Convenience util: like `thaw`, but takes a Base64-encoded string. See also `freeze-to-string`.
(throw-unfreezable x)(try-write-readable out x)(try-write-serializable out x)(write-id out id)(write-unfreezable out x)cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |