taoensso.nippy — com.taoensso/nippy 3.0.0-alpha1

High-performance serialization library for Clojure

*auto-freeze-compressor* clj

(fn [byte-array])->compressor used by `(freeze <x> {:compressor :auto}), nil => default

*custom-readers* clj

{<hash-or-byte-id> (fn [data-input])->read}

*final-freeze-fallback* clj

DEPRECATED: prefer *freeze-fallback.

*freeze-fallback* clj

(fn [data-output x])->freeze, nil => default

*freeze-serializable-allowlist* clj

Used when attempting to <freeze/thaw> an object that:

• Does not implement Nippy's Freezable protocol.
• Does implement Java's Serializable interface.

In this case, Java's Serializable interface will be permitted iff (<allowlist> <class-name>) predicate call returns true.

This is a security measure to prevent possible Remote Code Execution (RCE) when thawing malicious payloads. See [1] for details.

If freeze encounters a disallowed Serialized class, it will throw. If thaw encounters a disallowed Serialized class, it will:

• Throw if it's not possible to safely quarantine the object (object was frozen with Nippy < v2.15.0-final).

• Otherwise it will return a safely quarantined object of form {:nippy/unthawable {:class-name <> :content <quarantined-ba>}}.

  • Quarantined objects may be manually unquarantined with read-quarantined-serializable-object-unsafe!.

There are 2x allowlists:

• *freeze-serializable-allowlist* ; Checked when freezing
• *thaw-serializable-allowlist* ; Checked when thawing

Example values:

• (fn allow-class? [class-name] true) ; Arbitrary predicate fn
• #{"java.lang.Throwable", "clojure.lang.*"} ; Set of class-names

Note that class-names in sets may contain "*" wildcards.

Default allowlist values are:

• default-freeze-serializable-allowlist ; {"*"} => allow any class
• default-thaw-serializable-allowlist ; A set of common safe classes

Allowlist values may be overridden with binding, alter-var-root, or:

• taoensso.nippy.<freeze/thaw>-serializable-allowlist-base JVM property

• taoensso.nippy.<freeze/thaw>-serializable-allowlist-add JVM property

• TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_BASE env var

• TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_ADD env var

If present, these will be read as comma-separated lists of class names and formed into sets. Each initial allowlist value will then be: (into (or <?base> <default>) <?additions>).

I.e. you can use: - The "base" property/var to replace Nippy's default allowlists. - The "add" property/var to add to Nippy's default allowlists.

See also taoensso.encore/compile-str-filter, a util to help easily build more advanced predicate functions.

Upgrading from an older version of Nippy and unsure whether you've been using Nippy's Serializable support? Here's a snippet to ALLOW and RECORD any class requesting Nippy's Serializable fallback:

;; Deref for set of all class names that made use of Nippy's Serializable support: (defonce observed-serializables_ (atom #{}))

(let [f (fn allow-class? [class-name] (swap! observed-serializables_ conj class-name) ; Record class name true ; Allow any class )]

(alter-var-root #'*freeze-serializable-allowlist* (fn [_] f))
(alter-var-root   #'*thaw-serializable-allowlist* (fn [_] f)))

Thanks to Timo Mihaljov (@solita-timo-mihaljov) for an excellent report identifying this vulnerability.

[1] https://github.com/ptaoussanis/nippy/issues/130

*incl-metadata?* clj

Include metadata when freezing/thawing?

*thaw-serializable-allowlist* clj

Used when attempting to <freeze/thaw> an object that:

• Does not implement Nippy's Freezable protocol.
• Does implement Java's Serializable interface.

In this case, Java's Serializable interface will be permitted iff (<allowlist> <class-name>) predicate call returns true.

This is a security measure to prevent possible Remote Code Execution (RCE) when thawing malicious payloads. See [1] for details.

If freeze encounters a disallowed Serialized class, it will throw. If thaw encounters a disallowed Serialized class, it will:

• Throw if it's not possible to safely quarantine the object (object was frozen with Nippy < v2.15.0-final).

• Otherwise it will return a safely quarantined object of form {:nippy/unthawable {:class-name <> :content <quarantined-ba>}}.

  • Quarantined objects may be manually unquarantined with read-quarantined-serializable-object-unsafe!.

There are 2x allowlists:

• *freeze-serializable-allowlist* ; Checked when freezing
• *thaw-serializable-allowlist* ; Checked when thawing

Example values:

• (fn allow-class? [class-name] true) ; Arbitrary predicate fn
• #{"java.lang.Throwable", "clojure.lang.*"} ; Set of class-names

Note that class-names in sets may contain "*" wildcards.

Default allowlist values are:

• default-freeze-serializable-allowlist ; {"*"} => allow any class
• default-thaw-serializable-allowlist ; A set of common safe classes

Allowlist values may be overridden with binding, alter-var-root, or:

• taoensso.nippy.<freeze/thaw>-serializable-allowlist-base JVM property

• taoensso.nippy.<freeze/thaw>-serializable-allowlist-add JVM property

• TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_BASE env var

• TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_ADD env var

If present, these will be read as comma-separated lists of class names and formed into sets. Each initial allowlist value will then be: (into (or <?base> <default>) <?additions>).

I.e. you can use: - The "base" property/var to replace Nippy's default allowlists. - The "add" property/var to add to Nippy's default allowlists.

See also taoensso.encore/compile-str-filter, a util to help easily build more advanced predicate functions.

Upgrading from an older version of Nippy and unsure whether you've been using Nippy's Serializable support? Here's a snippet to ALLOW and RECORD any class requesting Nippy's Serializable fallback:

;; Deref for set of all class names that made use of Nippy's Serializable support: (defonce observed-serializables_ (atom #{}))

(let [f (fn allow-class? [class-name] (swap! observed-serializables_ conj class-name) ; Record class name true ; Allow any class )]

(alter-var-root #'*freeze-serializable-allowlist* (fn [_] f))
(alter-var-root   #'*thaw-serializable-allowlist* (fn [_] f)))

Thanks to Timo Mihaljov (@solita-timo-mihaljov) for an excellent report identifying this vulnerability.

[1] https://github.com/ptaoussanis/nippy/issues/130

-cache-proxy clj

{[<x> <meta>] <idx>} for freezing, {<idx> <x-with-meta>} for thawing.

aes128-cbc-encryptor clj

Default 128bit AES-CBC encryptor with many-round SHA-512 key-gen. See also aes-128-cbc-encryptor.

aes128-encryptor clj

Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen.

Password form [:salted "my-password"]

USE CASE: You want more than a small, finite number of passwords (e.g. each item encrypted will use a unique user-provided password).

IMPLEMENTATION: Uses a relatively cheap key hash, but automatically salts every key.

PROS: Each key is independent so would need to be attacked independently. CONS: Key caching impossible, so there's an inherent trade-off between encryption/decryption speed and the difficulty of attacking any particular key.

Slower than aes128-cached, and easier to attack any particular key - but keys are independent.

Password form [:cached "my-password"]

USE CASE: You want only a small, finite number of passwords (e.g. a limited number of staff/admins, or you'll be using a single password to encrypt many items).

IMPLEMENTATION: Uses a very expensive (but cached) key hash, and no salt.

PROS: Great amortized encryption/decryption speed. Expensive key hash makes attacking any particular key very difficult. CONS: Using a small number of keys for many encrypted items means that if any key is somehow compromised, all items encrypted with that key are compromised.

Faster than aes128-salted, and harder to attack any particular key - but increased danger if a key is somehow compromised.

aes128-gcm-encryptor clj

Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen.

Password form [:salted "my-password"]

USE CASE: You want more than a small, finite number of passwords (e.g. each item encrypted will use a unique user-provided password).

IMPLEMENTATION: Uses a relatively cheap key hash, but automatically salts every key.

PROS: Each key is independent so would need to be attacked independently. CONS: Key caching impossible, so there's an inherent trade-off between encryption/decryption speed and the difficulty of attacking any particular key.

Slower than aes128-cached, and easier to attack any particular key - but keys are independent.

Password form [:cached "my-password"]

USE CASE: You want only a small, finite number of passwords (e.g. a limited number of staff/admins, or you'll be using a single password to encrypt many items).

IMPLEMENTATION: Uses a very expensive (but cached) key hash, and no salt.

PROS: Great amortized encryption/decryption speed. Expensive key hash makes attacking any particular key very difficult. CONS: Using a small number of keys for many encrypted items means that if any key is somehow compromised, all items encrypted with that key are compromised.

Faster than aes128-salted, and harder to attack any particular key - but increased danger if a key is somehow compromised.

cache clj

(cache x)

Experimental, subject to change.

Wraps value so that future writes of the same wrapped value with same metadata will be efficiently encoded as references to this one.

(freeze [(cache "foo") (cache "foo") (cache "foo")]) will incl. a single "foo", plus 2x single-byte references to "foo".

compress clj

(compress compressor ba)

decompress clj

(decompress compressor ba)

decrypt clj

(decrypt encryptor pwd ba)

default-freeze-serializable-allowlist clj

Allows any class-name to be frozen using Java's Serializable interface. This is generally safe since RCE risk is present only when thawing. See also *freeze-serializable-allowlist*.

default-thaw-serializable-allowlist clj

A set of common safe class-names to allow to be frozen using Java's Serializable interface. PRs welcome for additions. See also *thaw-serializable-allowlist*.

encrypt clj

(encrypt encryptor pwd ba)

extend-freeze clj macro

(extend-freeze type custom-type-id [x out] & body)

Extends Nippy to support freezing of a custom type (ideally concrete) with given id of form:

• Keyword - 2 byte overhead, keywords hashed to 16 bit id
• ℕ∈[1, 128] - 0 byte overhead

NB: be careful about extending to interfaces, Ref. http://goo.gl/6gGRlU.

(defrecord MyRec [data]) (extend-freeze MyRec :foo/my-type [x data-output] ; Keyword id (.writeUTF [data-output] (:data x))) ;; or (extend-freeze MyRec 1 [x data-output] ; Byte id (.writeUTF [data-output] (:data x)))

extend-thaw clj macro

(extend-thaw custom-type-id [in] & body)

Extends Nippy to support thawing of a custom type with given id: (extend-thaw :foo/my-type [data-input] ; Keyword id (MyRec. (.readUTF data-input))) ;; or (extend-thaw 1 [data-input] ; Byte id (MyRec. (.readUTF data-input)))

fast-freeze clj

(fast-freeze x)

Like freeze but:

• Writes data without a Nippy header
• Drops all support for compression and encryption
• Must be thawed with fast-thaw

Equivalent to (but a little faster than) freeze with opts:

• :compressor nil
• :encryptor nil
• :no-header? true

fast-thaw clj

(fast-thaw ba)

Like thaw but:

• Drops all support for compression and encryption
• Supports only data frozen with fast-freeze

Equivalent to (but a little faster than) thaw with opts:

• :compressor nil
• :encryptor nil
• :no-header? true

freezable? clj

(freezable? x)

(freezable? x {:keys [allow-clojure-reader? allow-java-serializable?]})

Alpha - subject to change. Returns truthy iff Nippy appears to support freezing the given argument.

:allow-clojure-reader? and :allow-java-serializable? options may be used to enable the relevant roundtrip fallback test(s). These tests are only moderately reliable since they're cached by arg type and don't test for pre/post serialization value equality (there's no good general way of doing so).

freeze clj

(freeze x)

(freeze x
        {:as opts
         :keys [compressor encryptor password serializable-allowlist
                incl-metadata?]
         :or {compressor :auto encryptor aes128-gcm-encryptor}})

Serializes arg (any Clojure data type) to a byte array. To freeze custom types, extend the Clojure reader or see extend-freeze.

freeze-fallback-as-str clj

DEPRECATED, use write-unfreezable

freeze-to-file clj

(freeze-to-file file x)

(freeze-to-file file x freeze-opts)

Convenience util: like freeze, but writes to (clojure.java.io/file <file>) and returns the byte array written. See also thaw-from-file.

freeze-to-out! clj

(freeze-to-out! data-output x)

Serializes arg (any Clojure data type) to a DataOutput. This is a low-level util: in most cases you'll want freeze instead.

freeze-to-string clj

(freeze-to-string x)

(freeze-to-string x freeze-opts)

Convenience util: like freeze, but returns a Base64-encoded string. See also thaw-from-string.

IFreezable1 clj protocol

-freeze-without-meta! clj

(-freeze-without-meta! x data-output)

IFreezable2 clj protocol

-freeze-with-meta! clj

(-freeze-with-meta! x data-output)

inspect-ba clj

(inspect-ba ba)

(inspect-ba ba thaw-opts)

Alpha - subject to change

lz4-compressor clj

Default net.jpountz.lz4 compressor: Ratio: low. Write speed: very high. Read speed: very high.

A good general-purpose compressor, competitive with Snappy.

Thanks to Max Penet (@mpenet) for our first implementation, Ref. https://github.com/mpenet/nippy-lz4

lz4hc-compressor clj

Like lz4-compressor but trades some write speed for ratio.

lzma2-compressor clj

Default org.tukaani.xz.LZMA2 compressor: Ratio: high. Write speed: very slow (also currently single-threaded). Read speed: slow.

A specialized compressor for large, low-write data in space-sensitive environments.

read-quarantined-serializable-object-unsafe! clj

(read-quarantined-serializable-object-unsafe! m)

Given a quarantined Serializable object like {:nippy/unthawable {:class-name <> :content <quarantined-ba>}}, reads and returns the object WITHOUT regard for *thaw-serializable-allowlist*.

MAY BE UNSAFE! Don't call this unless you absolutely trust the payload to not contain any malicious code.

See *thaw-serializable-allowlist* for more info.

set-auto-freeze-compressor! clj

(set-auto-freeze-compressor! x)

DEPRECATED, just use alter-var-root

set-freeze-fallback! clj

(set-freeze-fallback! x)

DEPRECATED, just use alter-var-root

snappy-compressor clj

Default org.iq80.snappy.Snappy compressor: Ratio: low. Write speed: very high. Read speed: very high.

A good general-purpose compressor.

stress-data clj

Reference data used for tests & benchmarks

stress-data-benchable clj

Reference data with stuff removed that breaks reader or other utils we'll be benching against

stress-data-comparable clj

Reference data with stuff removed that breaks roundtrip equality

StressRecord clj

swap-custom-readers! clj

(swap-custom-readers! f)

DEPRECATED, just use alter-var-root

thaw clj

(thaw ba)

(thaw ba
      {:as opts
       :keys [v1-compatibility? compressor encryptor password
              serializable-allowlist incl-metadata?]
       :or {compressor :auto encryptor :auto}})

Deserializes a frozen Nippy byte array to its original Clojure data type. To thaw custom types, extend the Clojure reader or see extend-thaw.

** By default, supports data frozen with Nippy v2+ ONLY ** Add {:v1-compatibility? true} option to support thawing of data frozen with legacy versions of Nippy.

Options include: :v1-compatibility? - support data frozen by legacy versions of Nippy? :compressor - :auto (checks header, default) an ICompressor, or nil :encryptor - :auto (checks header, default), an IEncryptor, or nil

thaw-from-file clj

(thaw-from-file file)

(thaw-from-file file thaw-opts)

Convenience util: like thaw, but reads from (clojure.java.io/file <file>).

To thaw from a resource on classpath (e.g in Leiningen resources dir): (thaw-from-file (clojure.java.io/resource "my-resource-name.npy"))

See also freeze-to-file.

thaw-from-in! clj

(thaw-from-in! data-input)

Deserializes a frozen object from given DataInput to its original Clojure data type.

This is a low-level util: in most cases you'll want thaw instead.

thaw-from-string clj

(thaw-from-string s)

(thaw-from-string s thaw-opts)

Convenience util: like thaw, but takes a Base64-encoded string. See also freeze-to-string.

throw-unfreezable clj

(throw-unfreezable x)

try-write-readable clj

(try-write-readable out x)

try-write-serializable clj

(try-write-serializable out x)

write-id clj macro

(write-id out id)

write-unfreezable clj

(write-unfreezable out x)