taoensso.nippy — com.taoensso/nippy 3.5.0-alpha1

High-performance serialization library for Clojure.

*auto-freeze-compressor* clj

(fn [byte-array])->compressor used by `(freeze <x> {:compressor :auto}), nil => default

*custom-readers* clj

{<hash-or-byte-id> (fn [data-input])->read}

*freeze-fallback* clj

Controls Nippy's behaviour when trying to freeze an object with a type for which Nippy doesn't currently have a native (protocol) implementation.

Possible values:

1. nil (no fallback, default) Tries the following in order:

   • Freeze with Java's Serializable interface if this seems possible
   • Freeze with Clojure's reader if this seems possible
   • Throw

2. :write-unfreezable keyword Tries the following in order:

   • Freeze with Java's Serializable interface if this seems possible
   • Freeze with Clojure's reader if this seems possible
   • Freeze a {:nippy/unfreezable {:type _}} placeholder value

3. [Advanced] Custom (fn [^java.io.DataOutput out obj]) that must write an appropriate object type id and payload to the given DataOutput stream.

*freeze-serializable-allowlist* clj

Used when attempting to <freeze/thaw> an object that:

• Does NOT implement Nippy's Freezable protocol.
• DOES implement Java's Serializable interface.

In this case, an allowlist will be checked to see if Java's Serializable interface may be used.

This is a security measure to prevent possible Remote Code Execution (RCE) when thawing malicious payloads. See [1] for details.

If freeze encounters a disallowed Serializable class, it will throw. If thaw encounters a disallowed Serializable class, it will:

• Throw if it's not possible to safely quarantine the object (object was frozen with Nippy < v2.15.0-final).

• Otherwise it will return a safely quarantined object of form {:nippy/unthawable {:class-name <> :content <quarantined-ba>}}.

  • Quarantined objects may be manually unquarantined with read-quarantined-serializable-object-unsafe!.

There are 2x allowlists:

• *freeze-serializable-allowlist* ; Checked when freezing
• *thaw-serializable-allowlist* ; Checked when thawing

Example allowlist values:

• (fn allow-class? [class-name] true) ; Arbitrary predicate fn
• #{"java.lang.Throwable", "clojure.lang.*"} ; Set of class-names
• "allow-and-record" ; Special value, see [2]

Note that class-names in sets may contain "*" wildcards.

Default allowlist values are:

• default-freeze-serializable-allowlist ; {"*"} => allow any class
• default-thaw-serializable-allowlist ; A set of common safe classes

Allowlist values may be overridden with binding, alter-var-root, or:

• taoensso.nippy.<freeze/thaw>-serializable-allowlist-base JVM property value

• taoensso.nippy.<freeze/thaw>-serializable-allowlist-add JVM property value

• TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_BASE Environment variable value

• TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_ADD Environment variable value

If present, these will be read as comma-separated lists of class names and formed into sets. Each initial allowlist value will then be: (into (or <?base> <default>) <?additions>).

I.e. you can use: - The "base" property/var to REPLACE Nippy's default allowlists. - The "add" property/var to ADD TO Nippy's default allowlists.

The special "allow-and-record" value is also possible, see [2].

Upgrading from an older version of Nippy and unsure whether you've been using Nippy's Serializable support, or which classes to allow? See [2].

See also taoensso.encore/name-filter for a util to help easily build more advanced predicate functions.

Thanks to Timo Mihaljov (@solita-timo-mihaljov) for an excellent report identifying this vulnerability.

[1] https://github.com/ptaoussanis/nippy/issues/130 [2] See allow-and-record-any-serializable-class-unsafe.

*incl-metadata?* clj

Include metadata when freezing/thawing?

*thaw-serializable-allowlist* clj

Used when attempting to <freeze/thaw> an object that:

• Does NOT implement Nippy's Freezable protocol.
• DOES implement Java's Serializable interface.

In this case, an allowlist will be checked to see if Java's Serializable interface may be used.

This is a security measure to prevent possible Remote Code Execution (RCE) when thawing malicious payloads. See [1] for details.

If freeze encounters a disallowed Serializable class, it will throw. If thaw encounters a disallowed Serializable class, it will:

• Throw if it's not possible to safely quarantine the object (object was frozen with Nippy < v2.15.0-final).

• Otherwise it will return a safely quarantined object of form {:nippy/unthawable {:class-name <> :content <quarantined-ba>}}.

  • Quarantined objects may be manually unquarantined with read-quarantined-serializable-object-unsafe!.

There are 2x allowlists:

• *freeze-serializable-allowlist* ; Checked when freezing
• *thaw-serializable-allowlist* ; Checked when thawing

Example allowlist values:

• (fn allow-class? [class-name] true) ; Arbitrary predicate fn
• #{"java.lang.Throwable", "clojure.lang.*"} ; Set of class-names
• "allow-and-record" ; Special value, see [2]

Note that class-names in sets may contain "*" wildcards.

Default allowlist values are:

• default-freeze-serializable-allowlist ; {"*"} => allow any class
• default-thaw-serializable-allowlist ; A set of common safe classes

Allowlist values may be overridden with binding, alter-var-root, or:

• taoensso.nippy.<freeze/thaw>-serializable-allowlist-base JVM property value

• taoensso.nippy.<freeze/thaw>-serializable-allowlist-add JVM property value

• TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_BASE Environment variable value

• TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_ADD Environment variable value

If present, these will be read as comma-separated lists of class names and formed into sets. Each initial allowlist value will then be: (into (or <?base> <default>) <?additions>).

I.e. you can use: - The "base" property/var to REPLACE Nippy's default allowlists. - The "add" property/var to ADD TO Nippy's default allowlists.

The special "allow-and-record" value is also possible, see [2].

Upgrading from an older version of Nippy and unsure whether you've been using Nippy's Serializable support, or which classes to allow? See [2].

See also taoensso.encore/name-filter for a util to help easily build more advanced predicate functions.

Thanks to Timo Mihaljov (@solita-timo-mihaljov) for an excellent report identifying this vulnerability.

[1] https://github.com/ptaoussanis/nippy/issues/130 [2] See allow-and-record-any-serializable-class-unsafe.

*thaw-xform* clj

Experimental, subject to change. Feedback welcome!

Transducer to use when thawing standard Clojure collection types (vectors, maps, sets, lists, etc.).

Allows fast+flexible inspection and manipulation of data being thawed.

Key-val style data structures like maps will provide MapEntry args to reducing function. Use map-entry?, key, val utils for these.

Example transducers:

(map (fn [x] (println x) x)) ; Print each coll item thawed

(comp (map (fn [x] (if (= x :secret) :redacted x))) ; Replace secrets (remove (fn [x] ; Remove maps with a truthy :remove? (or (and (map? x) (:remove? x)) (and (map-entry? x) (= (key x) :remove?) (val y)))))))

Note that while this is a very powerful feature, correctly writing and debugging transducers and reducing fns can be tricky.

To help, if Nippy encounters an error while applying your xform, it will throw a detailed ExceptionInfo with message "Error thrown via *thaw-xform*" to help you debug.

-cache-proxy clj

{[<x> <meta>] <idx>} for freezing, {<idx> <x-with-meta>} for thawing.

aes128-cbc-encryptor clj

Default 128bit AES-CBC encryptor with many-round SHA-512 key-gen. See also aes-128-cbc-encryptor.

aes128-encryptor clj

Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen.

Password form [:salted "my-password"]

USE CASE: You want more than a small, finite number of passwords (e.g. each item encrypted will use a unique user-provided password).

IMPLEMENTATION: Uses a relatively cheap key hash, but automatically salts every key.

PROS: Each key is independent so would need to be attacked independently. CONS: Key caching impossible, so there's an inherent trade-off between encryption/decryption speed and the difficulty of attacking any particular key.

Slower than aes128-cached, and easier to attack any particular key - but keys are independent.

Password form [:cached "my-password"]

USE CASE: You want only a small, finite number of passwords (e.g. a limited number of staff/admins, or you'll be using a single password to encrypt many items).

IMPLEMENTATION: Uses a very expensive (but cached) key hash, and no salt.

PROS: Great amortized encryption/decryption speed. Expensive key hash makes attacking any particular key very difficult. CONS: Using a small number of keys for many encrypted items means that if any key is somehow compromised, all items encrypted with that key are compromised.

Faster than aes128-salted, and harder to attack any particular key - but increased danger if a key is somehow compromised.

aes128-gcm-encryptor clj

Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen.

Password form [:salted "my-password"]

USE CASE: You want more than a small, finite number of passwords (e.g. each item encrypted will use a unique user-provided password).

IMPLEMENTATION: Uses a relatively cheap key hash, but automatically salts every key.

PROS: Each key is independent so would need to be attacked independently. CONS: Key caching impossible, so there's an inherent trade-off between encryption/decryption speed and the difficulty of attacking any particular key.

Slower than aes128-cached, and easier to attack any particular key - but keys are independent.

Password form [:cached "my-password"]

USE CASE: You want only a small, finite number of passwords (e.g. a limited number of staff/admins, or you'll be using a single password to encrypt many items).

IMPLEMENTATION: Uses a very expensive (but cached) key hash, and no salt.

PROS: Great amortized encryption/decryption speed. Expensive key hash makes attacking any particular key very difficult. CONS: Using a small number of keys for many encrypted items means that if any key is somehow compromised, all items encrypted with that key are compromised.

Faster than aes128-salted, and harder to attack any particular key - but increased danger if a key is somehow compromised.

allow-and-record-any-serializable-class-unsafe clj

(allow-and-record-any-serializable-class-unsafe class-name)

A predicate (fn allow-class? [class-name]) fn that can be assigned to *freeze-serializable-allowlist* and/or *thaw-serializable-allowlist* that:

• Will allow ANY class to use Nippy's Serializable support (unsafe).
• And will record {<class-name> <frequency-allowed>} for the <=1000 classes that ~most frequently made use of this support.

get-recorded-serializable-classes returns the recorded state.

This predicate is provided as a convenience for users upgrading from previous versions of Nippy that allowed the use of Serializable for all classes by default.

While transitioning from an unsafe->safe configuration, you can use this predicate (unsafe) to record information about which classes have been using Nippy's Serializable support in your environment.

Once some time has passed, you can check the recorded state. If you're satisfied that all recorded classes are safely Serializable, you can then merge the recorded classes into Nippy's default allowlist/s, e.g.:

(alter-var-root #'thaw-serializable-allowlist* (fn [_] (into default-thaw-serializable-allowlist (keys (get-recorded-serializable-classes)))))

cache clj

(cache x)

Wraps value so that future writes of the same wrapped value with same metadata will be efficiently encoded as references to this one.

(freeze [(cache "foo") (cache "foo") (cache "foo")]) will incl. a single "foo", plus 2x single-byte references to "foo".

compress clj

(compress compressor ba)

decompress clj

(decompress compressor ba)

decrypt clj

(decrypt encryptor pwd ba)

default-freeze-serializable-allowlist clj

Allows any class-name to be frozen using Java's Serializable interface. This is generally safe since RCE risk is present only when thawing. See also *freeze-serializable-allowlist*.

default-thaw-serializable-allowlist clj

A set of common safe class-names to allow to be frozen using Java's Serializable interface. PRs welcome for additions. See also *thaw-serializable-allowlist*.

encrypt clj

(encrypt encryptor pwd ba)

extend-freeze clj macro

(extend-freeze type custom-type-id [x out] & body)

Extends Nippy to support freezing of a custom type (ideally concrete) with given id of form:

• ℕ∈[1, 128] - 0 byte overhead. You are responsible for managing ids.
• (Namespaced) keyword - 2 byte overhead. Keyword will be hashed to 16 bit int, collisions will throw at compile-time.

NB: be careful about extending to interfaces, Ref. http://goo.gl/6gGRlU.

(defrecord MyRec [data]) (extend-freeze MyRec :foo/my-type [x data-output] ; Keyword id (.writeUTF [data-output] (:data x))) ;; or (extend-freeze MyRec 1 [x data-output] ; Byte id (.writeUTF [data-output] (:data x)))

extend-thaw clj macro

(extend-thaw custom-type-id [in] & body)

Extends Nippy to support thawing of a custom type with given id: (extend-thaw :foo/my-type [data-input] ; Keyword id (MyRec. (.readUTF data-input))) ;; or (extend-thaw 1 [data-input] ; Byte id (MyRec. (.readUTF data-input)))

fast-freeze clj

(fast-freeze x)

Like freeze but:

• Writes data without a Nippy header
• Drops all support for compression and encryption
• Must be thawed with fast-thaw

Equivalent to (but a little faster than) freeze with opts {:no-header? true, :compressor nil, :encryptor nil}.

Intended for use only by advanced users that clearly understand the tradeoffs. I STRONGLY recommend that most users prefer the standard freeze since:

• The Nippy header is useful for data portability and preservation
• Compression is often benefitial at little/no cost
• The performance difference between freeze and fast-freeze is often negligible in practice.

fast-thaw clj

(fast-thaw ba)

Like thaw but:

• Supports only data frozen with fast-freeze
• Drops all support for compression and encryption

Equivalent to (but a little faster than) thaw with opts: {:no-header? true, :compressor nil, :encryptor nil}.

freezable? clj

(freezable? x)

(freezable? x
            {:as opts
             :keys [recursive? allow-clojure-reader? allow-java-serializable?]
             :or {recursive? true}})

Alpha, subject to change. Returns ∈ #{:native :maybe-clojure-reader :maybe-java-serializable nil}, truthy iff Nippy seems to support freezing for the given argument.

Important: result can be inaccurate in some cases. To be completely sure you unfortunately need to try freeze then thaw the argument, and check the thawed value.

Options include: recursive? - Check recursively into given arg? allow-clojure-reader? - Allow freezing with Clojure's reader? allow-java-serializable? - Allow freezing with Java's Serializable?

freeze clj

(freeze x)

(freeze x
        {:as opts
         :keys [compressor encryptor password serializable-allowlist
                incl-metadata?]
         :or {compressor :auto encryptor aes128-gcm-encryptor}})

Serializes arg (any Clojure data type) to a byte array. To freeze custom types, extend the Clojure reader or see extend-freeze.

freeze-to-file clj

(freeze-to-file file x)

(freeze-to-file file x freeze-opts)

Convenience util: like freeze, but writes to (clojure.java.io/file <file>).

freeze-to-out! clj

(freeze-to-out! data-output x)

Serializes arg (any Clojure data type) to a DataOutput. This is a low-level util: in most cases you'll want freeze instead.

freeze-to-string clj

(freeze-to-string x)

(freeze-to-string x freeze-opts)

Convenience util: like freeze, but returns a Base64-encoded string. See also thaw-from-string.

get-recorded-serializable-classes clj

(get-recorded-serializable-classes)

Returns {<class-name> <frequency>} of the <=1000 classes that ~most frequently made use of Nippy's Serializable support via allow-and-record-any-serializable-class-unsafe.

See that function's docstring for more info.

IFreezable clj protocol

Private implementation detail. Protocol that types must implement to support native freezing by Nippy. Don't use this directly, instead see extend-freeze.

-freezable? clj

(-freezable? _)

-freeze-without-meta! clj

(-freeze-without-meta! _ data-output)

IFreezableWithMeta clj protocol

Private implementation detail. Wrapper protocol around IFreezable used to handle possible metadata.

-freeze-with-meta! clj

(-freeze-with-meta! _ data-output)

inspect-ba clj

(inspect-ba ba)

(inspect-ba ba thaw-opts)

Experimental, subject to change. Feedback welcome!

lz4-compressor clj

Default LZ4 compressor:

• Compression ratio: C (0.58 on reference benchmark).
• Compression speed: A (240 msecs on reference benchmark).
• Decompression speed: A+ (30 msecs on reference benchmark).

Good general-purpose compressor, favours speed. See taoensso.nippy-benchmarks for detailed comparative benchmarks.

lzma2-compressor clj

Default LZMA2 compressor:

• Compression ratio: A+ (0.4 on reference benchmark).
• Compression speed: E (18.5 secs on reference benchmark).
• Decompression speed: D (12 secs on reference benchmark).

Specialized compressor, strongly favours ratio. See taoensso.nippy-benchmarks for detailed comparative benchmarks.

public-types-spec clj

Public representation of Nippy's internal type schema. For use by tooling and advanced users.

HIGHLY EXPERIMENTAL! Subject to breaking change without notice. Currently completely untested, may contain bugs. Intended for use only by early adopters to give design feedback.

Format: {<type-id> {:keys [type-id type-kw payload-spec deprecated?]}},

• type-id: A +ive single-byte identifier like 110. -ive type ids are reserved for custom user-defined types.

• type-kw: A keyword like :kw-sm, suffixes used to differentiate subtypes of different sizes: -0 ; Empty => 0 byte payload / element-count -sm ; Small => 1 byte (byte) payload / element-count -md ; Medium => 2 byte (short) payload / element-count -lg ; Large => 4 byte (int) payload / element-count -xl ; Extra large => 8 byte (long) payload / element-count

• payload-spec examples:

  • nil ; No spec available (e.g. unpredictable payload)

  • [] ; Type has no payload

  • [[:bytes 4]] ; Type has payload of exactly 4 bytes

  • [[:bytes 2] [:elements 2]] ; Type has payload of exactly 2 bytes, ; followed by 2 elements

  • [[:bytes {:read 2}] [:elements {:read 4 :multiplier 2 :unsigned? true}]]

    ; Type has payload of <short-count> bytes, followed by ; <unsigned-int-count>*2 (multiplier) elements

  Note that payload-spec can be handy for skipping over items in data stream without fully reading every item.

read-quarantined-serializable-object-unsafe! clj

(read-quarantined-serializable-object-unsafe! m)

Given a quarantined Serializable object like {:nippy/unthawable {:class-name <> :content <quarantined-ba>}}, reads and returns the object WITHOUT regard for *thaw-serializable-allowlist*.

MAY BE UNSAFE! Don't call this unless you absolutely trust the payload to not contain any malicious code.

See *thaw-serializable-allowlist* for more info.

snappy-compressor clj

Default Snappy compressor:

• Compression ratio: C (0.58 on reference benchmark).
• Compression speed: A+ (210 msecs on reference benchmark).
• Decompression speed: B (130 msecs on reference benchmark). Good general-purpose compressor, favours speed. See taoensso.nippy-benchmarks for detailed comparative benchmarks.

stress-data clj

(stress-data {:keys [comparable?] :as opts})

Returns map of reference stress data for use by tests, benchmarks, etc.

thaw clj

(thaw ba)

(thaw ba
      {:as opts
       :keys [v1-compatibility? compressor encryptor password
              serializable-allowlist incl-metadata? thaw-xform]
       :or {compressor :auto encryptor :auto}})

Deserializes a frozen Nippy byte array to its original Clojure data type. To thaw custom types, extend the Clojure reader or see extend-thaw.

** By default, supports data frozen with Nippy v2+ ONLY ** Add {:v1-compatibility? true} option to support thawing of data frozen with legacy versions of Nippy.

Options include: :v1-compatibility? - support data frozen by legacy versions of Nippy? :compressor - :auto (checks header, default) an ICompressor, or nil :encryptor - :auto (checks header, default), an IEncryptor, or nil

thaw-from-file clj

(thaw-from-file file)

(thaw-from-file file thaw-opts)

Convenience util: like thaw, but reads from (clojure.java.io/file <file>).

thaw-from-in! clj

(thaw-from-in! data-input)

Deserializes a frozen object from given DataInput to its original Clojure data type.

This is a low-level util: in most cases you'll want thaw instead.

thaw-from-resource clj

(thaw-from-resource res)

(thaw-from-resource res thaw-opts)

Convenience util: like thaw, but reads from (clojure.java.io/resource <res>).

thaw-from-string clj

(thaw-from-string s)

(thaw-from-string s thaw-opts)

Convenience util: like thaw, but takes a Base64-encoded string. See also freeze-to-string.

try-write-readable clj deprecated

(try-write-readable out x)

try-write-serializable clj deprecated

(try-write-serializable out x)

with-cache clj macro

(with-cache & body)

Executes body with support for freezing/thawing cached values.

This is a low-level util: you won't need to use this yourself unless you're using freeze-to-out! or thaw-from-in! (also low-level utils).

See also cache.

write-id clj macro

(write-id out id)

write-unfreezable clj

(write-unfreezable out x)

zstd-compressor clj

Default Zstd (Zstandard) compressor:

• Compression ratio: B (0.53 on reference benchmark).
• Compression speed: C (1300 msecs on reference benchmark).
• Decompression speed: B (400 msecs on reference benchmark).

Good general-purpose compressor, balances ratio & speed. See taoensso.nippy-benchmarks for detailed comparative benchmarks.