Liking cljdoc? Tell your friends :D
Clojure only.

taoensso.nippy

High-performance serialization library for Clojure.

*auto-freeze-compressor*clj

(fn [byte-array])->compressor used by `(freeze <x> {:compressor :auto}),
nil => default
source

*custom-readers*clj

{<hash-or-byte-id> (fn [data-input])->read}
source

*freeze-fallback*clj

Controls Nippy's behaviour when trying to freeze an item for which Nippy
doesn't currently have a native freeze/thaw implementation.

Possible values:

  1. `nil` (no freeze-fallback, default)
     Tries the following in order:
       - Freeze with Java's `Serializable` interface if possible
       - Freeze with Clojure's reader                if possible
       - Throw

  2. `:write-unfreezable` keyword
     Tries the following in order:
       - Freeze with Java's `Serializable` interface if possible
       - Freeze with Clojure's reader                if possible
       - Freeze a {:nippy/unfreezable {:type _}} placeholder value

  3. [Advanced] Custom (fn [^java.io.DataOutput out item]) that must
     write exactly one value to the given `DataOutput` stream
source

*freeze-serializable-allowlist*clj

Used when attempting to <freeze/thaw> an object that:
  - Does NOT implement Nippy's `Freezable`    protocol.
  - DOES     implement Java's  `Serializable` interface.

In this case, an allowlist will be checked to see if Java's
`Serializable` interface may be used.

This is a security measure to prevent possible Remote Code Execution
(RCE) when thawing malicious payloads. See [1] for details.

If `freeze` encounters a disallowed `Serializable` class, it will throw.
If `thaw`   encounters a disallowed `Serializable` class, it will:

  - Throw if it's not possible to safely quarantine the object
    (object was frozen with Nippy < v2.15.0-final).

  - Otherwise it will return a safely quarantined object of form
    `{:nippy/unthawable {:class-name <> :content <quarantined-ba>}}`.
    - Quarantined objects may be manually unquarantined with
      `read-quarantined-serializable-object-unsafe!`.

There are 2x allowlists:
  - `*freeze-serializable-allowlist*` ; Checked when freezing
  -   `*thaw-serializable-allowlist*` ; Checked when thawing

Example allowlist values:
  - `(fn allow-class? [class-name] true)`            ; Arbitrary predicate fn
  - `#{"java.lang.Throwable", "clojure.lang.*"}` ; Set of class-names
  - `"allow-and-record"`                           ; Special value, see [2]

  Note that class-names in sets may contain "*" wildcards.

Default allowlist values are:
  - default-freeze-serializable-allowlist ; `{"*"}` => allow any class
  -   default-thaw-serializable-allowlist ; A set of common safe classes

Allowlist values may be overridden with `binding`, `alter-var-root`, or:

  - `taoensso.nippy.<freeze/thaw>-serializable-allowlist-base` JVM property value
  - `taoensso.nippy.<freeze/thaw>-serializable-allowlist-add`  JVM property value

  - `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_BASE` Environment variable value
  - `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_ADD`  Environment variable value

If present, these will be read as comma-separated lists of class names
and formed into sets. Each initial allowlist value will then be:
(into (or <?base> <default>) <?additions>).

  I.e. you can use:
    - The "base" property/var to REPLACE Nippy's default allowlists.
    - The "add"  property/var to ADD TO  Nippy's default allowlists.

The special `"allow-and-record"` value is also possible, see [2].

Upgrading from an older version of Nippy and unsure whether you've been
using Nippy's `Serializable` support, or which classes to allow? See [2].

See also `taoensso.encore/name-filter` for a util to help easily build
more advanced predicate functions.

Thanks to Timo Mihaljov (@solita-timo-mihaljov) for an excellent report
identifying this vulnerability.

[1] https://github.com/ptaoussanis/nippy/issues/130
[2] See `allow-and-record-any-serializable-class-unsafe`.
source

*incl-metadata?*clj

Include metadata when freezing/thawing?
source

*thaw-serializable-allowlist*clj

Used when attempting to <freeze/thaw> an object that:
  - Does NOT implement Nippy's `Freezable`    protocol.
  - DOES     implement Java's  `Serializable` interface.

In this case, an allowlist will be checked to see if Java's
`Serializable` interface may be used.

This is a security measure to prevent possible Remote Code Execution
(RCE) when thawing malicious payloads. See [1] for details.

If `freeze` encounters a disallowed `Serializable` class, it will throw.
If `thaw`   encounters a disallowed `Serializable` class, it will:

  - Throw if it's not possible to safely quarantine the object
    (object was frozen with Nippy < v2.15.0-final).

  - Otherwise it will return a safely quarantined object of form
    `{:nippy/unthawable {:class-name <> :content <quarantined-ba>}}`.
    - Quarantined objects may be manually unquarantined with
      `read-quarantined-serializable-object-unsafe!`.

There are 2x allowlists:
  - `*freeze-serializable-allowlist*` ; Checked when freezing
  -   `*thaw-serializable-allowlist*` ; Checked when thawing

Example allowlist values:
  - `(fn allow-class? [class-name] true)`            ; Arbitrary predicate fn
  - `#{"java.lang.Throwable", "clojure.lang.*"}` ; Set of class-names
  - `"allow-and-record"`                           ; Special value, see [2]

  Note that class-names in sets may contain "*" wildcards.

Default allowlist values are:
  - default-freeze-serializable-allowlist ; `{"*"}` => allow any class
  -   default-thaw-serializable-allowlist ; A set of common safe classes

Allowlist values may be overridden with `binding`, `alter-var-root`, or:

  - `taoensso.nippy.<freeze/thaw>-serializable-allowlist-base` JVM property value
  - `taoensso.nippy.<freeze/thaw>-serializable-allowlist-add`  JVM property value

  - `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_BASE` Environment variable value
  - `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_ADD`  Environment variable value

If present, these will be read as comma-separated lists of class names
and formed into sets. Each initial allowlist value will then be:
(into (or <?base> <default>) <?additions>).

  I.e. you can use:
    - The "base" property/var to REPLACE Nippy's default allowlists.
    - The "add"  property/var to ADD TO  Nippy's default allowlists.

The special `"allow-and-record"` value is also possible, see [2].

Upgrading from an older version of Nippy and unsure whether you've been
using Nippy's `Serializable` support, or which classes to allow? See [2].

See also `taoensso.encore/name-filter` for a util to help easily build
more advanced predicate functions.

Thanks to Timo Mihaljov (@solita-timo-mihaljov) for an excellent report
identifying this vulnerability.

[1] https://github.com/ptaoussanis/nippy/issues/130
[2] See `allow-and-record-any-serializable-class-unsafe`.
source

*thaw-xform*clj

Experimental, subject to change. Feedback welcome!

Transducer to use when thawing standard Clojure collection types
(vectors, maps, sets, lists, etc.).

Allows fast+flexible inspection and manipulation of data being thawed.

Key-val style data structures like maps will provide `MapEntry` args
to reducing function. Use `map-entry?`, `key`, `val` utils for these.

Example transducers:

  (map (fn [x] (println x) x)) ; Print each coll item thawed

  (comp
    (map    (fn [x] (if (= x :secret) :redacted x))) ; Replace secrets
    (remove (fn [x] ; Remove maps with a truthy :remove?
              (or
                (and (map?       x) (:remove? x))
                (and (map-entry? x) (= (key x) :remove?) (val y)))))))

Note that while this is a very powerful feature, correctly writing
and debugging transducers and reducing fns can be tricky.

To help, if Nippy encounters an error while applying your xform, it
will throw a detailed `ExceptionInfo` with message
"Error thrown via `*thaw-xform*`" to help you debug.
source

-cache-proxyclj

{[<x> <meta>] <idx>} for freezing, {<idx> <x-with-meta>} for thawing.
source

aes128-cbc-encryptorclj

Default 128bit AES-CBC encryptor with many-round SHA-512 key-gen.
See also `aes-128-cbc-encryptor`.
source

aes128-encryptorclj

Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen.

Password form [:salted "my-password"]
---------------------------------------
USE CASE: You want more than a small, finite number of passwords (e.g. each
          item encrypted will use a unique user-provided password).

IMPLEMENTATION: Uses a relatively cheap key hash, but automatically salts
                every key.

PROS: Each key is independent so would need to be attacked independently.
CONS: Key caching impossible, so there's an inherent trade-off between
      encryption/decryption speed and the difficulty of attacking any
      particular key.

Slower than `aes128-cached`, and easier to attack any particular key - but
keys are independent.

Password form [:cached "my-password"]
---------------------------------------
USE CASE: You want only a small, finite number of passwords (e.g. a limited
          number of staff/admins, or you'll be using a single password to
          encrypt many items).

IMPLEMENTATION: Uses a _very_ expensive (but cached) key hash, and no salt.

PROS: Great amortized encryption/decryption speed. Expensive key hash makes
      attacking any particular key very difficult.
CONS: Using a small number of keys for many encrypted items means that if any
      key _is_ somehow compromised, _all_ items encrypted with that key are
      compromised.

Faster than `aes128-salted`, and harder to attack any particular key - but
increased danger if a key is somehow compromised.
source

aes128-gcm-encryptorclj

Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen.

Password form [:salted "my-password"]
---------------------------------------
USE CASE: You want more than a small, finite number of passwords (e.g. each
          item encrypted will use a unique user-provided password).

IMPLEMENTATION: Uses a relatively cheap key hash, but automatically salts
                every key.

PROS: Each key is independent so would need to be attacked independently.
CONS: Key caching impossible, so there's an inherent trade-off between
      encryption/decryption speed and the difficulty of attacking any
      particular key.

Slower than `aes128-cached`, and easier to attack any particular key - but
keys are independent.

Password form [:cached "my-password"]
---------------------------------------
USE CASE: You want only a small, finite number of passwords (e.g. a limited
          number of staff/admins, or you'll be using a single password to
          encrypt many items).

IMPLEMENTATION: Uses a _very_ expensive (but cached) key hash, and no salt.

PROS: Great amortized encryption/decryption speed. Expensive key hash makes
      attacking any particular key very difficult.
CONS: Using a small number of keys for many encrypted items means that if any
      key _is_ somehow compromised, _all_ items encrypted with that key are
      compromised.

Faster than `aes128-salted`, and harder to attack any particular key - but
increased danger if a key is somehow compromised.
source

allow-and-record-any-serializable-class-unsafeclj

(allow-and-record-any-serializable-class-unsafe class-name)
A predicate (fn allow-class? [class-name]) fn that can be assigned
to `*freeze-serializable-allowlist*` and/or
     `*thaw-serializable-allowlist*` that:

  - Will allow ANY class to use Nippy's `Serializable` support (unsafe).
  - And will record {<class-name> <frequency-allowed>} for the <=1000
    classes that ~most frequently made use of this support.

`get-recorded-serializable-classes` returns the recorded state.

This predicate is provided as a convenience for users upgrading from
previous versions of Nippy that allowed the use of `Serializable` for all
classes by default.

While transitioning from an unsafe->safe configuration, you can use
this predicate (unsafe) to record information about which classes have
been using Nippy's `Serializable` support in your environment.

Once some time has passed, you can check the recorded state. If you're
satisfied that all recorded classes are safely `Serializable`, you can
then merge the recorded classes into Nippy's default allowlist/s, e.g.:

(alter-var-root #'thaw-serializable-allowlist*
  (fn [_] (into default-thaw-serializable-allowlist
            (keys (get-recorded-serializable-classes)))))
source

cacheclj

(cache x)
Wraps value so that future writes of the same wrapped value with same
metadata will be efficiently encoded as references to this one.

(freeze [(cache "foo") (cache "foo") (cache "foo")])
  will incl. a single "foo", plus 2x single-byte references to "foo".
source

compressclj

(compress compressor ba)
source

decompressclj

(decompress compressor ba)
source

decryptclj

(decrypt encryptor pwd ba)
source

default-freeze-serializable-allowlistclj

Allows *any* class-name to be frozen using Java's `Serializable` interface.
This is generally safe since RCE risk is present only when thawing.
See also `*freeze-serializable-allowlist*`.
source

default-thaw-serializable-allowlistclj

A set of common safe class-names to allow to be frozen using Java's
`Serializable` interface. PRs welcome for additions.
See also `*thaw-serializable-allowlist*`.
source

encryptclj

(encrypt encryptor pwd ba)
source

extend-freezecljmacro

(extend-freeze type custom-type-id [x out] & body)
Extends Nippy to support freezing of a custom type (ideally concrete) with
given id of form:

  * ℕ∈[1, 128]           - 0 byte overhead. You are responsible for managing ids.
  * (Namespaced) keyword - 2 byte overhead. Keyword will be hashed to 16 bit int,
                           collisions will throw at compile-time.

NB: be careful about extending to interfaces, Ref. <http://goo.gl/6gGRlU>.

(defrecord MyRec [data])
(extend-freeze MyRec :foo/my-type [x data-output] ; Keyword id
  (.writeUTF [data-output] (:data x)))
;; or
(extend-freeze MyRec 1 [x data-output] ; Byte id
  (.writeUTF [data-output] (:data x)))
source

extend-thawcljmacro

(extend-thaw custom-type-id [in] & body)
Extends Nippy to support thawing of a custom type with given id:
(extend-thaw :foo/my-type [data-input] ; Keyword id
  (MyRec. (.readUTF data-input)))
;; or
(extend-thaw 1 [data-input] ; Byte id
  (MyRec. (.readUTF data-input)))
source

fast-freezeclj

(fast-freeze x)
Like `freeze` but:
  - Writes data without a Nippy header
  - Drops all support for compression and encryption
  - Must be thawed with `fast-thaw`

Equivalent to (but a little faster than) `freeze` with opts
  {:no-header? true, :compressor nil, :encryptor nil}.

Intended for use only by advanced users that clearly understand the tradeoffs.
I STRONGLY recommend that most users prefer the standard `freeze` since:
  - The Nippy header is useful for data portability and preservation
  - Compression is often benefitial at little/no cost
  - The performance difference between `freeze` and `fast-freeze` is
    often negligible in practice.
source

fast-thawclj

(fast-thaw ba)
Like `thaw` but:
  - Supports only data frozen with `fast-freeze`
  - Drops all support for compression and encryption

Equivalent to (but a little faster than) `thaw` with opts:
  {:no-header? true, :compressor nil, :encryptor nil}.
source

freezable?clj

(freezable? x)
(freezable? x
            {:as opts
             :keys [recursive? allow-clojure-reader? allow-java-serializable?]
             :or {recursive? true}})
Alpha, subject to change.
Returns ∈ #{:native :maybe-clojure-reader :maybe-java-serializable nil},
truthy iff Nippy seems to support freezing for the given argument.

Important: result can be inaccurate in some cases. To be completely sure you
unfortunately need to try freeze then thaw the argument, and check the thawed
value.

Options include:
  `recursive?`               - Check recursively into given arg?
  `allow-clojure-reader?`    - Allow freezing with Clojure's reader?
  `allow-java-serializable?` - Allow freezing with Java's `Serializable`?
source

freezeclj

(freeze x)
(freeze x
        {:as opts
         :keys [compressor encryptor password serializable-allowlist
                incl-metadata?]
         :or {compressor :auto encryptor aes128-gcm-encryptor}})
Serializes arg (any Clojure data type) to a byte array.
To freeze custom types, extend the Clojure reader or see `extend-freeze`.
source

freeze-to-fileclj

(freeze-to-file file x)
(freeze-to-file file x freeze-opts)
Convenience util: like `freeze`, but writes to `(clojure.java.io/file <file>)`.
source

freeze-to-out!clj

(freeze-to-out! data-output x)
Serializes arg (any Clojure data type) to a DataOutput.
This is a low-level util: in most cases you'll want `freeze` instead.
source

freeze-to-stringclj

(freeze-to-string x)
(freeze-to-string x freeze-opts)
Convenience util: like `freeze`, but returns a Base64-encoded string.
See also `thaw-from-string`.
source

get-recorded-serializable-classesclj

(get-recorded-serializable-classes)
Returns {<class-name> <frequency>} of the <=1000 classes that ~most
frequently made use of Nippy's `Serializable` support via
`allow-and-record-any-serializable-class-unsafe`.

See that function's docstring for more info.
source

IFreezablecljprotocol

Private implementation detail.
Protocol that types must implement to support native freezing by Nippy.
Don't use this directly, instead see `extend-freeze`.

-freezable?clj

(-freezable? _)

-freeze-without-meta!clj

(-freeze-without-meta! _ data-output)
source

IFreezableWithMetacljprotocol

Private implementation detail.
Wrapper protocol around `IFreezable` used to handle possible metadata.

-freeze-with-meta!clj

(-freeze-with-meta! _ data-output)
source

inspect-baclj

(inspect-ba ba)
(inspect-ba ba thaw-opts)
Experimental, subject to change. Feedback welcome!
source

lz4-compressorclj

Default `LZ4` compressor:
  -   Compression ratio: `C`  (0.58      on reference benchmark).
  -   Compression speed: `A`  (240 msecs on reference benchmark).
  - Decompression speed: `A+` (30  msecs on reference benchmark).

Good general-purpose compressor, favours speed.
See `taoensso.nippy-benchmarks` for detailed comparative benchmarks.
source

lzma2-compressorclj

Default `LZMA2` compressor:
  -   Compression ratio: `A+` (0.4       on reference benchmark).
  -   Compression speed: `E`  (18.5 secs on reference benchmark).
  - Decompression speed: `D`  (12   secs on reference benchmark).

Specialized compressor, strongly favours ratio.
See `taoensso.nippy-benchmarks` for detailed comparative benchmarks.
source

public-types-specclj

Public representation of Nippy's internal type schema.
For use by tooling and advanced users.

**HIGHLY EXPERIMENTAL!**
Subject to breaking change without notice.
Currently completely untested, may contain bugs.
Intended for use only by early adopters to give design feedback.

Format:
  {<type-id> {:keys [type-id type-kw payload-spec deprecated?]}},

  - `type-id`: A +ive single-byte identifier like `110`.
               -ive type ids are reserved for custom user-defined types.

  - `type-kw`: A keyword like `:kw-sm`,
    suffixes used to differentiate subtypes of different sizes:
      -0  ; Empty       => 0 byte         payload / element-count
      -sm ; Small       => 1 byte (byte)  payload / element-count
      -md ; Medium      => 2 byte (short) payload / element-count
      -lg ; Large       => 4 byte (int)   payload / element-count
      -xl ; Extra large => 8 byte (long)  payload / element-count

  - `payload-spec` examples:
    - nil                        ; No spec available (e.g. unpredictable payload)
    - []                         ; Type has no payload
    - [[:bytes 4]]               ; Type has    payload of exactly 4 bytes
    - [[:bytes 2] [:elements 2]] ; Type has    payload of exactly 2 bytes,
                                 ; followed by 2 elements

    - [[:bytes    {:read 2}]
       [:elements {:read 4 :multiplier 2 :unsigned? true}]]

      ; Type has payload of <short-count> bytes, followed by
      ; <unsigned-int-count>*2 (multiplier) elements

    Note that `payload-spec` can be handy for skipping over items in
    data stream without fully reading every item.
source

read-quarantined-serializable-object-unsafe!clj

(read-quarantined-serializable-object-unsafe! m)
Given a quarantined Serializable object like
{:nippy/unthawable {:class-name <> :content <quarantined-ba>}}, reads and
returns the object WITHOUT regard for `*thaw-serializable-allowlist*`.

**MAY BE UNSAFE!** Don't call this unless you absolutely trust the payload
to not contain any malicious code.

See `*thaw-serializable-allowlist*` for more info.
source

snappy-compressorclj

Default `Snappy` compressor:
  -   Compression ratio: `C`  (0.58      on reference benchmark).
  -   Compression speed: `A+` (210 msecs on reference benchmark).
  - Decompression speed: `B`  (130 msecs on reference benchmark).
Good general-purpose compressor, favours speed.
See `taoensso.nippy-benchmarks` for detailed comparative benchmarks.
source

stress-dataclj

(stress-data {:keys [comparable?] :as opts})
Returns map of reference stress data for use by tests, benchmarks, etc.
source

thawclj

(thaw ba)
(thaw ba
      {:as opts
       :keys [v1-compatibility? compressor encryptor password
              serializable-allowlist incl-metadata? thaw-xform]
       :or {compressor :auto encryptor :auto}})
Deserializes a frozen Nippy byte array to its original Clojure data type.
To thaw custom types, extend the Clojure reader or see `extend-thaw`.

** By default, supports data frozen with Nippy v2+ ONLY **
Add `{:v1-compatibility? true}` option to support thawing of data frozen with
legacy versions of Nippy.

Options include:
  :v1-compatibility? - support data frozen by legacy versions of Nippy?
  :compressor - :auto (checks header, default)  an ICompressor, or nil
  :encryptor  - :auto (checks header, default), an IEncryptor,  or nil
source

thaw-from-fileclj

(thaw-from-file file)
(thaw-from-file file thaw-opts)
Convenience util: like `thaw`, but reads from `(clojure.java.io/file <file>)`.
source

thaw-from-in!clj

(thaw-from-in! data-input)
Deserializes a frozen object from given DataInput to its original Clojure
data type.

This is a low-level util: in most cases you'll want `thaw` instead.
source

thaw-from-resourceclj

(thaw-from-resource res)
(thaw-from-resource res thaw-opts)
Convenience util: like `thaw`, but reads from `(clojure.java.io/resource <res>)`.
source

thaw-from-stringclj

(thaw-from-string s)
(thaw-from-string s thaw-opts)
Convenience util: like `thaw`, but takes a Base64-encoded string.
See also `freeze-to-string`.
source

throw-unfreezableclj

(throw-unfreezable x)
source

try-write-readableclj

(try-write-readable out x)
source

try-write-serializableclj

(try-write-serializable out x)
source

with-cachecljmacro

(with-cache & body)
Executes body with support for freezing/thawing cached values.

This is a low-level util: you won't need to use this yourself unless
you're using `freeze-to-out!` or `thaw-from-in!` (also low-level utils).

See also `cache`.
source

write-idcljmacro

(write-id out id)
source

write-unfreezableclj

(write-unfreezable out x)
source

zstd-compressorclj

Default `Zstd` (`Zstandard`) compressor:
  -   Compression ratio: `B` (0.53       on reference benchmark).
  -   Compression speed: `C` (1300 msecs on reference benchmark).
  - Decompression speed: `B` (400  msecs on reference benchmark).

Good general-purpose compressor, balances ratio & speed.
See `taoensso.nippy-benchmarks` for detailed comparative benchmarks.
source

cljdoc is a website building & hosting documentation for Clojure/Script libraries

Keyboard shortcutsReport a problemcljdoc on GitHub
× close