Liking cljdoc? Tell your friends :D
Clojure only.

taoensso.nippy

High-performance serialization library for Clojure.

*auto-freeze-compressor*clj

(fn [byte-array])->compressor used by `(freeze <x> {:compressor :auto}),
nil => default
source

*custom-readers*clj

{<hash-or-byte-id> (fn [data-input])->read}
source

*freeze-fallback*clj

Controls Nippy's behaviour when trying to freeze an item for which Nippy
doesn't currently have a native freeze/thaw implementation.

Possible values:

  1. `nil` (no freeze-fallback, default)
     Tries the following in order:
       - Freeze with Java's `Serializable` interface if possible
       - Freeze with Clojure's reader                if possible
       - Throw

  2. `:write-unfreezable` keyword
     Tries the following in order:
       - Freeze with Java's `Serializable` interface if possible
       - Freeze with Clojure's reader                if possible
       - Freeze a {:nippy/unfreezable {:type _}} placeholder value

  3. [Advanced] Custom (fn [^java.io.DataOutput out item]) that must
     write exactly one value to the given `DataOutput` stream
source

*freeze-serializable-allowlist*clj

Used when attempting to <freeze/thaw> an object that:
  - Does NOT implement Nippy's `Freezable`    protocol.
  - DOES     implement Java's  `Serializable` interface.

In this case, an allowlist will be checked to see if Java's
`Serializable` interface may be used.

This is a security measure to prevent possible Remote Code Execution
(RCE) when thawing malicious payloads. See [1] for details.

If `freeze` encounters a disallowed `Serializable` class, it will throw.
If `thaw`   encounters a disallowed `Serializable` class, it will:

  - Throw if it's not possible to safely quarantine the object
    (object was frozen with Nippy < v2.15.0-final).

  - Otherwise it will return a safely quarantined object of form
    `{:nippy/unthawable {:class-name <> :content <quarantined-ba>}}`.
    - Quarantined objects may be manually unquarantined with
      `read-quarantined-serializable-object-unsafe!`.

There are 2x allowlists:
  - `*freeze-serializable-allowlist*` ; Checked when freezing
  -   `*thaw-serializable-allowlist*` ; Checked when thawing

Example allowlist values:
  - `(fn allow-class? [class-name] true)`            ; Arbitrary predicate fn
  - `#{"java.lang.Throwable", "clojure.lang.*"}` ; Set of class-names
  - `"allow-and-record"`                           ; Special value, see [2]

  Note that class-names in sets may contain "*" wildcards.

Default allowlist values are:
  - default-freeze-serializable-allowlist ; `{"*"}` => allow any class
  -   default-thaw-serializable-allowlist ; A set of common safe classes

Allowlist values may be overridden with `binding`, `alter-var-root`, or:

  - `taoensso.nippy.<freeze/thaw>-serializable-allowlist-base` JVM property value
  - `taoensso.nippy.<freeze/thaw>-serializable-allowlist-add`  JVM property value

  - `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_BASE` Environment variable value
  - `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_ADD`  Environment variable value

If present, these will be read as comma-separated lists of class names
and formed into sets. Each initial allowlist value will then be:
(into (or <?base> <default>) <?additions>).

  I.e. you can use:
    - The "base" property/var to REPLACE Nippy's default allowlists.
    - The "add"  property/var to ADD TO  Nippy's default allowlists.

The special `"allow-and-record"` value is also possible, see [2].

Upgrading from an older version of Nippy and unsure whether you've been
using Nippy's `Serializable` support, or which classes to allow? See [2].

See also `taoensso.encore/name-filter` for a util to help easily build
more advanced predicate functions.

Thanks to Timo Mihaljov (@solita-timo-mihaljov) for an excellent report
identifying this vulnerability.

[1] https://github.com/ptaoussanis/nippy/issues/130
[2] See `allow-and-record-any-serializable-class-unsafe`.
source

*incl-metadata?*clj

Include metadata when freezing/thawing?
source

*thaw-serializable-allowlist*clj

Used when attempting to <freeze/thaw> an object that:
  - Does NOT implement Nippy's `Freezable`    protocol.
  - DOES     implement Java's  `Serializable` interface.

In this case, an allowlist will be checked to see if Java's
`Serializable` interface may be used.

This is a security measure to prevent possible Remote Code Execution
(RCE) when thawing malicious payloads. See [1] for details.

If `freeze` encounters a disallowed `Serializable` class, it will throw.
If `thaw`   encounters a disallowed `Serializable` class, it will:

  - Throw if it's not possible to safely quarantine the object
    (object was frozen with Nippy < v2.15.0-final).

  - Otherwise it will return a safely quarantined object of form
    `{:nippy/unthawable {:class-name <> :content <quarantined-ba>}}`.
    - Quarantined objects may be manually unquarantined with
      `read-quarantined-serializable-object-unsafe!`.

There are 2x allowlists:
  - `*freeze-serializable-allowlist*` ; Checked when freezing
  -   `*thaw-serializable-allowlist*` ; Checked when thawing

Example allowlist values:
  - `(fn allow-class? [class-name] true)`            ; Arbitrary predicate fn
  - `#{"java.lang.Throwable", "clojure.lang.*"}` ; Set of class-names
  - `"allow-and-record"`                           ; Special value, see [2]

  Note that class-names in sets may contain "*" wildcards.

Default allowlist values are:
  - default-freeze-serializable-allowlist ; `{"*"}` => allow any class
  -   default-thaw-serializable-allowlist ; A set of common safe classes

Allowlist values may be overridden with `binding`, `alter-var-root`, or:

  - `taoensso.nippy.<freeze/thaw>-serializable-allowlist-base` JVM property value
  - `taoensso.nippy.<freeze/thaw>-serializable-allowlist-add`  JVM property value

  - `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_BASE` Environment variable value
  - `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_ADD`  Environment variable value

If present, these will be read as comma-separated lists of class names
and formed into sets. Each initial allowlist value will then be:
(into (or <?base> <default>) <?additions>).

  I.e. you can use:
    - The "base" property/var to REPLACE Nippy's default allowlists.
    - The "add"  property/var to ADD TO  Nippy's default allowlists.

The special `"allow-and-record"` value is also possible, see [2].

Upgrading from an older version of Nippy and unsure whether you've been
using Nippy's `Serializable` support, or which classes to allow? See [2].

See also `taoensso.encore/name-filter` for a util to help easily build
more advanced predicate functions.

Thanks to Timo Mihaljov (@solita-timo-mihaljov) for an excellent report
identifying this vulnerability.

[1] https://github.com/ptaoussanis/nippy/issues/130
[2] See `allow-and-record-any-serializable-class-unsafe`.
source

*thaw-xform*clj

Experimental, subject to change. Feedback welcome!

Transducer to use when thawing standard Clojure collection types
(vectors, maps, sets, lists, etc.).

Allows fast+flexible inspection and manipulation of data being thawed.

Key-val style data structures like maps will provide `MapEntry` args
to reducing function. Use `map-entry?`, `key`, `val` utils for these.

Example transducers:

  (map (fn [x] (println x) x)) ; Print each coll item thawed

  (comp
    (map    (fn [x] (if (= x :secret) :redacted x))) ; Replace secrets
    (remove (fn [x] ; Remove maps with a truthy :remove?
              (or
                (and (map?       x) (:remove? x))
                (and (map-entry? x) (= (key x) :remove?) (val y)))))))

Note that while this is a very powerful feature, correctly writing
and debugging transducers and reducing fns can be tricky.

To help, if Nippy encounters an error while applying your xform, it
will throw a detailed `ExceptionInfo` with message
"Error thrown via `*thaw-xform*`" to help you debug.
source

-cache-proxyclj

{[<x> <meta>] <idx>} for freezing, {<idx> <x-with-meta>} for thawing.
source

aes128-cbc-encryptorclj

Default 128bit AES-CBC encryptor with many-round SHA-512 key-gen.
See also `aes-128-cbc-encryptor`.
source

aes128-encryptorclj

Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen.

Password form [:salted "my-password"]
---------------------------------------
USE CASE: You want more than a small, finite number of passwords (e.g. each
          item encrypted will use a unique user-provided password).

IMPLEMENTATION: Uses a relatively cheap key hash, but automatically salts
                every key.

PROS: Each key is independent so would need to be attacked independently.
CONS: Key caching impossible, so there's an inherent trade-off between
      encryption/decryption speed and the difficulty of attacking any
      particular key.

Slower than `aes128-cached`, and easier to attack any particular key - but
keys are independent.

Password form [:cached "my-password"]
---------------------------------------
USE CASE: You want only a small, finite number of passwords (e.g. a limited
          number of staff/admins, or you'll be using a single password to
          encrypt many items).

IMPLEMENTATION: Uses a _very_ expensive (but cached) key hash, and no salt.

PROS: Great amortized encryption/decryption speed. Expensive key hash makes
      attacking any particular key very difficult.
CONS: Using a small number of keys for many encrypted items means that if any
      key _is_ somehow compromised, _all_ items encrypted with that key are
      compromised.

Faster than `aes128-salted`, and harder to attack any particular key - but
increased danger if a key is somehow compromised.
source

aes128-gcm-encryptorclj

Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen.

Password form [:salted "my-password"]
---------------------------------------
USE CASE: You want more than a small, finite number of passwords (e.g. each
          item encrypted will use a unique user-provided password).

IMPLEMENTATION: Uses a relatively cheap key hash, but automatically salts
                every key.

PROS: Each key is independent so would need to be attacked independently.
CONS: Key caching impossible, so there's an inherent trade-off between
      encryption/decryption speed and the difficulty of attacking any
      particular key.

Slower than `aes128-cached`, and easier to attack any particular key - but
keys are independent.

Password form [:cached "my-password"]
---------------------------------------
USE CASE: You want only a small, finite number of passwords (e.g. a limited
          number of staff/admins, or you'll be using a single password to
          encrypt many items).

IMPLEMENTATION: Uses a _very_ expensive (but cached) key hash, and no salt.

PROS: Great amortized encryption/decryption speed. Expensive key hash makes
      attacking any particular key very difficult.
CONS: Using a small number of keys for many encrypted items means that if any
      key _is_ somehow compromised, _all_ items encrypted with that key are
      compromised.

Faster than `aes128-salted`, and harder to attack any particular key - but
increased danger if a key is somehow compromised.
source

allow-and-record-any-serializable-class-unsafeclj

(allow-and-record-any-serializable-class-unsafe class-name)
A predicate (fn allow-class? [class-name]) fn that can be assigned
to `*freeze-serializable-allowlist*` and/or
     `*thaw-serializable-allowlist*` that:

  - Will allow ANY class to use Nippy's `Serializable` support (unsafe).
  - And will record {<class-name> <frequency-allowed>} for the <=1000
    classes that ~most frequently made use of this support.

`get-recorded-serializable-classes` returns the recorded state.

This predicate is provided as a convenience for users upgrading from
previous versions of Nippy that allowed the use of `Serializable` for all
classes by default.

While transitioning from an unsafe->safe configuration, you can use
this predicate (unsafe) to record information about which classes have
been using Nippy's `Serializable` support in your environment.

Once some time has passed, you can check the recorded state. If you're
satisfied that all recorded classes are safely `Serializable`, you can
then merge the recorded classes into Nippy's default allowlist/s, e.g.:

(alter-var-root #'thaw-serializable-allowlist*
  (fn [_] (into default-thaw-serializable-allowlist
            (keys (get-recorded-serializable-classes)))))
source

cacheclj

(cache x)
Experimental, subject to change. Feedback welcome!

Wraps value so that future writes of the same wrapped value with same
metadata will be efficiently encoded as references to this one.

(freeze [(cache "foo") (cache "foo") (cache "foo")])
  will incl. a single "foo", plus 2x single-byte references to "foo".
source

compressclj

(compress compressor ba)
source

decompressclj

(decompress compressor ba)
source

decryptclj

(decrypt encryptor pwd ba)
source

default-freeze-serializable-allowlistclj

Allows *any* class-name to be frozen using Java's `Serializable` interface.
This is generally safe since RCE risk is present only when thawing.
See also `*freeze-serializable-allowlist*`.
source

default-thaw-serializable-allowlistclj

A set of common safe class-names to allow to be frozen using Java's
`Serializable` interface. PRs welcome for additions.
See also `*thaw-serializable-allowlist*`.
source

encryptclj

(encrypt encryptor pwd ba)
source

extend-freezecljmacro

(extend-freeze type custom-type-id [x out] & body)
Extends Nippy to support freezing of a custom type (ideally concrete) with
given id of form:

  * Keyword    - 2 byte overhead, keywords hashed to 16 bit id
  * ℕ∈[1, 128] - 0 byte overhead

NB: be careful about extending to interfaces, Ref. <http://goo.gl/6gGRlU>.

(defrecord MyRec [data])
(extend-freeze MyRec :foo/my-type [x data-output] ; Keyword id
  (.writeUTF [data-output] (:data x)))
;; or
(extend-freeze MyRec 1 [x data-output] ; Byte id
  (.writeUTF [data-output] (:data x)))
source

extend-thawcljmacro

(extend-thaw custom-type-id [in] & body)
Extends Nippy to support thawing of a custom type with given id:
(extend-thaw :foo/my-type [data-input] ; Keyword id
  (MyRec. (.readUTF data-input)))
;; or
(extend-thaw 1 [data-input] ; Byte id
  (MyRec. (.readUTF data-input)))
source

fast-freezeclj

(fast-freeze x)
Like `freeze` but:
  - Writes data without a Nippy header
  - Drops all support for compression and encryption
  - Must be thawed with `fast-thaw`

Equivalent to (but a little faster than) `freeze` with opts:
  - :compressor nil
  - :encryptor  nil
  - :no-header? true
source

fast-thawclj

(fast-thaw ba)
Like `thaw` but:
  - Drops all support for compression and encryption
  - Supports only data frozen with `fast-freeze`

Equivalent to (but a little faster than) `thaw` with opts:
  - :compressor nil
  - :encryptor  nil
  - :no-header? true
source

freezable?clj

(freezable? x)
(freezable? x
            {:as opts
             :keys [recursive? allow-clojure-reader? allow-java-serializable?]
             :or {recursive? true}})
Alpha, subject to change.
Returns ∈ #{:native :maybe-clojure-reader :maybe-java-serializable nil},
truthy iff Nippy seems to support freezing for the given argument.

Important: result can be inaccurate in some cases. To be completely sure you
unfortunately need to try freeze then thaw the argument, and check the thawed
value.

Options include:
  `recursive?`               - Check recursively into given arg?
  `allow-clojure-reader?`    - Allow freezing with Clojure's reader?
  `allow-java-serializable?` - Allow freezing with Java's `Serializable`?
source

freezeclj

(freeze x)
(freeze x
        {:as opts
         :keys [compressor encryptor password serializable-allowlist
                incl-metadata?]
         :or {compressor :auto encryptor aes128-gcm-encryptor}})
Serializes arg (any Clojure data type) to a byte array.
To freeze custom types, extend the Clojure reader or see `extend-freeze`.
source

freeze-to-fileclj

(freeze-to-file file x)
(freeze-to-file file x freeze-opts)
Convenience util: like `freeze`, but writes to `(clojure.java.io/file <file>)`.
source

freeze-to-out!clj

(freeze-to-out! data-output x)
Serializes arg (any Clojure data type) to a DataOutput.
This is a low-level util: in most cases you'll want `freeze` instead.
source

freeze-to-stringclj

(freeze-to-string x)
(freeze-to-string x freeze-opts)
Convenience util: like `freeze`, but returns a Base64-encoded string.
See also `thaw-from-string`.
source

get-recorded-serializable-classesclj

(get-recorded-serializable-classes)
Returns {<class-name> <frequency>} of the <=1000 classes that ~most
frequently made use of Nippy's `Serializable` support via
`allow-and-record-any-serializable-class-unsafe`.

See that function's docstring for more info.
source

IFreezablecljprotocol

Private implementation detail.
Protocol that types must implement to support native freezing by Nippy.
Don't use this directly, instead see `extend-freeze`.

-freezable?clj

(-freezable? _)

-freeze-without-meta!clj

(-freeze-without-meta! _ data-output)
source

IFreezableWithMetacljprotocol

Private implementation detail.
Wrapper protocol around `IFreezable` used to handle possible metadata.

-freeze-with-meta!clj

(-freeze-with-meta! _ data-output)
source

inspect-baclj

(inspect-ba ba)
(inspect-ba ba thaw-opts)
Experimental, subject to change. Feedback welcome!
source

lz4-compressorclj

Default `LZ4` compressor:
  -   Compression ratio: `C`  (0.58      on reference benchmark).
  -   Compression speed: `A`  (238 msecs on reference benchmark).
  - Decompression speed: `A+` (31  msecs on reference benchmark).

Good general-purpose compressor, favours speed.
See `taoensso.nippy-benchmarks` for detailed comparative benchmarks.
source

lz4hc-compressorcljdeprecated

Different LZ4 modes no longer supported, prefer `lz4-compressor`.
source

lzma2-compressorclj

Default `LZMA2` compressor:
  -   Compression ratio: `A+` (0.4       on reference benchmark).
  -   Compression speed: `E`  (18.5 secs on reference benchmark).
  - Decompression speed: `D`  (11.8 secs on reference benchmark).

Specialized compressor, strongly favours ratio.
See `taoensso.nippy-benchmarks` for detailed comparative benchmarks.
source

public-types-specclj

Public representation of Nippy's internal type schema.
For use by tooling and advanced users.

**HIGHLY EXPERIMENTAL!**
Subject to breaking change without notice.
Currently completely untested, may contain bugs.
Intended for use only by early adopters to give design feedback.

Format:
  {<type-id> {:keys [type-id type-kw payload-spec deprecated?]}},

  - `type-id`: A +ive single-byte identifier like `110`.
               -ive type ids are reserved for custom user-defined types.

  - `type-kw`: A keyword like `:kw-sm`,
    suffixes used to differentiate subtypes of different sizes:
      -0  ; Empty       => 0 byte         payload / element-count
      -sm ; Small       => 1 byte (byte)  payload / element-count
      -md ; Medium      => 2 byte (short) payload / element-count
      -lg ; Large       => 4 byte (int)   payload / element-count
      -xl ; Extra large => 8 byte (long)  payload / element-count

  - `payload-spec` examples:
    - nil                        ; No spec available (e.g. unpredictable payload)
    - []                         ; Type has no payload
    - [[:bytes 4]]               ; Type has a payload of exactly 4 bytes
    - [[:bytes 2] [:elements 2]] ; Type has a payload of exactly 2 bytes, then
                                 ; 2 elements

    - [[:bytes    {:read 2}]
       [:elements {:read 4 :multiplier 2 :unsigned? true}]]

      ; Type has payload of <short-count> bytes, then
      ; <unsigned-int-count>*2 (multiplier) elements

    Note that `payload-spec` can be handy for skipping over items in
    data stream without fully reading every item.
source

read-quarantined-serializable-object-unsafe!clj

(read-quarantined-serializable-object-unsafe! m)
Given a quarantined Serializable object like
{:nippy/unthawable {:class-name <> :content <quarantined-ba>}}, reads and
returns the object WITHOUT regard for `*thaw-serializable-allowlist*`.

**MAY BE UNSAFE!** Don't call this unless you absolutely trust the payload
to not contain any malicious code.

See `*thaw-serializable-allowlist*` for more info.
source

snappy-compressorclj

Default `Snappy` compressor:
  -   Compression ratio: `C`  (0.58      on reference benchmark).
  -   Compression speed: `A+` (206 msecs on reference benchmark).
  - Decompression speed: `B`  (134 msecs on reference benchmark).
Good general-purpose compressor, favours speed.
See `taoensso.nippy-benchmarks` for detailed comparative benchmarks.
source

stress-dataclj

(stress-data {:keys [comparable?] :as opts})
Returns map of reference stress data for use by tests, benchmarks, etc.
source

thawclj

(thaw ba)
(thaw ba
      {:as opts
       :keys [v1-compatibility? compressor encryptor password
              serializable-allowlist incl-metadata? thaw-xform]
       :or {compressor :auto encryptor :auto}})
Deserializes a frozen Nippy byte array to its original Clojure data type.
To thaw custom types, extend the Clojure reader or see `extend-thaw`.

** By default, supports data frozen with Nippy v2+ ONLY **
Add `{:v1-compatibility? true}` option to support thawing of data frozen with
legacy versions of Nippy.

Options include:
  :v1-compatibility? - support data frozen by legacy versions of Nippy?
  :compressor - :auto (checks header, default)  an ICompressor, or nil
  :encryptor  - :auto (checks header, default), an IEncryptor,  or nil
source

thaw-from-fileclj

(thaw-from-file file)
(thaw-from-file file thaw-opts)
Convenience util: like `thaw`, but reads from `(clojure.java.io/file <file>)`.
source

thaw-from-in!clj

(thaw-from-in! data-input)
Deserializes a frozen object from given DataInput to its original Clojure
data type.

This is a low-level util: in most cases you'll want `thaw` instead.
source

thaw-from-resourceclj

(thaw-from-resource res)
(thaw-from-resource res thaw-opts)
Convenience util: like `thaw`, but reads from `(clojure.java.io/resource <res>)`.
source

thaw-from-stringclj

(thaw-from-string s)
(thaw-from-string s thaw-opts)
Convenience util: like `thaw`, but takes a Base64-encoded string.
See also `freeze-to-string`.
source

throw-unfreezableclj

(throw-unfreezable x)
source

try-write-readableclj

(try-write-readable out x)
source

try-write-serializableclj

(try-write-serializable out x)
source

write-idcljmacro

(write-id out id)
source

write-unfreezableclj

(write-unfreezable out x)
source

zstd-compressorclj

Default `Zstd` (`Zstandard`) compressor:
  -   Compression ratio: `B` (0.53       on reference benchmark).
  -   Compression speed: `C` (1300 msecs on reference benchmark).
  - Decompression speed: `B` (400  msecs on reference benchmark).

Good general-purpose compressor, balances ratio & speed.
See `taoensso.nippy-benchmarks` for detailed comparative benchmarks.
source

cljdoc is a website building & hosting documentation for Clojure/Script libraries

Keyboard shortcutsReport a problemcljdoc on GitHub
× close