High-performance serialization library for Clojure.
(fn [byte-array])->compressor used by `(freeze <x> {:compressor :auto}), nil => default
Controls Nippy's behaviour when trying to freeze an item for which Nippy doesn't currently have a native freeze/thaw implementation. Possible values: 1. `nil` (no freeze-fallback, default) Tries the following in order: - Freeze with Java's `Serializable` interface if possible - Freeze with Clojure's reader if possible - Throw 2. `:write-unfreezable` keyword Tries the following in order: - Freeze with Java's `Serializable` interface if possible - Freeze with Clojure's reader if possible - Freeze a {:nippy/unfreezable {:type _}} placeholder value 3. [Advanced] Custom (fn [^java.io.DataOutput out item]) that must write exactly one value to the given `DataOutput` stream
Used when attempting to <freeze/thaw> an object that: - Does NOT implement Nippy's `Freezable` protocol. - DOES implement Java's `Serializable` interface. In this case, an allowlist will be checked to see if Java's `Serializable` interface may be used. This is a security measure to prevent possible Remote Code Execution (RCE) when thawing malicious payloads. See [1] for details. If `freeze` encounters a disallowed `Serializable` class, it will throw. If `thaw` encounters a disallowed `Serializable` class, it will: - Throw if it's not possible to safely quarantine the object (object was frozen with Nippy < v2.15.0-final). - Otherwise it will return a safely quarantined object of form `{:nippy/unthawable {:class-name <> :content <quarantined-ba>}}`. - Quarantined objects may be manually unquarantined with `read-quarantined-serializable-object-unsafe!`. There are 2x allowlists: - `*freeze-serializable-allowlist*` ; Checked when freezing - `*thaw-serializable-allowlist*` ; Checked when thawing Example allowlist values: - `(fn allow-class? [class-name] true)` ; Arbitrary predicate fn - `#{"java.lang.Throwable", "clojure.lang.*"}` ; Set of class-names - `"allow-and-record"` ; Special value, see [2] Note that class-names in sets may contain "*" wildcards. Default allowlist values are: - default-freeze-serializable-allowlist ; `{"*"}` => allow any class - default-thaw-serializable-allowlist ; A set of common safe classes Allowlist values may be overridden with `binding`, `alter-var-root`, or: - `taoensso.nippy.<freeze/thaw>-serializable-allowlist-base` JVM property value - `taoensso.nippy.<freeze/thaw>-serializable-allowlist-add` JVM property value - `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_BASE` Environment variable value - `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_ADD` Environment variable value If present, these will be read as comma-separated lists of class names and formed into sets. Each initial allowlist value will then be: (into (or <?base> <default>) <?additions>). I.e. you can use: - The "base" property/var to REPLACE Nippy's default allowlists. - The "add" property/var to ADD TO Nippy's default allowlists. The special `"allow-and-record"` value is also possible, see [2]. Upgrading from an older version of Nippy and unsure whether you've been using Nippy's `Serializable` support, or which classes to allow? See [2]. See also `taoensso.encore/name-filter` for a util to help easily build more advanced predicate functions. Thanks to Timo Mihaljov (@solita-timo-mihaljov) for an excellent report identifying this vulnerability. [1] https://github.com/ptaoussanis/nippy/issues/130 [2] See `allow-and-record-any-serializable-class-unsafe`.
Used when attempting to <freeze/thaw> an object that: - Does NOT implement Nippy's `Freezable` protocol. - DOES implement Java's `Serializable` interface. In this case, an allowlist will be checked to see if Java's `Serializable` interface may be used. This is a security measure to prevent possible Remote Code Execution (RCE) when thawing malicious payloads. See [1] for details. If `freeze` encounters a disallowed `Serializable` class, it will throw. If `thaw` encounters a disallowed `Serializable` class, it will: - Throw if it's not possible to safely quarantine the object (object was frozen with Nippy < v2.15.0-final). - Otherwise it will return a safely quarantined object of form `{:nippy/unthawable {:class-name <> :content <quarantined-ba>}}`. - Quarantined objects may be manually unquarantined with `read-quarantined-serializable-object-unsafe!`. There are 2x allowlists: - `*freeze-serializable-allowlist*` ; Checked when freezing - `*thaw-serializable-allowlist*` ; Checked when thawing Example allowlist values: - `(fn allow-class? [class-name] true)` ; Arbitrary predicate fn - `#{"java.lang.Throwable", "clojure.lang.*"}` ; Set of class-names - `"allow-and-record"` ; Special value, see [2] Note that class-names in sets may contain "*" wildcards. Default allowlist values are: - default-freeze-serializable-allowlist ; `{"*"}` => allow any class - default-thaw-serializable-allowlist ; A set of common safe classes Allowlist values may be overridden with `binding`, `alter-var-root`, or: - `taoensso.nippy.<freeze/thaw>-serializable-allowlist-base` JVM property value - `taoensso.nippy.<freeze/thaw>-serializable-allowlist-add` JVM property value - `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_BASE` Environment variable value - `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_ADD` Environment variable value If present, these will be read as comma-separated lists of class names and formed into sets. Each initial allowlist value will then be: (into (or <?base> <default>) <?additions>). I.e. you can use: - The "base" property/var to REPLACE Nippy's default allowlists. - The "add" property/var to ADD TO Nippy's default allowlists. The special `"allow-and-record"` value is also possible, see [2]. Upgrading from an older version of Nippy and unsure whether you've been using Nippy's `Serializable` support, or which classes to allow? See [2]. See also `taoensso.encore/name-filter` for a util to help easily build more advanced predicate functions. Thanks to Timo Mihaljov (@solita-timo-mihaljov) for an excellent report identifying this vulnerability. [1] https://github.com/ptaoussanis/nippy/issues/130 [2] See `allow-and-record-any-serializable-class-unsafe`.
Experimental, subject to change. Feedback welcome! Transducer to use when thawing standard Clojure collection types (vectors, maps, sets, lists, etc.). Allows fast+flexible inspection and manipulation of data being thawed. Key-val style data structures like maps will provide `MapEntry` args to reducing function. Use `map-entry?`, `key`, `val` utils for these. Example transducers: (map (fn [x] (println x) x)) ; Print each coll item thawed (comp (map (fn [x] (if (= x :secret) :redacted x))) ; Replace secrets (remove (fn [x] ; Remove maps with a truthy :remove? (or (and (map? x) (:remove? x)) (and (map-entry? x) (= (key x) :remove?) (val y))))))) Note that while this is a very powerful feature, correctly writing and debugging transducers and reducing fns can be tricky. To help, if Nippy encounters an error while applying your xform, it will throw a detailed `ExceptionInfo` with message "Error thrown via `*thaw-xform*`" to help you debug.
Default 128bit AES-CBC encryptor with many-round SHA-512 key-gen. See also `aes-128-cbc-encryptor`.
Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen. Password form [:salted "my-password"] --------------------------------------- USE CASE: You want more than a small, finite number of passwords (e.g. each item encrypted will use a unique user-provided password). IMPLEMENTATION: Uses a relatively cheap key hash, but automatically salts every key. PROS: Each key is independent so would need to be attacked independently. CONS: Key caching impossible, so there's an inherent trade-off between encryption/decryption speed and the difficulty of attacking any particular key. Slower than `aes128-cached`, and easier to attack any particular key - but keys are independent. Password form [:cached "my-password"] --------------------------------------- USE CASE: You want only a small, finite number of passwords (e.g. a limited number of staff/admins, or you'll be using a single password to encrypt many items). IMPLEMENTATION: Uses a _very_ expensive (but cached) key hash, and no salt. PROS: Great amortized encryption/decryption speed. Expensive key hash makes attacking any particular key very difficult. CONS: Using a small number of keys for many encrypted items means that if any key _is_ somehow compromised, _all_ items encrypted with that key are compromised. Faster than `aes128-salted`, and harder to attack any particular key - but increased danger if a key is somehow compromised.
Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen. Password form [:salted "my-password"] --------------------------------------- USE CASE: You want more than a small, finite number of passwords (e.g. each item encrypted will use a unique user-provided password). IMPLEMENTATION: Uses a relatively cheap key hash, but automatically salts every key. PROS: Each key is independent so would need to be attacked independently. CONS: Key caching impossible, so there's an inherent trade-off between encryption/decryption speed and the difficulty of attacking any particular key. Slower than `aes128-cached`, and easier to attack any particular key - but keys are independent. Password form [:cached "my-password"] --------------------------------------- USE CASE: You want only a small, finite number of passwords (e.g. a limited number of staff/admins, or you'll be using a single password to encrypt many items). IMPLEMENTATION: Uses a _very_ expensive (but cached) key hash, and no salt. PROS: Great amortized encryption/decryption speed. Expensive key hash makes attacking any particular key very difficult. CONS: Using a small number of keys for many encrypted items means that if any key _is_ somehow compromised, _all_ items encrypted with that key are compromised. Faster than `aes128-salted`, and harder to attack any particular key - but increased danger if a key is somehow compromised.
(allow-and-record-any-serializable-class-unsafe class-name)
A predicate (fn allow-class? [class-name]) fn that can be assigned to `*freeze-serializable-allowlist*` and/or `*thaw-serializable-allowlist*` that: - Will allow ANY class to use Nippy's `Serializable` support (unsafe). - And will record {<class-name> <frequency-allowed>} for the <=1000 classes that ~most frequently made use of this support. `get-recorded-serializable-classes` returns the recorded state. This predicate is provided as a convenience for users upgrading from previous versions of Nippy that allowed the use of `Serializable` for all classes by default. While transitioning from an unsafe->safe configuration, you can use this predicate (unsafe) to record information about which classes have been using Nippy's `Serializable` support in your environment. Once some time has passed, you can check the recorded state. If you're satisfied that all recorded classes are safely `Serializable`, you can then merge the recorded classes into Nippy's default allowlist/s, e.g.: (alter-var-root #'thaw-serializable-allowlist* (fn [_] (into default-thaw-serializable-allowlist (keys (get-recorded-serializable-classes)))))
(cache x)
Wraps value so that future writes of the same wrapped value with same metadata will be efficiently encoded as references to this one. (freeze [(cache "foo") (cache "foo") (cache "foo")]) will incl. a single "foo", plus 2x single-byte references to "foo".
Allows *any* class-name to be frozen using Java's `Serializable` interface. This is generally safe since RCE risk is present only when thawing. See also `*freeze-serializable-allowlist*`.
A set of common safe class-names to allow to be frozen using Java's `Serializable` interface. PRs welcome for additions. See also `*thaw-serializable-allowlist*`.
(extend-freeze type custom-type-id [x out] & body)
Extends Nippy to support freezing of a custom type (ideally concrete) with given id of form: * ℕ∈[1, 128] - 0 byte overhead. You are responsible for managing ids. * (Namespaced) keyword - 2 byte overhead. Keyword will be hashed to 16 bit int, collisions will throw at compile-time. NB: be careful about extending to interfaces, Ref. <http://goo.gl/6gGRlU>. (defrecord MyRec [data]) (extend-freeze MyRec :foo/my-type [x data-output] ; Keyword id (.writeUTF [data-output] (:data x))) ;; or (extend-freeze MyRec 1 [x data-output] ; Byte id (.writeUTF [data-output] (:data x)))
(extend-thaw custom-type-id [in] & body)
Extends Nippy to support thawing of a custom type with given id: (extend-thaw :foo/my-type [data-input] ; Keyword id (MyRec. (.readUTF data-input))) ;; or (extend-thaw 1 [data-input] ; Byte id (MyRec. (.readUTF data-input)))
(fast-freeze x)
Like `freeze` but: - Writes data without a Nippy header - Drops all support for compression and encryption - Must be thawed with `fast-thaw` Equivalent to (but a little faster than) `freeze` with opts {:no-header? true, :compressor nil, :encryptor nil}. Intended for use only by advanced users that clearly understand the tradeoffs. I STRONGLY recommend that most users prefer the standard `freeze` since: - The Nippy header is useful for data portability and preservation - Compression is often benefitial at little/no cost - The performance difference between `freeze` and `fast-freeze` is often negligible in practice.
(fast-thaw ba)
Like `thaw` but: - Supports only data frozen with `fast-freeze` - Drops all support for compression and encryption Equivalent to (but a little faster than) `thaw` with opts: {:no-header? true, :compressor nil, :encryptor nil}.
(freezable? x)
(freezable? x
{:as opts
:keys [recursive? allow-clojure-reader? allow-java-serializable?]
:or {recursive? true}})
Alpha, subject to change. Returns ∈ #{:native :maybe-clojure-reader :maybe-java-serializable nil}, truthy iff Nippy seems to support freezing for the given argument. Important: result can be inaccurate in some cases. To be completely sure you unfortunately need to try freeze then thaw the argument, and check the thawed value. Options include: `recursive?` - Check recursively into given arg? `allow-clojure-reader?` - Allow freezing with Clojure's reader? `allow-java-serializable?` - Allow freezing with Java's `Serializable`?
(freeze x)
(freeze x
{:as opts
:keys [compressor encryptor password serializable-allowlist
incl-metadata?]
:or {compressor :auto encryptor aes128-gcm-encryptor}})
Serializes arg (any Clojure data type) to a byte array. To freeze custom types, extend the Clojure reader or see `extend-freeze`.
(freeze-to-file file x)
(freeze-to-file file x freeze-opts)
Convenience util: like `freeze`, but writes to `(clojure.java.io/file <file>)`.
(freeze-to-out! data-output x)
Serializes arg (any Clojure data type) to a DataOutput. This is a low-level util: in most cases you'll want `freeze` instead.
(freeze-to-string x)
(freeze-to-string x freeze-opts)
Convenience util: like `freeze`, but returns a Base64-encoded string. See also `thaw-from-string`.
(get-recorded-serializable-classes)
Returns {<class-name> <frequency>} of the <=1000 classes that ~most frequently made use of Nippy's `Serializable` support via `allow-and-record-any-serializable-class-unsafe`. See that function's docstring for more info.
Private implementation detail. Protocol that types must implement to support native freezing by Nippy. Don't use this directly, instead see `extend-freeze`.
(-freezable? _)
(-freeze-without-meta! _ data-output)
Private implementation detail. Wrapper protocol around `IFreezable` used to handle possible metadata.
(-freeze-with-meta! _ data-output)
(inspect-ba ba)
(inspect-ba ba thaw-opts)
Experimental, subject to change. Feedback welcome!
Default `LZ4` compressor: - Compression ratio: `C` (0.58 on reference benchmark). - Compression speed: `A` (240 msecs on reference benchmark). - Decompression speed: `A+` (30 msecs on reference benchmark). Good general-purpose compressor, favours speed. See `taoensso.nippy-benchmarks` for detailed comparative benchmarks.
Default `LZMA2` compressor: - Compression ratio: `A+` (0.4 on reference benchmark). - Compression speed: `E` (18.5 secs on reference benchmark). - Decompression speed: `D` (12 secs on reference benchmark). Specialized compressor, strongly favours ratio. See `taoensso.nippy-benchmarks` for detailed comparative benchmarks.
Public representation of Nippy's internal type schema. For use by tooling and advanced users. **HIGHLY EXPERIMENTAL!** Subject to breaking change without notice. Currently completely untested, may contain bugs. Intended for use only by early adopters to give design feedback. Format: {<type-id> {:keys [type-id type-kw payload-spec deprecated?]}}, - `type-id`: A +ive single-byte identifier like `110`. -ive type ids are reserved for custom user-defined types. - `type-kw`: A keyword like `:kw-sm`, suffixes used to differentiate subtypes of different sizes: -0 ; Empty => 0 byte payload / element-count -sm ; Small => 1 byte (byte) payload / element-count -md ; Medium => 2 byte (short) payload / element-count -lg ; Large => 4 byte (int) payload / element-count -xl ; Extra large => 8 byte (long) payload / element-count - `payload-spec` examples: - nil ; No spec available (e.g. unpredictable payload) - [] ; Type has no payload - [[:bytes 4]] ; Type has payload of exactly 4 bytes - [[:bytes 2] [:elements 2]] ; Type has payload of exactly 2 bytes, ; followed by 2 elements - [[:bytes {:read 2}] [:elements {:read 4 :multiplier 2 :unsigned? true}]] ; Type has payload of <short-count> bytes, followed by ; <unsigned-int-count>*2 (multiplier) elements Note that `payload-spec` can be handy for skipping over items in data stream without fully reading every item.
(read-quarantined-serializable-object-unsafe! m)
Given a quarantined Serializable object like {:nippy/unthawable {:class-name <> :content <quarantined-ba>}}, reads and returns the object WITHOUT regard for `*thaw-serializable-allowlist*`. **MAY BE UNSAFE!** Don't call this unless you absolutely trust the payload to not contain any malicious code. See `*thaw-serializable-allowlist*` for more info.
Default `Snappy` compressor: - Compression ratio: `C` (0.58 on reference benchmark). - Compression speed: `A+` (210 msecs on reference benchmark). - Decompression speed: `B` (130 msecs on reference benchmark). Good general-purpose compressor, favours speed. See `taoensso.nippy-benchmarks` for detailed comparative benchmarks.
(stress-data {:keys [comparable?] :as opts})
Returns map of reference stress data for use by tests, benchmarks, etc.
(thaw ba)
(thaw ba
{:as opts
:keys [v1-compatibility? compressor encryptor password
serializable-allowlist incl-metadata? thaw-xform]
:or {compressor :auto encryptor :auto}})
Deserializes a frozen Nippy byte array to its original Clojure data type. To thaw custom types, extend the Clojure reader or see `extend-thaw`. ** By default, supports data frozen with Nippy v2+ ONLY ** Add `{:v1-compatibility? true}` option to support thawing of data frozen with legacy versions of Nippy. Options include: :v1-compatibility? - support data frozen by legacy versions of Nippy? :compressor - :auto (checks header, default) an ICompressor, or nil :encryptor - :auto (checks header, default), an IEncryptor, or nil
(thaw-from-file file)
(thaw-from-file file thaw-opts)
Convenience util: like `thaw`, but reads from `(clojure.java.io/file <file>)`.
(thaw-from-in! data-input)
Deserializes a frozen object from given DataInput to its original Clojure data type. This is a low-level util: in most cases you'll want `thaw` instead.
(thaw-from-resource res)
(thaw-from-resource res thaw-opts)
Convenience util: like `thaw`, but reads from `(clojure.java.io/resource <res>)`.
(thaw-from-string s)
(thaw-from-string s thaw-opts)
Convenience util: like `thaw`, but takes a Base64-encoded string. See also `freeze-to-string`.
(with-cache & body)
Executes body with support for freezing/thawing cached values. This is a low-level util: you won't need to use this yourself unless you're using `freeze-to-out!` or `thaw-from-in!` (also low-level utils). See also `cache`.
Default `Zstd` (`Zstandard`) compressor: - Compression ratio: `B` (0.53 on reference benchmark). - Compression speed: `C` (1300 msecs on reference benchmark). - Decompression speed: `B` (400 msecs on reference benchmark). Good general-purpose compressor, balances ratio & speed. See `taoensso.nippy-benchmarks` for detailed comparative benchmarks.
cljdoc is a website building & hosting documentation for Clojure/Script libraries
× close