Liking cljdoc? Tell your friends :D

io.github.frenchy64.fully-satisfies.clearing-future

futures that clear conveyed bindings after execution.

Fixes memory leak described in https://clojure.atlassian.net/browse/CLJ-2619

futures that clear conveyed bindings after execution.

Fixes memory leak described in https://clojure.atlassian.net/browse/CLJ-2619
raw docstring

io.github.frenchy64.fully-satisfies.everyp

An implementation of clojure.core/every-pred with a simple definitional equivalence.

An implementation of clojure.core/every-pred with a simple definitional equivalence.
raw docstring

io.github.frenchy64.fully-satisfies.folda

Variant of clojure.core/areduce that supports naming the array.

Variant of `clojure.core/areduce` that supports naming the array.
raw docstring

io.github.frenchy64.fully-satisfies.head-releasing

Variants of clojure.core functions (and internal helpers) that release the head of seqs earlier, enabling lower peak memory usage in some cases.

For example, many higher-order functions in this namespace release strong references to arguments before calling their function arguments. Realizing the following example would be prone to failure with an OutOfMemoryError using clojure.core/map because map retains a strong reference to takes-75-percent-of-heap when calling its function argument, which prevents memory from being reclaimed to create another value of that size.

(map (fn [takes-75-percent-of-heap] (if (< (rand) 0.5) takes-75-percent-of-heap (generate-another-value-taking-75-percent-of-heap))) [takes-75-percent-of-heap])

In contrast, using head-releasing/map, the garbage collector can reclaim takes-75-percent-of-heap while calculating (generate-another-value-taking-75-percent-of-heap) because head-releasing/map does not strongly reference takes-75-percent-of-heap at that point.

The basic implementation trick to achieving this is to call (rest s) on the seq currently being processed before calling (f (first f)), so the strong reference to (first f) is transferred from the higher-order function to f during the call to f.

There are potential caveats to this approach: https://clojure.org/reference/lazy#_extension_iseqs If the underlying ISeq implementation defines rest in terms of next as in the article, then the functions in this namespace will force two seq elements into memory simultaneously. For example, the call below will throw an OutOfMemoryError before the fn is called because both elements of the seq will be realized.

(map (fn [takes-75-percent-of-heap] nil) (lazy-seq-where-rest-calls-next (cons (->takes-75-percent-of-heap) (lazy-seq [(->takes-75-percent-of-heap)]))))

Infinite lazy seqs such as cycle or repeat always hold strong references to all their elements, so the functions in this namespace will have no affect in the memory usage of processing these seqs.

Another caveat is that the functions here are perhaps more useful as validation of the leaky-seq-detection framework and for pedalogical purposes about sequences than as significant bump in real-world expressivity. See the tests for how these implementations were verified: io.github.frenchy64.fully-satisfies.leaky-seq-detection-test

The author of this namespace can only speculate why the original functions were written this way. Perhaps the idea of a fn releasing a strong reference to one of its arguments was too rare to risk the caveats in using the default implementation of ISeq. Chunked seqs are also so prevalent and have much higher memory requirements that the optimizations in this namespace might have been deemed insignificant. For example, map processing a chunk of 32 must have enough memory to hold both 32 elements of the input collection and 32 elements of the output collection simultaneously. Chunked seqs also make programs such as the above unidiomatic since they are so difficult to get right (see the hoops math.combinatorics jumps through to reduce the 32+32 elements of the previous example to 32+1).

At the very least, this work helps crystallize the differences between rest and next---or more precisely, their similarities: while rest does not realize the next element, nor tell us whether a seq has more elements, it is still an eager operation whose result, like next, releases a strong reference to the first element of the seq.

Variants of clojure.core functions (and internal helpers)
that release the head of seqs earlier, enabling lower peak memory
usage in some cases.

For example, many higher-order functions in this namespace release strong
references to arguments before calling their function arguments.
Realizing the following example would be prone to failure with an OutOfMemoryError using
clojure.core/map because map retains a strong reference to takes-75-percent-of-heap
when calling its function argument, which prevents memory from being reclaimed to create
another value of that size.

(map (fn [takes-75-percent-of-heap]
       (if (< (rand) 0.5)
         takes-75-percent-of-heap
         (generate-another-value-taking-75-percent-of-heap)))
     [takes-75-percent-of-heap])

In contrast, using head-releasing/map, the garbage collector can reclaim takes-75-percent-of-heap
while calculating (generate-another-value-taking-75-percent-of-heap) because
head-releasing/map does not strongly reference takes-75-percent-of-heap at that point.

The basic implementation trick to achieving this is to call (rest s) on the seq currently
being processed _before_ calling (f (first f)), so the strong reference to (first f)
is transferred from the higher-order function to f during the call to f.

There are potential caveats to this approach: https://clojure.org/reference/lazy#_extension_iseqs
If the underlying ISeq implementation defines rest in terms of next as in the article, then the
functions in this namespace will force two seq elements into memory simultaneously.
For example, the call below will throw an OutOfMemoryError before the fn is called because both
elements of the seq will be realized.

(map (fn [takes-75-percent-of-heap] nil)
     (lazy-seq-where-rest-calls-next
       (cons (->takes-75-percent-of-heap)
             (lazy-seq [(->takes-75-percent-of-heap)]))))

Infinite lazy seqs such as cycle or repeat always hold strong references to all their elements, so
the functions in this namespace will have no affect in the memory usage of processing these seqs.

Another caveat is that the functions here are perhaps more useful as
validation of the leaky-seq-detection framework and for pedalogical purposes about sequences
than as significant bump in real-world expressivity.
See the tests for how these implementations were verified:
  io.github.frenchy64.fully-satisfies.leaky-seq-detection-test

The author of this namespace can only speculate why the original functions were written this way.
Perhaps the idea of a fn releasing a strong reference to one of its arguments was too rare to risk
the caveats in using the default implementation of ISeq. Chunked seqs are also so prevalent
and have much higher memory requirements that the optimizations in this namespace might
have been deemed insignificant. For example, map processing a chunk of 32 must have enough memory
to hold both 32 elements of the input collection and 32 elements of the output collection simultaneously.
Chunked seqs also make programs such as the above unidiomatic since they are so difficult to get right
(see the hoops math.combinatorics jumps through to reduce the 32+32 elements of the previous example to
32+1).

At the very least, this work helps crystallize the differences between rest and next---or
more precisely, their similarities: while rest does not realize the next element, nor tell us whether
a seq has more elements, it is still an eager operation whose result, like next, releases a strong
reference to the first element of the seq.
raw docstring

io.github.frenchy64.fully-satisfies.lazier

Variants of clojure.core functions that are slightly lazier when processing and/or returning lazy seqs.

Some common (anti)patterns were to blame for the less-lazy versions of these functions. The main insight was that if you are using a counter to bound how many elements of a seq you are walking, you can be lazier by testing the counter before testing for more elements. For example,

int n = 0;
while(iter.hasNext() && n < CHUNK_SIZE)

will force n+1 elements and

int n = 0;
while(n < CHUNK_SIZE && iter.hasNext())

will force n elements. In this case, realizing the extra element has no utility because the chunk only fits CHUNK_SIZE elements.

Another problematic pattern was using seq-based implementations for sequence functions. Seqs must be non-empty, but a sequence can be empty, so implementations can be lazier.

Variants of clojure.core functions that are slightly lazier when
processing and/or returning lazy seqs.

Some common (anti)patterns were to blame for the less-lazy versions of
these functions. The main insight was that if you are using a counter to
bound how many elements of a seq you are walking, you can be lazier by
testing the counter _before_ testing for more elements. For example, 

```java
int n = 0;
while(iter.hasNext() && n < CHUNK_SIZE)
```

will force n+1 elements and

```java
int n = 0;
while(n < CHUNK_SIZE && iter.hasNext())
```

will force n elements. In this case, realizing the extra element has no utility because
the chunk only fits CHUNK_SIZE elements.

Another problematic pattern was using seq-based implementations for sequence functions.
Seqs must be non-empty, but a sequence can be empty, so implementations can be lazier.
raw docstring

io.github.frenchy64.fully-satisfies.leaky-seq-detection

A framework to detect memory leaks caused by holding onto the head of sequences.

The java.lang.ref.Cleaner class (JDK9+) provides hooks into garbage collection. You can register a function that is called when a value becomes phantom reachable, indicating is it a candidate for garbage collection.

The JVM is very likely to perform garbage collection right before throwing an OutOfMemoryError. Part of garbage collection is calculating whether references are reachable. We use this insight to force garbage collection (and hence, cleaners) to run, by inducing an OutOfMemoryError in try-forcing-cleaners!. Note that an OutOfMemoryError can leave the JVM in a bad state, so this strategy is best isolated away from other tests.

Tying these ideas together are reference-counting seqs and the is-strong testing macro. ref-counting-lazy-seq returns a lazy seq and an atom of all elements of the seq currently with strong references. This seq can now be passed to a sequence-processing function you would like to test for memory leaks.

is-strong then takes a set of seq indexes expected to have strong references and checks them against the atom tracking strong references.

Here's an example of asserting that a program adds or subtracts strong references to elements of a lazy seq at particular points.

(deftest example-cleaners-test (let [{:keys [strong lseq]} (ref-counting-lazy-seq {:n 10}) ;; seq of fresh Object's, length 10 ;; lseq=(...) _ (is-strong #{} strong) ;; no elements currently in memory lseq (seq lseq) ;; lseq=(0 ...) _ (is-strong #{0} strong) ;; just the first element in memory _ (nnext lseq) ;; lseq=(0 1 2...) _ (is-strong #{0 1 2} strong) ;; the first 3 elements in memory lseq (next lseq) ;; lseq=(1 2 ...) _ (is-strong #{1 2} strong) ;; the second 2 elements in memory lseq (rest lseq) ;; lseq=(2 ...) _ (is-strong #{2} strong) ;; the third element in memory lseq (rest lseq) ;; lseq=(...) _ (is-strong #{} strong) ;; no elements in memory ;; lseq=(3 ...) lseq (seq lseq) _ (is-strong #{3} strong) ;; fourth element in memory _ (class (first lseq)) ;; add a strong reference to lseq so previous line succeeds ;; lseq=nil _ (is-strong #{} strong) ;; lseq is entirely garbage collected ]))

See io.github.frenchy64.fully-satisfies.leaky-seq-detection-test for real-world examples of finding memory leaks in Clojure functions, and then verifying fixes for them.

A framework to detect memory leaks caused by holding onto the head of sequences.

The java.lang.ref.Cleaner class (JDK9+) provides hooks into garbage
collection. You can register a function that is called when a value
becomes phantom reachable, indicating is it a candidate for garbage
collection.

The JVM is very likely to perform garbage collection
right before throwing an OutOfMemoryError. Part of garbage
collection is calculating whether references are reachable.
We use this insight to force garbage collection (and hence, cleaners)
to run, by inducing an OutOfMemoryError in try-forcing-cleaners!.
Note that an OutOfMemoryError can leave the JVM in a bad state, so
this strategy is best isolated away from other tests.

Tying these ideas together are reference-counting seqs and the is-strong
testing macro. ref-counting-lazy-seq returns a lazy seq
and an atom of all elements of the seq currently with strong references.
This seq can now be passed to a sequence-processing function you would
like to test for memory leaks.

is-strong then takes a set of seq indexes expected to have strong references
and checks them against the atom tracking strong references.

Here's an example of asserting that a program adds or subtracts strong references to elements
of a lazy seq at particular points.

(deftest example-cleaners-test
  (let [{:keys [strong lseq]} (ref-counting-lazy-seq
                                {:n 10}) ;; seq of fresh Object's, length 10
        ;; lseq=(...)
        _ (is-strong #{} strong) ;; no elements currently in memory
        lseq (seq lseq)
        ;; lseq=(0 ...)
        _ (is-strong #{0} strong) ;; just the first element in memory
        _ (nnext lseq)
        ;; lseq=(0 1 2...)
        _ (is-strong #{0 1 2} strong) ;; the first 3 elements in memory
        lseq (next lseq)
        ;; lseq=(1 2 ...)
        _ (is-strong #{1 2} strong) ;; the second 2 elements in memory
        lseq (rest lseq)
        ;; lseq=(2 ...)
        _ (is-strong #{2} strong) ;; the third element in memory
        lseq (rest lseq)
        ;; lseq=(...)
        _ (is-strong #{} strong) ;; no elements in memory
        ;; lseq=(3 ...)
        lseq (seq lseq)
        _ (is-strong #{3} strong) ;; fourth element in memory
        _ (class (first lseq)) ;; add a strong reference to lseq so previous line succeeds
        ;; lseq=nil
        _ (is-strong #{} strong) ;; lseq is entirely garbage collected
        ]))

See io.github.frenchy64.fully-satisfies.leaky-seq-detection-test for real-world
examples of finding memory leaks in Clojure functions, and then verifying fixes for them.
raw docstring

io.github.frenchy64.fully-satisfies.linear

Linear-time sequence functions

Linear-time sequence functions
raw docstring

io.github.frenchy64.fully-satisfies.non-leaky-macros.clojure.core.async

Implementations of clojure.core.async macros that don't leak implementation details.

Implementations of clojure.core.async macros that don't leak implementation details.
raw docstring

io.github.frenchy64.fully-satisfies.non-leaky-macros.clojure.core.logic.pldb

Implementations of clojure.core.logic.pldb macros that don't leak implementation details.

Implementations of clojure.core.logic.pldb macros that don't leak implementation details.
raw docstring

io.github.frenchy64.fully-satisfies.non-leaky-macros.clojure.core.match.debug

Implementations of clojure.core.match.debug macros that don't leak implementation details.

Implementations of clojure.core.match.debug macros that don't leak implementation details.
raw docstring

io.github.frenchy64.fully-satisfies.non-leaky-macros.clojure.java.jmx

Implementations of clojure.java.jmx macros that don't leak implementation details.

Implementations of clojure.java.jmx macros that don't leak implementation details.
raw docstring

io.github.frenchy64.fully-satisfies.non-leaky-macros.clojure.java.shell

Implementations of clojure.java.test macros that don't leak implementation details.

Implementations of clojure.java.test macros that don't leak implementation details.
raw docstring

io.github.frenchy64.fully-satisfies.non-leaky-macros.clojure.pprint

Implementations of clojure.pprint macros that don't leak implementation details.

Implementations of clojure.pprint macros that don't leak implementation details.
raw docstring

io.github.frenchy64.fully-satisfies.non-leaky-macros.clojure.test.check.generators

Implementations of clojure.test.check.generators macros that don't leak implementation details.

Implementations of clojure.test.check.generators macros that don't leak implementation details.
raw docstring

io.github.frenchy64.fully-satisfies.non-leaky-macros.clojure.test.check.properties

Implementations of clojure.test.check.properties macros that don't leak implementation details.

Implementations of clojure.test.check.properties macros that don't leak implementation details.
raw docstring

io.github.frenchy64.fully-satisfies.non-leaky-macros.clojure.test.tap

Implementations of clojure.test.tap macros that don't leak implementation details.

Implementations of clojure.test.tap macros that don't leak implementation details.
raw docstring

io.github.frenchy64.fully-satisfies.non-leaky-macros.clojure.tools.trace

Implementations of clojure.tools.trace macros that don't leak implementation details.

Implementations of clojure.tools.trace macros that don't leak implementation details.
raw docstring

io.github.frenchy64.fully-satisfies.reify-spec

Provides a spec for clojure.core/reify via ::reify-args.

To register a spec for reify, call (register-reify-spec).

Provides a spec for clojure.core/reify via ::reify-args.

To register a spec for reify, call (register-reify-spec).
raw docstring

io.github.frenchy64.fully-satisfies.run-all

An alternative to clojure.core/run! that does not short-circuit on reduced.

An alternative to `clojure.core/run!` that does not short-circuit on reduced.
raw docstring

io.github.frenchy64.fully-satisfies.safer

Variants of clojure.core functions that improve thread-safety and general robustness when passed mutating collections.

We agree that 'Robust programs should not mutate arrays or Iterables that have seqs on them.' https://clojure.org/reference/sequences Eductions inhabit a middle ground. They are designed to be walked from first to last like seqs, but each element is recomputed when

For example, clojure.core/split-at could disagree on the take/drop parts of the collection if the coll is mutated between realizing the splits. Here, any one call to the eduction alternates between [0 1 2 3 4 5 6 7 8 9] and [9 8 7 6 5 4 3 2 1 0]. However, split-at incorrectly splits the eduction as [[0 1 2 3 4] [4 3 2 1 0]], and safer/split-at correctly splits it as [[0 1 2 3 4] [5 6 7 8 9]].

(deftest split-at-mutation-test (let [up-down (atom true) ed (eduction (map (fn [i] (when (zero? i) (swap! up-down not)) (if @up-down (- 9 i) i))) (range 10))] (is (= [[0 1 2 3 4] [4 3 2 1 0]] (split-at 5 ed))) (is (= [[0 1 2 3 4] [5 6 7 8 9]] (safer/split-at 5 ed)))))

See io.github.frenchy64.fully-satisfies.safer-test for more details.

The basic trick here is strategically calling seq earlier on the collection argument.

Variants of clojure.core functions that improve thread-safety and general robustness
when passed mutating collections.

We agree that 'Robust programs should not mutate arrays or Iterables that have seqs on them.'
https://clojure.org/reference/sequences
Eductions inhabit a middle ground. They are designed to be walked from first to last like seqs,
but each element is recomputed when 

For example, clojure.core/split-at could disagree on the take/drop parts of
the collection if the coll is mutated between realizing the splits.
Here, any one call to the eduction alternates between [0 1 2 3 4 5 6 7 8 9] and
[9 8 7 6 5 4 3 2 1 0]. However, split-at incorrectly splits the eduction as
[[0 1 2 3 4] [4 3 2 1 0]], and safer/split-at correctly splits it as
[[0 1 2 3 4] [5 6 7 8 9]].

(deftest split-at-mutation-test
  (let [up-down (atom true)
        ed (eduction (map (fn [i]
                            (when (zero? i)
                              (swap! up-down not))
                            (if @up-down
                              (- 9 i)
                              i)))
                     (range 10))]
    (is (= [[0 1 2 3 4] [4 3 2 1 0]] (split-at 5 ed)))
    (is (= [[0 1 2 3 4] [5 6 7 8 9]] (safer/split-at 5 ed)))))

See io.github.frenchy64.fully-satisfies.safer-test for more details.

The basic trick here is strategically calling seq earlier on the collection argument.
raw docstring

io.github.frenchy64.fully-satisfies.somef

An implementation of clojure.core/some-fn with a simple definitional equivalence.

An implementation of clojure.core/some-fn with a simple definitional equivalence.
raw docstring

io.github.frenchy64.fully-satisfies.uncaught-testing-contexts

Drop-in replacements for clojure.test/{deftest,testing} that (when used together) enhances uncaught exception error messages with the (most likely) testing context it was thrown from.

Example:

(deftest my-test (testing "foo" (doseq [v [1 2 3]] (testing v (assert (= 1 v))))))

With clojure.test/{deftest,testing} 1.10.3 (notice foo 2 is not mentioned):

user=> (test-var #'my-test) ;ERROR in (my-test) ;Uncaught exception, not in assertion. ;expected: nil ;actual: java.lang.AssertionError: Assert failed: false ;...

With {deftest,testing} in this namespace (notice foo 2 is mentioned):

user=> (test-var #'my-test) ;ERROR in (my-test) ;Uncaught exception, possibly thrown in testing context: foo 2 ;expected: nil ;actual: java.lang.AssertionError: Assert failed: false ;...

Drop-in replacements for `clojure.test/{deftest,testing}` that (when used
together) enhances uncaught exception error messages with the (most likely)
testing context it was thrown from.

Example:

  (deftest my-test
    (testing "foo"
      (doseq [v [1 2 3]]
        (testing v
          (assert (= 1 v))))))

With clojure.test/{deftest,testing} 1.10.3 (notice `foo 2` is not mentioned):

  user=> (test-var #'my-test)
  ;ERROR in (my-test)
  ;Uncaught exception, not in assertion.
  ;expected: nil
  ;actual: java.lang.AssertionError: Assert failed: false
  ;...
    
With {deftest,testing} in this namespace (notice `foo 2` is mentioned):

  user=> (test-var #'my-test)
  ;ERROR in (my-test)
  ;Uncaught exception, possibly thrown in testing context: foo 2
  ;expected: nil
  ;actual: java.lang.AssertionError: Assert failed: false
  ;...
raw docstring

io.github.frenchy64.fully-satisfies.uniform

Variants of clojure.core functions that are generalized to work uniformly for all values.

In all cases, a namespaced keyword was used as a special value that, if provided to the function, would break its promised semantics. The fixes involved replacing these special values with globally unique ones that are inaccessible to normal users (or more practically, unlikely to be generated with a generator like gen/any, or one that sources its values from the keyword interning table).

These corner-cases are demonstrated in io.github.frenchy64.fully-satisfies.uniform-test.

An effective generator that could find such defects reliably could generate keywords that occur in the source code of the functions reachable from the generative property (using static analysis, but perhaps this is also retrievable dynamically from the bytecode). On the other hand, such invasive analyses could yield false-negatives by providing values normally inaccessible to the user.

Variants of clojure.core functions that are generalized
to work uniformly for all values.

In all cases, a namespaced keyword was used as a special value
that, if provided to the function, would break its promised semantics.
The fixes involved replacing these special values with globally unique
ones that are inaccessible to normal users (or more practically, unlikely
to be generated with a generator like gen/any, or one that sources its
values from the keyword interning table).

These corner-cases are demonstrated in io.github.frenchy64.fully-satisfies.uniform-test.

An effective generator that could find such defects reliably could generate
keywords that occur in the source code of the functions reachable from the
generative property (using static analysis, but perhaps this is also retrievable
dynamically from the bytecode). On the other hand, such invasive analyses could
yield false-negatives by providing values normally inaccessible to the user.
raw docstring

cljdoc is a website building & hosting documentation for Clojure/Script libraries

× close