futures that clear conveyed bindings after execution.
Fixes memory leak described in https://clojure.atlassian.net/browse/CLJ-2619
futures that clear conveyed bindings after execution. Fixes memory leak described in https://clojure.atlassian.net/browse/CLJ-2619
An implementation of clojure.core/every-pred with a simple definitional equivalence.
An implementation of clojure.core/every-pred with a simple definitional equivalence.
Variant of clojure.core/areduce
that supports naming the array.
Variant of `clojure.core/areduce` that supports naming the array.
Variants of clojure.core functions (and internal helpers) that release the head of seqs earlier, enabling lower peak memory usage in some cases.
For example, many higher-order functions in this namespace release strong references to arguments before calling their function arguments. Realizing the following example would be prone to failure with an OutOfMemoryError using clojure.core/map because map retains a strong reference to takes-75-percent-of-heap when calling its function argument, which prevents memory from being reclaimed to create another value of that size.
(map (fn [takes-75-percent-of-heap] (if (< (rand) 0.5) takes-75-percent-of-heap (generate-another-value-taking-75-percent-of-heap))) [takes-75-percent-of-heap])
In contrast, using head-releasing/map, the garbage collector can reclaim takes-75-percent-of-heap while calculating (generate-another-value-taking-75-percent-of-heap) because head-releasing/map does not strongly reference takes-75-percent-of-heap at that point.
The basic implementation trick to achieving this is to call (rest s) on the seq currently being processed before calling (f (first f)), so the strong reference to (first f) is transferred from the higher-order function to f during the call to f.
There are potential caveats to this approach: https://clojure.org/reference/lazy#_extension_iseqs If the underlying ISeq implementation defines rest in terms of next as in the article, then the functions in this namespace will force two seq elements into memory simultaneously. For example, the call below will throw an OutOfMemoryError before the fn is called because both elements of the seq will be realized.
(map (fn [takes-75-percent-of-heap] nil) (lazy-seq-where-rest-calls-next (cons (->takes-75-percent-of-heap) (lazy-seq [(->takes-75-percent-of-heap)]))))
Infinite lazy seqs such as cycle or repeat always hold strong references to all their elements, so the functions in this namespace will have no affect in the memory usage of processing these seqs.
Another caveat is that the functions here are perhaps more useful as validation of the leaky-seq-detection framework and for pedalogical purposes about sequences than as significant bump in real-world expressivity. See the tests for how these implementations were verified: io.github.frenchy64.fully-satisfies.leaky-seq-detection-test
The author of this namespace can only speculate why the original functions were written this way. Perhaps the idea of a fn releasing a strong reference to one of its arguments was too rare to risk the caveats in using the default implementation of ISeq. Chunked seqs are also so prevalent and have much higher memory requirements that the optimizations in this namespace might have been deemed insignificant. For example, map processing a chunk of 32 must have enough memory to hold both 32 elements of the input collection and 32 elements of the output collection simultaneously. Chunked seqs also make programs such as the above unidiomatic since they are so difficult to get right (see the hoops math.combinatorics jumps through to reduce the 32+32 elements of the previous example to 32+1).
At the very least, this work helps crystallize the differences between rest and next---or more precisely, their similarities: while rest does not realize the next element, nor tell us whether a seq has more elements, it is still an eager operation whose result, like next, releases a strong reference to the first element of the seq.
Variants of clojure.core functions (and internal helpers) that release the head of seqs earlier, enabling lower peak memory usage in some cases. For example, many higher-order functions in this namespace release strong references to arguments before calling their function arguments. Realizing the following example would be prone to failure with an OutOfMemoryError using clojure.core/map because map retains a strong reference to takes-75-percent-of-heap when calling its function argument, which prevents memory from being reclaimed to create another value of that size. (map (fn [takes-75-percent-of-heap] (if (< (rand) 0.5) takes-75-percent-of-heap (generate-another-value-taking-75-percent-of-heap))) [takes-75-percent-of-heap]) In contrast, using head-releasing/map, the garbage collector can reclaim takes-75-percent-of-heap while calculating (generate-another-value-taking-75-percent-of-heap) because head-releasing/map does not strongly reference takes-75-percent-of-heap at that point. The basic implementation trick to achieving this is to call (rest s) on the seq currently being processed _before_ calling (f (first f)), so the strong reference to (first f) is transferred from the higher-order function to f during the call to f. There are potential caveats to this approach: https://clojure.org/reference/lazy#_extension_iseqs If the underlying ISeq implementation defines rest in terms of next as in the article, then the functions in this namespace will force two seq elements into memory simultaneously. For example, the call below will throw an OutOfMemoryError before the fn is called because both elements of the seq will be realized. (map (fn [takes-75-percent-of-heap] nil) (lazy-seq-where-rest-calls-next (cons (->takes-75-percent-of-heap) (lazy-seq [(->takes-75-percent-of-heap)])))) Infinite lazy seqs such as cycle or repeat always hold strong references to all their elements, so the functions in this namespace will have no affect in the memory usage of processing these seqs. Another caveat is that the functions here are perhaps more useful as validation of the leaky-seq-detection framework and for pedalogical purposes about sequences than as significant bump in real-world expressivity. See the tests for how these implementations were verified: io.github.frenchy64.fully-satisfies.leaky-seq-detection-test The author of this namespace can only speculate why the original functions were written this way. Perhaps the idea of a fn releasing a strong reference to one of its arguments was too rare to risk the caveats in using the default implementation of ISeq. Chunked seqs are also so prevalent and have much higher memory requirements that the optimizations in this namespace might have been deemed insignificant. For example, map processing a chunk of 32 must have enough memory to hold both 32 elements of the input collection and 32 elements of the output collection simultaneously. Chunked seqs also make programs such as the above unidiomatic since they are so difficult to get right (see the hoops math.combinatorics jumps through to reduce the 32+32 elements of the previous example to 32+1). At the very least, this work helps crystallize the differences between rest and next---or more precisely, their similarities: while rest does not realize the next element, nor tell us whether a seq has more elements, it is still an eager operation whose result, like next, releases a strong reference to the first element of the seq.
Variants of clojure.core functions that are slightly lazier when processing and/or returning lazy seqs.
Some common (anti)patterns were to blame for the less-lazy versions of these functions. The main insight was that if you are using a counter to bound how many elements of a seq you are walking, you can be lazier by testing the counter before testing for more elements. For example,
int n = 0;
while(iter.hasNext() && n < CHUNK_SIZE)
will force n+1 elements and
int n = 0;
while(n < CHUNK_SIZE && iter.hasNext())
will force n elements. In this case, realizing the extra element has no utility because the chunk only fits CHUNK_SIZE elements.
Another problematic pattern was using seq-based implementations for sequence functions. Seqs must be non-empty, but a sequence can be empty, so implementations can be lazier.
Variants of clojure.core functions that are slightly lazier when processing and/or returning lazy seqs. Some common (anti)patterns were to blame for the less-lazy versions of these functions. The main insight was that if you are using a counter to bound how many elements of a seq you are walking, you can be lazier by testing the counter _before_ testing for more elements. For example, ```java int n = 0; while(iter.hasNext() && n < CHUNK_SIZE) ``` will force n+1 elements and ```java int n = 0; while(n < CHUNK_SIZE && iter.hasNext()) ``` will force n elements. In this case, realizing the extra element has no utility because the chunk only fits CHUNK_SIZE elements. Another problematic pattern was using seq-based implementations for sequence functions. Seqs must be non-empty, but a sequence can be empty, so implementations can be lazier.
A framework to detect memory leaks caused by holding onto the head of sequences.
The java.lang.ref.Cleaner class (JDK9+) provides hooks into garbage collection. You can register a function that is called when a value becomes phantom reachable, indicating is it a candidate for garbage collection.
The JVM is very likely to perform garbage collection right before throwing an OutOfMemoryError. Part of garbage collection is calculating whether references are reachable. We use this insight to force garbage collection (and hence, cleaners) to run, by inducing an OutOfMemoryError in try-forcing-cleaners!. Note that an OutOfMemoryError can leave the JVM in a bad state, so this strategy is best isolated away from other tests.
Tying these ideas together are reference-counting seqs and the is-strong testing macro. ref-counting-lazy-seq returns a lazy seq and an atom of all elements of the seq currently with strong references. This seq can now be passed to a sequence-processing function you would like to test for memory leaks.
is-strong then takes a set of seq indexes expected to have strong references and checks them against the atom tracking strong references. It will continue to induce OutOfMemoryError's until the expected references are found, or eventually fail via an is test assertion.
Note that it's best to build up the set of expected strong references rather than whittle it down. From a usability standpoint, starting with (is-strong #{} live) will likely print the actual set of strongly referenced indexes, from which you can source the expected strong references (from the result itself or a subset). But false-positives are possible if an overly broad superset of the actual strong references are provided, since is-strong may short-circuit its search earlier than the cleaners can run since. Such results can lead to false conclusions about how lazy a function is, especially if you have a pre-conceived notion of the results! The JVM property io.github.frenchy64.fully-satisfies.leaky-seq-detection.is-strong.false-positive-detection=true will more aggressively search for such false-positives, which may be suitable for a cron CI job.
Here's an example of asserting that a program adds or subtracts strong references to elements of a lazy seq at particular points.
(deftest example-cleaners-test (let [{:keys [strong lseq]} (ref-counting-lazy-seq {:n 10}) ;; seq of fresh Object's, length 10 ;; lseq=(...) _ (is-strong #{} strong) ;; no elements currently in memory lseq (seq lseq) ;; lseq=(0 ...) _ (is-strong #{0} strong) ;; just the first element in memory _ (nnext lseq) ;; lseq=(0 1 2...) _ (is-strong #{0 1 2} strong) ;; the first 3 elements in memory lseq (next lseq) ;; lseq=(1 2 ...) _ (is-strong #{1 2} strong) ;; the second 2 elements in memory lseq (rest lseq) ;; lseq=(2 ...) _ (is-strong #{2} strong) ;; the third element in memory lseq (rest lseq) ;; lseq=(...) _ (is-strong #{} strong) ;; no elements in memory ;; lseq=(3 ...) lseq (seq lseq) _ (is-strong #{3} strong) ;; fourth element in memory _ (class (first lseq)) ;; add a strong reference to lseq so previous line succeeds ;; lseq=nil _ (is-strong #{} strong) ;; lseq is entirely garbage collected ]))
See io.github.frenchy64.fully-satisfies.leaky-seq-detection-test for real-world examples of finding memory leaks in Clojure functions, and then verifying fixes for them.
A framework to detect memory leaks caused by holding onto the head of sequences. The java.lang.ref.Cleaner class (JDK9+) provides hooks into garbage collection. You can register a function that is called when a value becomes phantom reachable, indicating is it a candidate for garbage collection. The JVM is very likely to perform garbage collection right before throwing an OutOfMemoryError. Part of garbage collection is calculating whether references are reachable. We use this insight to force garbage collection (and hence, cleaners) to run, by inducing an OutOfMemoryError in try-forcing-cleaners!. Note that an OutOfMemoryError can leave the JVM in a bad state, so this strategy is best isolated away from other tests. Tying these ideas together are reference-counting seqs and the is-strong testing macro. ref-counting-lazy-seq returns a lazy seq and an atom of all elements of the seq currently with strong references. This seq can now be passed to a sequence-processing function you would like to test for memory leaks. is-strong then takes a set of seq indexes expected to have strong references and checks them against the atom tracking strong references. It will continue to induce OutOfMemoryError's until the expected references are found, or eventually fail via an is test assertion. Note that it's best to build up the set of expected strong references rather than whittle it down. From a usability standpoint, starting with (is-strong #{} live) will likely print the actual set of strongly referenced indexes, from which you can source the expected strong references (from the result itself or a subset). But false-positives are possible if an overly broad superset of the actual strong references are provided, since is-strong may short-circuit its search earlier than the cleaners can run since. Such results can lead to false conclusions about how lazy a function is, especially if you have a pre-conceived notion of the results! The JVM property io.github.frenchy64.fully-satisfies.leaky-seq-detection.is-strong.false-positive-detection=true will more aggressively search for such false-positives, which may be suitable for a cron CI job. Here's an example of asserting that a program adds or subtracts strong references to elements of a lazy seq at particular points. (deftest example-cleaners-test (let [{:keys [strong lseq]} (ref-counting-lazy-seq {:n 10}) ;; seq of fresh Object's, length 10 ;; lseq=(...) _ (is-strong #{} strong) ;; no elements currently in memory lseq (seq lseq) ;; lseq=(0 ...) _ (is-strong #{0} strong) ;; just the first element in memory _ (nnext lseq) ;; lseq=(0 1 2...) _ (is-strong #{0 1 2} strong) ;; the first 3 elements in memory lseq (next lseq) ;; lseq=(1 2 ...) _ (is-strong #{1 2} strong) ;; the second 2 elements in memory lseq (rest lseq) ;; lseq=(2 ...) _ (is-strong #{2} strong) ;; the third element in memory lseq (rest lseq) ;; lseq=(...) _ (is-strong #{} strong) ;; no elements in memory ;; lseq=(3 ...) lseq (seq lseq) _ (is-strong #{3} strong) ;; fourth element in memory _ (class (first lseq)) ;; add a strong reference to lseq so previous line succeeds ;; lseq=nil _ (is-strong #{} strong) ;; lseq is entirely garbage collected ])) See io.github.frenchy64.fully-satisfies.leaky-seq-detection-test for real-world examples of finding memory leaks in Clojure functions, and then verifying fixes for them.
Linear-time sequence functions
Linear-time sequence functions
Implementations of clojure.core macros that don't leak implementation details.
Implementations of clojure.core macros that don't leak implementation details.
Implementations of clojure.core.async macros that don't leak implementation details.
Implementations of clojure.core.async macros that don't leak implementation details.
Implementations of clojure.core.logic.pldb macros that don't leak implementation details.
Implementations of clojure.core.logic.pldb macros that don't leak implementation details.
Implementations of clojure.core.match.debug macros that don't leak implementation details.
Implementations of clojure.core.match.debug macros that don't leak implementation details.
Implementations of clojure.java.jmx macros that don't leak implementation details.
Implementations of clojure.java.jmx macros that don't leak implementation details.
Implementations of clojure.java.test macros that don't leak implementation details.
Implementations of clojure.java.test macros that don't leak implementation details.
Implementations of clojure.pprint macros that don't leak implementation details.
Implementations of clojure.pprint macros that don't leak implementation details.
Implementations of clojure.test macros that don't leak implementation details.
Implementations of clojure.test macros that don't leak implementation details.
Implementations of clojure.test.check.generators macros that don't leak implementation details.
Implementations of clojure.test.check.generators macros that don't leak implementation details.
Implementations of clojure.test.check.properties macros that don't leak implementation details.
Implementations of clojure.test.check.properties macros that don't leak implementation details.
Implementations of clojure.test.tap macros that don't leak implementation details.
Implementations of clojure.test.tap macros that don't leak implementation details.
Implementations of clojure.tools.trace macros that don't leak implementation details.
Implementations of clojure.tools.trace macros that don't leak implementation details.
Provides a spec for clojure.core/reify via ::reify-args.
To register a spec for reify, call (register-reify-spec).
Provides a spec for clojure.core/reify via ::reify-args. To register a spec for reify, call (register-reify-spec).
An alternative to clojure.core/run!
that does not short-circuit on reduced.
An alternative to `clojure.core/run!` that does not short-circuit on reduced.
Variants of clojure.core functions that improve thread-safety and general robustness when passed mutating collections.
We agree that 'Robust programs should not mutate arrays or Iterables that have seqs on them.' https://clojure.org/reference/sequences Eductions inhabit a middle ground. They are designed to be walked from first to last like seqs, but each element is recomputed when
For example, clojure.core/split-at could disagree on the take/drop parts of the collection if the coll is mutated between realizing the splits. Here, any one call to the eduction alternates between [0 1 2 3 4 5 6 7 8 9] and [9 8 7 6 5 4 3 2 1 0]. However, split-at incorrectly splits the eduction as [[0 1 2 3 4] [4 3 2 1 0]], and safer/split-at correctly splits it as [[0 1 2 3 4] [5 6 7 8 9]].
(deftest split-at-mutation-test (let [up-down (atom true) ed (eduction (map (fn [i] (when (zero? i) (swap! up-down not)) (if @up-down (- 9 i) i))) (range 10))] (is (= [[0 1 2 3 4] [4 3 2 1 0]] (split-at 5 ed))) (is (= [[0 1 2 3 4] [5 6 7 8 9]] (safer/split-at 5 ed)))))
See io.github.frenchy64.fully-satisfies.safer-test for more details.
The basic trick here is strategically calling seq earlier on the collection argument.
Variants of clojure.core functions that improve thread-safety and general robustness when passed mutating collections. We agree that 'Robust programs should not mutate arrays or Iterables that have seqs on them.' https://clojure.org/reference/sequences Eductions inhabit a middle ground. They are designed to be walked from first to last like seqs, but each element is recomputed when For example, clojure.core/split-at could disagree on the take/drop parts of the collection if the coll is mutated between realizing the splits. Here, any one call to the eduction alternates between [0 1 2 3 4 5 6 7 8 9] and [9 8 7 6 5 4 3 2 1 0]. However, split-at incorrectly splits the eduction as [[0 1 2 3 4] [4 3 2 1 0]], and safer/split-at correctly splits it as [[0 1 2 3 4] [5 6 7 8 9]]. (deftest split-at-mutation-test (let [up-down (atom true) ed (eduction (map (fn [i] (when (zero? i) (swap! up-down not)) (if @up-down (- 9 i) i))) (range 10))] (is (= [[0 1 2 3 4] [4 3 2 1 0]] (split-at 5 ed))) (is (= [[0 1 2 3 4] [5 6 7 8 9]] (safer/split-at 5 ed))))) See io.github.frenchy64.fully-satisfies.safer-test for more details. The basic trick here is strategically calling seq earlier on the collection argument.
An implementation of clojure.core/some-fn with a simple definitional equivalence.
An implementation of clojure.core/some-fn with a simple definitional equivalence.
Drop-in replacements for clojure.test/{deftest,testing}
that (when used
together) enhances uncaught exception error messages with the (most likely)
testing context it was thrown from.
Example:
(deftest my-test (testing "foo" (doseq [v [1 2 3]] (testing v (assert (= 1 v))))))
With clojure.test/{deftest,testing} 1.10.3 (notice foo 2
is not mentioned):
user=> (test-var #'my-test) ;ERROR in (my-test) ;Uncaught exception, not in assertion. ;expected: nil ;actual: java.lang.AssertionError: Assert failed: false ;...
With {deftest,testing} in this namespace (notice foo 2
is mentioned):
user=> (test-var #'my-test) ;ERROR in (my-test) ;Uncaught exception, possibly thrown in testing context: foo 2 ;expected: nil ;actual: java.lang.AssertionError: Assert failed: false ;...
Drop-in replacements for `clojure.test/{deftest,testing}` that (when used together) enhances uncaught exception error messages with the (most likely) testing context it was thrown from. Example: (deftest my-test (testing "foo" (doseq [v [1 2 3]] (testing v (assert (= 1 v)))))) With clojure.test/{deftest,testing} 1.10.3 (notice `foo 2` is not mentioned): user=> (test-var #'my-test) ;ERROR in (my-test) ;Uncaught exception, not in assertion. ;expected: nil ;actual: java.lang.AssertionError: Assert failed: false ;... With {deftest,testing} in this namespace (notice `foo 2` is mentioned): user=> (test-var #'my-test) ;ERROR in (my-test) ;Uncaught exception, possibly thrown in testing context: foo 2 ;expected: nil ;actual: java.lang.AssertionError: Assert failed: false ;...
Variants of clojure.core functions that are generalized to work uniformly for all values.
In all cases, a namespaced keyword was used as a special value that, if provided to the function, would break its promised semantics. The fixes involved replacing these special values with globally unique ones that are inaccessible to normal users (or more practically, unlikely to be generated with a generator like gen/any, or one that sources its values from the keyword interning table).
These corner-cases are demonstrated in io.github.frenchy64.fully-satisfies.uniform-test.
An effective generator that could find such defects reliably could generate keywords that occur in the source code of the functions reachable from the generative property (using static analysis, but perhaps this is also retrievable dynamically from the bytecode). On the other hand, such invasive analyses could yield false-negatives by providing values normally inaccessible to the user.
Variants of clojure.core functions that are generalized to work uniformly for all values. In all cases, a namespaced keyword was used as a special value that, if provided to the function, would break its promised semantics. The fixes involved replacing these special values with globally unique ones that are inaccessible to normal users (or more practically, unlikely to be generated with a generator like gen/any, or one that sources its values from the keyword interning table). These corner-cases are demonstrated in io.github.frenchy64.fully-satisfies.uniform-test. An effective generator that could find such defects reliably could generate keywords that occur in the source code of the functions reachable from the generative property (using static analysis, but perhaps this is also retrievable dynamically from the bytecode). On the other hand, such invasive analyses could yield false-negatives by providing values normally inaccessible to the user.
cljdoc is a website building & hosting documentation for Clojure/Script libraries
× close