Scientific-method ledger for the non-ARTrie tech-debt repair plan tracked at
/home/dylon/.claude/plans/rust-backtrace-1-rust-log-debug-cargo-n-purrfect-lemon.md.
For the per-phase architectural refactor (Phase 3 dedup work, where benchmark
deltas matter), entries go in tier1-ledger.md instead. Use this file for
correctness fixes, API changes, doc updates, and CI/infra changes — anything
where the metric of success is test-count delta, warning-count delta, doc
accuracy, or new-test coverage rather than throughput/latency.
Per item:
| Field | Content |
|---|---|
| Item | Plan ID (e.g., Z2, A1, B7, D4) |
| Date | YYYY-MM-DD (resolved) |
| Commit | git SHA |
| Before | Warning count / test count / doc claim |
| After | Same metrics post-fix |
| New tests | Path + test name list |
| Notes | Surprises, follow-up risks |
| Item | Date | Commit | Before | After | New tests | Notes |
|---|---|---|---|---|---|---|
| Z1 | 2026-05-20 | b7630ad | Proofs/MapRefinement.v was untracked per the audit | Already committed (audit was stale); no action needed beyond verification | — | Discovered via git ls-files. |
| Z2 | 2026-05-21 | (pending Phase 0 commit) | VERIFICATION_RESULTS.md claimed 10 modules / ~2700 LOC, listed 2 outstanding Admitted theorems | Now lists 15 modules / ~6503 LOC, 0 Admitted / 0 Axiom; per-file proof tally + commit-link to b7630ad + efe1943 | — | "Insufficient Hypotheses" reframed as resolved; recommendations 1-3 marked done. |
| Z3 | 2026-05-21 | (pending) | formal-verification/README.md Proof Status table marked Bucket/PathCompression/ARTrieSpec/StructuralInvariants as "Partial"; directory tree omitted 5 .v files; Operations/ + Proofs/ marked "(TODO)"; Future Work #1/#2 outstanding | All 15 modules listed as "Complete (0 Admitted)"; directory tree updated; Operations/ documented as reserved-for-extraction; Future Work #1/#2 marked Done with proof refs | — | Added proof-reference lines to "Key Theorems" pointing at ARTrieSpec.v:672,685 and MapRefinement.v. |
| Z4 | 2026-05-21 | (pending) | .gitignore missed formal-verification/rocq/**/.*.aux | Added; .aux files now explicitly ignored | — | Cosmetic — they were untracked due to dot-prefix anyway. |
| D6 | 2026-05-21 | (pending) | build.rs had no cargo:rerun-if-changed for the .proto file | Emits cargo:rerun-if-changed=proto/libdictenstein.proto when protobuf feature is on | — | One-line fix. |
| D4 | 2026-05-21 | (pending) | Cargo.toml declared 3 zero-use features (simd, scdawg-bloom, scdawg-simd); group-commit mislabeled "REJECTED" | 3 dead features removed; group-commit relabeled EXPERIMENTAL with cross-ref to regression doc; README.md Features section now matches reality (11 features listed) | — | Verified rg --type rust 'feature\s*=\s*"…"' returned 0 hits for each removed feature. |
| D8 | 2026-05-21 | (pending) | 5 *-results.log files at repo root (Jan dates, no regen script) | Moved to docs/sanitizers/ with date-stamped filenames + docs/sanitizers/README.md archive doc + scripts/run-sanitizers.sh regen script | — | Files were already gitignored (*_results.log), so no git mv — plain mv. |
cargo build --features persistent-artrie → clean (12 warnings, 0 errors)cargo build --all-features → clean (19 warnings, 0 errors)cd formal-verification/rocq && make → Nothing to be done for 'all' (15/15 .vo already current)git status → 6 modified, 3 new directories/files, all expectedunused Result warnings in
dict_impl.rs:518 and mmap_ctor.rs:1168 and similar sites. These are
Phase-1/Phase-7-flavored cleanup items, not regressions from Phase 0. The
prior ARTrie plan's Phase-7 cleanup dropped warnings from 74 to 13; the
remaining 13 are dead-code warnings about helper methods, not import issues.CHANGELOG.md created (this is the first Phase 0 change captured there).| Item | Date | Before | After | New tests | Notes |
|---|---|---|---|---|---|
| A1 (BijectiveDictionary::get_term → Cow) | 2026-05-21 | Trait returned Option<&str>; BijectiveMap used unsafe pointer dereference; vocab impls stubbed to None (violated bijection invariant) | Returns Option<Cow<'_, str>>. BijectiveMap clones into Cow::Owned (no unsafe). Vocab impls reconstruct via parent-pointer backtracking and return Cow::Owned | tests/bijective_trait_invariant.rs — 3 tests | Predicate fix: tests use .as_deref() since Cow doesn't directly equal &str. |
| A2 (MappedDictionary::get_value default) | 2026-05-21 | Default discarded contains result, returned None. Any impl that forgot to override was silently broken | Required method. All 15 impls already override; build catches new impls that miss | Build alone | One-line trait change. |
| A3 (value-preserving serializers) | 2026-05-21 | serialize/deserialize shipped Vec<String> over the wire — values dropped on round-trip. MappedDictionary impls reloaded with get_value == None | Added DictionaryFromTermsWithValues trait + extract_terms_with_values helper + serialize_with_values/deserialize_with_values on bincode/json/plaintext. Added from_terms_with_values inherent on DAWG / DAWG-char / DAWG-u64 / Scdawg | tests/serialization_value_roundtrip.rs — 6 tests (5 round-trip + 1 legacy-path regression guard) | Char/u64 backend support deferred to B2 (needs extract_terms generalized over Unit). |
| A4 (SharedVocabARTrie mapped mutation semantics) | 2026-05-21, superseded 2026-06-13 | 7 trait methods silently dropped user input (insert_with_value ignored value, union_with ignored merge_fn, update_or_insert ignored both, remove/remove_prefix/upsert ignored args, increment returned Err) | Initial hardening made unsupported calls warn instead of silently discarding input. The current implementation has since replaced the mapped-write stubs: insert_with_value, update_or_insert, and union_with now perform real index-aware writes for PersistentVocabARTrie and SharedVocabARTrie; append-only removal-style methods still return documented sentinel values. | tests/vocab_trait_honesty.rs; tests/persistent_vocab_wal_atomicity_correspondence.rs; tests/persistent_vocab_checkpoint_correspondence.rs; tests/persistent_shared_concurrency_correspondence.rs; tests/dictionary_law_correspondence.rs; tests/bijective_trait_invariant.rs | Current contract: supplied u64 values are requested vocabulary indices for new terms, existing indices are immutable, missing union terms preserve source indices, and conflicting merge results warn rather than remap a term. |
| A5 (matches_any naming) | 2026-05-21 | Vec/HashSet/SmallVec::matches_any and matches_all were identical, both treating the whole collection as the predicate target. Vec's body admitted the bug ("real usage would pass element predicates") | Added type Atom: ?Sized to FilterableValue. Scalars: Atom = Self. Collections: Atom = T, with per-element semantics via self.iter().any(predicate) / .all(predicate) | 7 unit tests updated, 3 new collection tests | Trait was used only inside value.rs, so the breaking change has no external blast radius. |
| A6 (extract_terms iterative) | 2026-05-21 | Recursive DFS in serialization::extract_terms; pathological deep single-child chains (~80k+) overflow the default 8MB thread stack | Iterative Vec-stack rewrite with per-frame edge collection. Pop the descent byte on backtrack via current_term.truncate(frame.depth) | 2 new tests: 1k-deep DAT + 50k-deep DynamicDawg | Bug found during implementation: original "depth = current_term.len() AFTER push" caused wrong truncation on backtrack. Fixed to "depth = len BEFORE push." |
| A7 (unwrap → expect) | 2026-05-21 | 5 unwrap() sites in production code without forensic messages | 5 sites replaced with .expect(...) carrying the invariant that proves the option is Some | None (sites already covered) | Sites: suffix_automaton{,_char}.rs (suffix-link walk + as_mut), dynamic_dawg_char.rs (sig_to_canonical get_mut). bijective_map.rs panics were intentional with messages; left as-is. |
| A8 (transducer doc rot) | 2026-05-21 | 6 rust,ignore doc-tests in suffix_automaton{,_char}.rs referenced a non-existent libdictenstein::transducer module | Rewrote 6 doc-tests to use libdictenstein::prelude::* + the actual SuffixAutomaton{,Char} API. Documented that the transducer lives downstream in liblevenshtein. All 6 converted from rust,ignore to rust (now actually compile + execute) | 6 doc-tests promoted | Also fixed a pre-existing bug at prefetch_api.rs:14: orphan /// ```` opening a never-closed doc fence that brokecargo test --doc` whenever this file was visited. |
cargo build --all-features → clean (19 warnings, 0 errors)cargo test --all-features --no-fail-fast → 2257 passed, 1 flaky (io_uring concurrent allocation test, passed on retry), 175 ignored| Item | Date | Before | After | New tests | Notes |
|---|---|---|---|---|---|
| B1 (DictionaryFactory expansion) | 2026-05-21 | 4 backends exposed (PathMap, DAT, DynamicDawg, SuffixAutomaton) | 11 backends — added DoubleArrayTrieChar, DynamicDawgChar, DynamicDawgU64, SuffixAutomatonChar, Scdawg, ScdawgChar, PathMapDictionaryChar. Persistent ARTrie family deliberately excluded (needs file paths). | +1 unicode_backends test, +1 grew test counts | DoubleArrayTrieChar uses empty() not new(). |
| B2 (Unit-generic serializer, partial) | 2026-05-21 | DictionarySerializer and extract_terms/extract_terms_with_values forced D::Node: DictionaryNode<Unit = u8>. Char/u64 backends couldn't round-trip via the value-preserving path | Added extract_terms_char + extract_terms_with_values_char parallel helpers. Added serialize_with_values_char methods on bincode/json/plaintext (deserialize_with_values is unit-agnostic since wire format = Vec<(String, V)>) | 3 new char-unicode roundtrip tests | u64 (DynamicDawgU64) deferred — u64-keyed terms don't trivially round-trip through String; needs format design. |
| B3 (Scdawg parity) | 2026-05-21 | Scdawg<V> missing inherent get_value and MappedDictionary impl. ScdawgChar<V> had get_value but no MappedDictionary impl either | Added inherent Scdawg::get_value. Added impl MappedDictionary for both Scdawg and ScdawgChar | Build alone | Inherent from_terms_with_values was added in A3. Now Scdawg variants can use the value-preserving serializers. |
| B4 (DAWG suffix-share cache doc) | 2026-05-21 | find_or_create_suffix + 3 helpers in dynamic_dawg{,_char}.rs marked #[allow(dead_code)] with brief NOTE | Created docs/dynamic_dawg/suffix_cache_bug.md with the invariant violation, design candidates, and re-enable checklist. In-code NOTE updated to reference the doc | None | Code stays per CLAUDE.md "never disable by deleting" rule. |
| B5 (DAT reserved fields) | 2026-05-21 | free_list and rebuild_threshold fields serialized but unread, brief #[allow(dead_code)] comment | Fields preserved (touching them breaks on-disk format) with detailed "RESERVED FOR FUTURE" docstrings explaining what they hold and why they're kept | Build alone | Conservative — bumping format version requires migration test infrastructure that doesn't exist yet. |
| B6 (MutableDictionary deprecation) | 2026-05-21 | Trait existed alongside MutableMappedDictionary with no documentation of the relationship | Trait kept (not deprecated — it has remove which MutableMappedDictionary lacks). Docstring rewritten to call out the complementarity: set-like vs value-aware, both useful, neither subsumed | Build alone | Plan's deprecation proposal didn't survive closer inspection; the traits are genuinely complementary. |
| B7 (ARTrieAtomicOps fold-in) | 2026-05-21 | ARTrieAtomicOps defined as an extension trait with increment/upsert/compare_and_swap — overlapping signatures with ARTrie (different return types); zero impl sites | Trait body commented out per CLAUDE.md; empty stub kept behind #[deprecated] so external callers naming the trait get a warning. Removed re-export warning by #[allow(deprecated)] on the pub use line | Build alone | compare_and_swap lives as inherent on PersistentARTrie{,Char} (atomic_ops.rs); never needed a trait. |
| B8 (EvictableARTrie &self) | 2026-05-21 | enable_eviction/disable_eviction/force_eviction took &mut self, defeating Arc-sharing of SharedARTrie<V> | Changed to &self on trait + 3 impl sites (SharedARTrie, SharedCharARTrie, SharedVocabARTrie) | Build alone | The impls already acquired write guards through &self internally; the &mut self bound was performative. |
| B9 (DurabilityPolicy return path) | 2026-05-21 | ARTrie::durability_policy() returned crate::persistent_artrie::dict_impl::DurabilityPolicy despite the real type living at persistent_artrie_core::durability::DurabilityPolicy | Return type points at the canonical core path; existing dict_impl::DurabilityPolicy re-export retained for back-compat for one release | Build alone | The byte-side pub use re-export already existed at dict_impl.rs:354 from a prior Phase. |
| B10 (iter_prefix_units) | 2026-05-21 | ARTrie::iter_prefix returned Box<dyn Iterator<Item = String>> only — lossy for non-Unit=u8 impls | Added sibling iter_prefix_units returning Box<dyn Iterator<Item = Vec<Self::Unit>>> with a default impl that round-trips through the existing iter_prefix. Backends with native unit-level traversal can override for efficiency | Build alone | Default fallback uses Self::Unit: From<u8> bound; impls without that bound must override. |
cargo build --all-features → clean (0 errors)cargo test --all-features --no-fail-fast → 2261 passed, 0 failed, 175 ignored#[allow(deprecated)])L-effort architectural refactors (C1/C3/C4/C5/C6) are tracked in tier1-ledger.md since they involve LOC-measurable decompositions modeled on the ARTrie Phase 5/6 pattern. Quick-win Tier C items handled in this session:
| Item | Date | Before | After | Notes |
|---|---|---|---|---|
| C2 (serde_helpers extraction) | 2026-05-21 | serialize_arc_vec / deserialize_arc_vec / serialize_arc_vec_vec / deserialize_arc_vec_vec duplicated byte-for-byte across double_array_trie.rs and double_array_trie_char.rs | Single canonical home at src/serialization/serde_helpers.rs; both DAT files use the helpers and reference them unqualified in the serde attribute strings | Build clean, 16 DAT tests pass. |
| C7 (sync_compat::RwLock parity) | 2026-05-21 | Std-fallback RwLock wrapper lacked try_read/try_write (parking_lot has both) — silent API divergence between backends | Added try_read/try_write to the std-fallback wrapper, returning Option<Guard> to match parking_lot's shape. Added 2 unit tests | Single canonical type per build (cfg-gated); the trait abstraction the audit suggested would add runtime cost for no real benefit (at most one backend compiles). |
Every Tier C item from the audit is now closed end-to-end. Each algorithmic-method block that used to be duplicated per byte/char variant now lives in exactly ONE canonical location:
| Item | Status | LOC delta | Canonical module |
|---|---|---|---|
| C1 (DAWG variants) | COMPLETE — node BloomFilter/NodeSignature shared via crate::bloom_filter/crate::node_signature (commit c4e2bc8); shared DawgNode<U, V> via crate::dawg_core::DawgNode (commit 5bbf835); shared inner state via type alias DynamicDawgInner = DawgCore<u8, V> + DynamicDawgCharInner = DawgCore<char, V> (commits ec28431, 4bd182d); u64 variant's lock-free architecture documented as deliberate divergence (commit cb54f34) | -1280 LOC across the family | src/dawg_core.rs (DawgCore<U, V>) |
| C3 (SuffixAutomaton variants) | COMPLETE — shared SuffixNode<U, V> (commit a6d461c); shared inner SuffixAutomatonInner<U, V> with the on-line extend(unit: U) algorithm (commit 0ea1c33). Byte and char variants alias to the canonical core | -395 LOC net | src/suffix_automaton_core/ |
| C4 (Scdawg variants) | COMPLETE — shared ScdawgNode<U, V> (commit a422d23); shared inner ScdawgCoreInner<U, V> with sa_extend / compute_left_edges / IS-features (commits 42ed7b6, a30aa4b, fc17450); cross-variant parity tests (commit fb96e78). +12 parity tests added | -660 LOC net | src/scdawg_core/ |
| C5 (DAT variants) | COMPLETE — shared DATCoreShared<U, V> storage struct (commits f88fa92, 2f1bb97); shared algorithmic methods contains_term_from / term_value_from parameterized by root_state (commits 0cdc1b7, e2eacca). Byte (root_state=1) and char (root_state=0) DATs both delegate to the canonical core | -150 LOC | src/dat_core/ |
| C6 (union_zipper) | COMPLETE — 1632-LOC god-object split into 4 modules: mod, merge_strategies, lattice, semiring_lattice (commit 583dac0) | net +42 LOC for module overhead | src/union_zipper/ |
Summary: All C1/C3/C4/C5/C6 architectural dedup items end-to-end
complete. Zero duplicate algorithmic methods between byte and char
variants. Every Inner state machine has exactly one canonical
implementation. No remaining "multi-week deferred" items in this Tier.
The original handoff docs (docs/benchmarks/c{1,3,4,5}-*-handoff.md)
remain available for historical reference of the migration plan that
was actually executed.
| Item | Date | Before | After | Notes |
|---|---|---|---|---|
| D1 (target-cpu=native scope) | 2026-05-21 | .cargo/config.toml set target-cpu=native unconditionally, producing binaries that silently emit illegal-instruction signals on slightly older x86_64 hardware | Replaced with target-feature=+aes,+sse2 — the minimum required by PathMap's gxhash. Comments explain how to opt back into native via RUSTFLAGS env var for release/bench builds | Build still clean. |
| Item | Date | Result |
|---|---|---|
| D5 (bincode 1.3 → 2.0) | 2026-05-21 | Done via src/serialization/bincode_compat.rs shim. bincode 2.x's bincode::serde::* API + bincode::config::legacy() config preserves bincode 1.x's wire-format byte-for-byte, so the format-version constant did not need bumping. All 105 call-sites migrated mechanically via sed. Test suite passes (2294 tests). lib.rs ARTrieAtomicOps re-export gated behind the persistent-artrie feature to fix the --features serialization standalone build. |
(Filled in.)
| Item | Date | Before | After | Notes |
|---|---|---|---|---|
| D9 (lib.rs backend table) | 2026-05-21 | Table listed 7 backends, omitted Scdawg/ScdawgChar/PathMap/PathMapChar and the entire persistent-ARTrie family | Two tables: in-memory (11 entries) + disk-backed (3 entries). All backends linked via intra-doc anchors | Pre-existing broken-link warnings in cargo doc are not from this change. |
| D9 (README Quick Start) | 2026-05-21 | Quick Start used use libdictenstein::prelude::*; but didn't explain why the prelude is needed; backend table missing 4 in-memory + 3 disk-backed entries | Comments explain prelude usage. README table mirrors the new lib.rs table; pointer to DictionaryFactory for unified construction |
| Item | Estimate | Reason for deferral |
|---|---|---|
| D7 (96 rust,ignore doc-tests) | 1 week | Each block needs manual review to decide between rust, rust,no_run, and updated source paths. 6 done as part of A8; remaining 90 require similar per-block judgment. |
| D9 (mmap-architecture.md path refresh) | 1-2 days | References old monolithic file paths (pre-Phase-6 ARTrie decomposition). |
| D9 (docs/algorithms/implementations/{scdawg,bijective}.md) | 2-3 days | Net-new files. |
cargo test --all-features --no-fail-fast: 2288 passed, 0 failed,
151 ignored across 26 test binaries.cargo fmt --all -- --check: clean.cd formal-verification/rocq && make: green (all 15 .v files up to
date).cargo build --all-features: 0 errors, ~19 warnings (all pre-existing
unused-Result warnings in legacy code paths).cargo test --all-features --doc: 152 passed, 0 failed, 150 ignored.bijective_trait_invariant.rs (3),
serialization_value_roundtrip.rs (10),
vocab_trait_honesty.rs (11). Plus +25 unit tests sprinkled across the
source tree (extract_terms deep-chain $\times$ 2, factory unicode_backends,
sync_compat try_read/try_write $\times$ 2, value FilterableValue per-element $\times$
3, suffix_automaton doc-tests $\times$ 6).The following plan items are L-effort architectural refactors that the plan itself estimated as multi-week or multi-day and cannot reasonably fit in a single session:
| Item | Estimate | Why deferred |
|---|---|---|
| C1 (DawgCore consolidation) | 2-3 weeks | ~1000-1200 LOC reduction across 3 DAWG variants (DynamicDawg, DynamicDawgChar, DynamicDawgU64). Mirrors ARTrie Phase 5. |
| C3 (SuffixAutomatonCore) | 3-4 weeks | ~1500 LOC reduction; involves redesigning the on-line suffix automaton state to be generic over Unit: CharUnit. |
| C4 (ScdawgCore) | 3-4 weeks | ~900 LOC reduction. Similar shape to C3. |
| C5 (DAT generic) | 1-2 weeks | 3000 LOC across 4 DAT files; the edges: Arc<Vec<Vec<u8>>> vs Arc<Vec<Vec<char>>> difference percolates through every method. Existing partial DATShared infrastructure helps but doesn't shortcut the bulk of the work. |
| D5 (bincode 1.x → 3.0 migration) | 2-4 days | bincode 3.0 has a completely different API surface (bincode::serde::encode_to_vec etc.); migration touches every serializer impl + every on-disk format check. |
| D7 (90 remaining rust,ignore doc-tests) | 1 week | Each block needs per-block API rewriting. 25 of 148 originally rust,ignore blocks were rescued; the remaining 123 demonstrate API patterns that need context-specific rewrites (e.g., let trie = PersistentARTrie::create("path")? followed by generic-ARTrie-trait method calls, which only work via SharedARTrie::create's Arc<RwLock<…>>-wrapped variant). |
These items are real but they're optimization opportunities, not correctness blockers. The code as it stands today is correct, well-tested (2288 passing tests), formatted (cargo fmt clean), documented, and CI-ed.
Updated /home/dylon/.claude/projects/-home-dylon-Workspace-f1r3fly-io-libdictenstein/memory/MEMORY.md
with the final state. The remaining items above are tracked as
follow-up sessions in that file as well.
Can you improve this documentation?Edit on GitHub
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |