JayKobe6k/ClariTree

Watch

Fork 0

Reduce LadybugDB publication time and memory with typed node COPY #40

New issue

Closed

opened 2026-09-11 00:44:28 +00:00 by JayKobe6k · 2 comments

JayKobe6k commented

2026-09-11 00:44:28 +00:00

Owner

Problem and impact

ClariTree writes source-anchor and semantic value nodes through prepared UNWIND parameter batches even though value relationships already use LadybugDB CSV COPY. On a representative 40-translation-unit workload, node and anchor publication account for most persistence time and retain substantially more native memory. This makes index creation and update slower and raises OOM pressure without improving correctness.

Environment and workload

ClariTree baseline: 654b2fbcf8e3ccc7ce7428b637d8b2736f67de19
LadybugDB crate: lbug 0.20.2
Linux x86_64; Rust/Cargo 1.93.0; Clang/libclang 21.1.8
Corpus: melee-event-match at 6b6f8a8a987c2e871b8b96baa863aaa99e406f33
Deterministic first 40 and 200 compilation-plan entries
Release harness: tests/profile_lineage.rs
Buffer pool: 4096 MiB
Cgroups: 40 TU MemoryMax=8G; 200 TU MemoryMax=16G; MemorySwapMax=0; explicit runtime limits

Reproduction after building the release profile test:

nix develop --command cargo test --release --test profile_lineage --no-run
systemd-run --user --wait --collect --pipe \
  --property=MemoryHigh=7G --property=MemoryMax=8G \
  --property=MemorySwapMax=0 --property=RuntimeMaxSec=300 \
  nix develop /path/to/ClariTree --command env \
  CLARITREE_PROFILE_PROJECT=/path/to/melee-event-match \
  CLARITREE_PROFILE_LIMIT=40 CLARITREE_PROFILE_PERSISTENCE=1 \
  CLARITREE_PROFILE_BUFFER_POOL_MIB=4096 \
  /path/to/ClariTree/target/release/deps/profile_lineage-<hash> \
  --ignored --nocapture

Verify /tmp user quota as well as free space, or set TMPDIR before nix develop; one 200-TU repeat was excluded after an environmental Disk quota exceeded checkpoint failure.

Baseline and profiling evidence

The 40-TU baseline persisted 28,375 definitions, 28,847 uses, 62,693 lineage relationships, and 729,601 source facts in 50.960 s, peaking near 5.6 GiB. Phase timing localized approximately 16.948 s to source nodes, 14.183 s to source relationships, 8.508 s to definitions, and 7.711 s to uses; validation was under approximately 1.4 s.

A typed-COPY candidate produced identical counts in three no-swap runs:

Run	Persistence	Peak memory
1	29.517 s	2.1G
2	28.315 s	2.0G
3	25.155 s	2.0G

The 28.315 s median is 44.4% below baseline and peak memory is approximately 63% lower. A comparable 200-TU tmpfs run fell from 318.381 s / 7,597,436 KiB to 201.081 s / 5.7G, a directional 36.8% time reduction and roughly 25% lower peak. Two root-disk candidate runs completed in 261.530 s and 316.149 s at 5.7G but have no matching root-disk baseline and are not used for the improvement claim.

Root cause

Baseline value-node writers build wide typed parameter rows and execute prepared UNWIND batches: src/persistence/ladybug.rs lines 3501-3600.
Source anchors and their four endpoint families follow the same parameter path: src/persistence/ladybug_source.rs lines 1571-1697.
The generic parameter batching path constructs a LIST<STRUCT> and checkpoints each batch: src/persistence/ladybug_source.rs lines 1377-1426.
Existing relationship COPY demonstrates a bounded native bulk-import path: src/persistence/ladybug_bulk.rs lines 232-337.

The causal mechanism is repeated construction and native conversion of very wide parameter structs plus less efficient per-row insertion. Bounded CSV COPY removes that conversion path and releases each temporary file after one configured chunk.

Proposed implementation

Generalize the typed relationship COPY adapter to explicit node and relationship targets.
Import SourceAnchor, ValueDefinition, ValueUse, and all four source-anchor endpoint families in configured bounded chunks.
Keep schema order, target selection, CSV encoding, and raw Cypher private to the LadybugDB adapter.
Quote every present CSV value while leaving absent values as unquoted empty fields, preserving Some("") versus NULL; double embedded quotes and preserve commas, newlines, backslashes, Unicode, and BLOB \xNN bytes. Disable COPY auto-detection and set delimiter, quote, escape, header, and parallel options explicitly.
Preserve explicit checkpoints, staging rollback, count/fingerprint validation, read-only reopen validation, and existing public APIs.
Update batch-setting help and persistence architecture documentation.

Alternatives considered:

Removing explicit per-batch checkpoints changed 50.960 s to 48.519 s with unchanged 5.6G peak; this was within observed variance and was reverted.
Buffer-pool tuning is separately tracked by #39 and did not materially change this workload above 128 MiB.
Retaining undocumented environment switches for legacy writers would add production behavior and global-test hazards; use commit-based baselines instead.

Expected benefit, confidence, and risks

Expected: 35-45% lower 40-TU persistence time and materially lower peak memory; confidence high for this corpus and medium for other projects.
Risks: schema-column drift, null/empty-string collapse, CSV escaping errors, BLOB corruption, and native COPY/checkpoint failures.
Compatibility: no schema or frontend contract change; batch option meanings broaden from parameter-only writes to typed COPY chunks.
Rollback: revert to the parameter writers; persisted schema and database compatibility are unchanged.

Acceptance criteria

Source anchors, value definitions, value uses, and all four source-anchor relationships use bounded typed COPY.
Exact property/decoder parity holds across one-row and multi-row chunks, including all six lineage families and declaration/ownership relationships.
Tests prove null versus empty-string distinction and preserve commas, quotes, newlines, backslashes, Unicode, and BLOB fingerprints.
Counts and relationship fingerprints remain identical for the pinned 40-TU and 200-TU workloads.
Three comparable 40-TU runs improve median persistence by at least 25% without increasing peak memory.
Atomic staging/publication, final checkpointing, durability reporting, and writable/read-only validation remain unchanged.
CLI help and architecture documentation describe the revised batch semantics.
Formatting, Clippy, all-target/all-feature tests, rustdoc, and Windows-target cross-compilation pass through Nix.

Related: #38 owns streaming typed mapping; #39 owns buffer-pool configuration.

## Problem and impact ClariTree writes source-anchor and semantic value nodes through prepared `UNWIND` parameter batches even though value relationships already use LadybugDB CSV `COPY`. On a representative 40-translation-unit workload, node and anchor publication account for most persistence time and retain substantially more native memory. This makes index creation and update slower and raises OOM pressure without improving correctness. ## Environment and workload - ClariTree baseline: `654b2fbcf8e3ccc7ce7428b637d8b2736f67de19` - LadybugDB crate: `lbug` 0.20.2 - Linux x86_64; Rust/Cargo 1.93.0; Clang/libclang 21.1.8 - Corpus: `melee-event-match` at `6b6f8a8a987c2e871b8b96baa863aaa99e406f33` - Deterministic first 40 and 200 compilation-plan entries - Release harness: `tests/profile_lineage.rs` - Buffer pool: 4096 MiB - Cgroups: 40 TU `MemoryMax=8G`; 200 TU `MemoryMax=16G`; `MemorySwapMax=0`; explicit runtime limits Reproduction after building the release profile test: ```sh nix develop --command cargo test --release --test profile_lineage --no-run systemd-run --user --wait --collect --pipe \ --property=MemoryHigh=7G --property=MemoryMax=8G \ --property=MemorySwapMax=0 --property=RuntimeMaxSec=300 \ nix develop /path/to/ClariTree --command env \ CLARITREE_PROFILE_PROJECT=/path/to/melee-event-match \ CLARITREE_PROFILE_LIMIT=40 CLARITREE_PROFILE_PERSISTENCE=1 \ CLARITREE_PROFILE_BUFFER_POOL_MIB=4096 \ /path/to/ClariTree/target/release/deps/profile_lineage-<hash> \ --ignored --nocapture ``` Verify `/tmp` user quota as well as free space, or set `TMPDIR` before `nix develop`; one 200-TU repeat was excluded after an environmental `Disk quota exceeded` checkpoint failure. ## Baseline and profiling evidence The 40-TU baseline persisted 28,375 definitions, 28,847 uses, 62,693 lineage relationships, and 729,601 source facts in **50.960 s**, peaking near **5.6 GiB**. Phase timing localized approximately 16.948 s to source nodes, 14.183 s to source relationships, 8.508 s to definitions, and 7.711 s to uses; validation was under approximately 1.4 s. A typed-COPY candidate produced identical counts in three no-swap runs: | Run | Persistence | Peak memory | |---|---:|---:| | 1 | 29.517 s | 2.1G | | 2 | 28.315 s | 2.0G | | 3 | 25.155 s | 2.0G | The **28.315 s median is 44.4% below baseline** and peak memory is approximately **63% lower**. A comparable 200-TU tmpfs run fell from **318.381 s / 7,597,436 KiB** to **201.081 s / 5.7G**, a directional 36.8% time reduction and roughly 25% lower peak. Two root-disk candidate runs completed in 261.530 s and 316.149 s at 5.7G but have no matching root-disk baseline and are not used for the improvement claim. ## Root cause - Baseline value-node writers build wide typed parameter rows and execute prepared `UNWIND` batches: [src/persistence/ladybug.rs lines 3501-3600](https://git.jacoby6000.com/JayKobe6k/ClariTree/src/commit/654b2fbcf8e3ccc7ce7428b637d8b2736f67de19/src/persistence/ladybug.rs#L3501-L3600). - Source anchors and their four endpoint families follow the same parameter path: [src/persistence/ladybug_source.rs lines 1571-1697](https://git.jacoby6000.com/JayKobe6k/ClariTree/src/commit/654b2fbcf8e3ccc7ce7428b637d8b2736f67de19/src/persistence/ladybug_source.rs#L1571-L1697). - The generic parameter batching path constructs a `LIST<STRUCT>` and checkpoints each batch: [src/persistence/ladybug_source.rs lines 1377-1426](https://git.jacoby6000.com/JayKobe6k/ClariTree/src/commit/654b2fbcf8e3ccc7ce7428b637d8b2736f67de19/src/persistence/ladybug_source.rs#L1377-L1426). - Existing relationship COPY demonstrates a bounded native bulk-import path: [src/persistence/ladybug_bulk.rs lines 232-337](https://git.jacoby6000.com/JayKobe6k/ClariTree/src/commit/654b2fbcf8e3ccc7ce7428b637d8b2736f67de19/src/persistence/ladybug_bulk.rs#L232-L337). The causal mechanism is repeated construction and native conversion of very wide parameter structs plus less efficient per-row insertion. Bounded CSV COPY removes that conversion path and releases each temporary file after one configured chunk. ## Proposed implementation 1. Generalize the typed relationship COPY adapter to explicit node and relationship targets. 2. Import `SourceAnchor`, `ValueDefinition`, `ValueUse`, and all four source-anchor endpoint families in configured bounded chunks. 3. Keep schema order, target selection, CSV encoding, and raw Cypher private to the LadybugDB adapter. 4. Quote every present CSV value while leaving absent values as unquoted empty fields, preserving `Some("")` versus `NULL`; double embedded quotes and preserve commas, newlines, backslashes, Unicode, and BLOB `\xNN` bytes. Disable COPY auto-detection and set delimiter, quote, escape, header, and parallel options explicitly. 5. Preserve explicit checkpoints, staging rollback, count/fingerprint validation, read-only reopen validation, and existing public APIs. 6. Update batch-setting help and persistence architecture documentation. Alternatives considered: - Removing explicit per-batch checkpoints changed 50.960 s to 48.519 s with unchanged 5.6G peak; this was within observed variance and was reverted. - Buffer-pool tuning is separately tracked by #39 and did not materially change this workload above 128 MiB. - Retaining undocumented environment switches for legacy writers would add production behavior and global-test hazards; use commit-based baselines instead. ## Expected benefit, confidence, and risks - Expected: 35-45% lower 40-TU persistence time and materially lower peak memory; confidence high for this corpus and medium for other projects. - Risks: schema-column drift, null/empty-string collapse, CSV escaping errors, BLOB corruption, and native COPY/checkpoint failures. - Compatibility: no schema or frontend contract change; batch option meanings broaden from parameter-only writes to typed COPY chunks. - Rollback: revert to the parameter writers; persisted schema and database compatibility are unchanged. ## Acceptance criteria - [ ] Source anchors, value definitions, value uses, and all four source-anchor relationships use bounded typed COPY. - [ ] Exact property/decoder parity holds across one-row and multi-row chunks, including all six lineage families and declaration/ownership relationships. - [ ] Tests prove null versus empty-string distinction and preserve commas, quotes, newlines, backslashes, Unicode, and BLOB fingerprints. - [ ] Counts and relationship fingerprints remain identical for the pinned 40-TU and 200-TU workloads. - [ ] Three comparable 40-TU runs improve median persistence by at least 25% without increasing peak memory. - [ ] Atomic staging/publication, final checkpointing, durability reporting, and writable/read-only validation remain unchanged. - [ ] CLI help and architecture documentation describe the revised batch semantics. - [ ] Formatting, Clippy, all-target/all-feature tests, rustdoc, and Windows-target cross-compilation pass through Nix. Related: #38 owns streaming typed mapping; #39 owns buffer-pool configuration.

JayKobe6k self-assigned this

2026-09-11 00:44:42 +00:00

JayKobe6k commented

2026-09-11 00:51:45 +00:00

Author

Owner

The implementation candidate now passes the complete required validation through the pinned Nix environment:

cargo fmt --all -- --check
cargo lint
cargo test --all-targets --all-features (274 unit tests plus every integration suite)
cargo doc --no-deps --all-features
cargo check --all-targets --all-features --target x86_64-pc-windows-gnu
git diff --check

Review found and fixed a null/empty-string CSV distinction defect. Present fields are now always quoted while absent fields remain unquoted; adversarial tests cover empty strings, nulls, commas, quotes, newlines, backslashes, Unicode, and fingerprint BLOB encoding. An initially added standalone native database test caused parallel-suite interference and was removed; the existing value-node database test now verifies the distinction without adding another concurrent LadybugDB instance. A fresh post-fix review found no remaining actionable defect.

The candidate remains uncommitted in the local worktree; no commit or push has been claimed.

The implementation candidate now passes the complete required validation through the pinned Nix environment: - `cargo fmt --all -- --check` - `cargo lint` - `cargo test --all-targets --all-features` (274 unit tests plus every integration suite) - `cargo doc --no-deps --all-features` - `cargo check --all-targets --all-features --target x86_64-pc-windows-gnu` - `git diff --check` Review found and fixed a null/empty-string CSV distinction defect. Present fields are now always quoted while absent fields remain unquoted; adversarial tests cover empty strings, nulls, commas, quotes, newlines, backslashes, Unicode, and fingerprint BLOB encoding. An initially added standalone native database test caused parallel-suite interference and was removed; the existing value-node database test now verifies the distinction without adding another concurrent LadybugDB instance. A fresh post-fix review found no remaining actionable defect. The candidate remains uncommitted in the local worktree; no commit or push has been claimed.

JayKobe6k commented

2026-09-11 01:03:16 +00:00

Author

Owner

Implemented and pushed to main as 32959f55abed0b7bd92d94670fcf9c9abb452a1e.

The delivered implementation retains the measured 40-TU median improvement (50.960 s to 28.315 s, 44.4%) and approximately 63% lower peak memory, with exact count parity. Full required Nix validation passed before commit, and commit hooks passed formatting and linting.

Implemented and pushed to `main` as [`32959f55abed0b7bd92d94670fcf9c9abb452a1e`](https://git.jacoby6000.com/JayKobe6k/ClariTree/commit/32959f55abed0b7bd92d94670fcf9c9abb452a1e). The delivered implementation retains the measured 40-TU median improvement (50.960 s to 28.315 s, 44.4%) and approximately 63% lower peak memory, with exact count parity. Full required Nix validation passed before commit, and commit hooks passed formatting and linting.

JayKobe6k closed this issue

2026-09-11 01:03:16 +00:00

JayKobe6k referenced this issue

2026-09-11 01:03:16 +00:00

Bound indexing peak memory by streaming typed persistence mapping #38

JayKobe6k referenced this issue

2026-09-11 02:25:27 +00:00

Reduce high-volume source-fact publication with typed COPY #41