Reduce high-volume source-fact publication with typed COPY #41

New issue

Closed

opened 2026-09-11 02:25:27 +00:00 by JayKobe6k · 1 comment

JayKobe6k commented

2026-09-11 02:25:27 +00:00

Owner

Problem and impact

After typed COPY was introduced for source anchors and semantic value nodes in #40, ClariTree still publishes the highest-volume source-fact nodes and structural relationships through prepared UNWIND parameter batches. On representative 40- and 200-translation-unit workloads these remaining writes dominate LadybugDB publication time. Index creation and updates therefore spend avoidable CPU and native-allocation work converting typed Rust records into wide parameter structures.

Environment and workload

ClariTree baseline: 32959f55abed0b7bd92d94670fcf9c9abb452a1e
LadybugDB crate: lbug 0.20.2
Linux 6.18.46 x86_64; Rust/Cargo 1.93.0; Clang/libclang 21.1.8
Corpus: melee-event-match at 6b6f8a8a987c2e871b8b96baa863aaa99e406f33
Deterministic first 40 and 200 compilation-plan entries
Release harness: tests/profile_lineage.rs
Buffer pool: 4096 MiB for normal comparable runs
Transient user cgroups with MemorySwapMax=0, explicit memory limits, and runtime limits

Reproduction after building the release profile test:

nix develop --command cargo test --release --test profile_lineage --no-run
systemd-run --user --wait --collect --pipe \
  --property=MemoryMax=32G --property=MemorySwapMax=0 \
  --property=TimeoutStartSec=900 \
  nix develop /path/to/ClariTree --command env \
  TMPDIR=/path/to/ClariTree/target/profile-tmp \
  CLARITREE_PROFILE_PROJECT=/path/to/melee-event-match \
  CLARITREE_PROFILE_LIMIT=40 CLARITREE_PROFILE_PERSISTENCE=1 \
  CLARITREE_PROFILE_BUFFER_POOL_MIB=4096 \
  cargo test --manifest-path /path/to/ClariTree/Cargo.toml \
  --release --test profile_lineage -- --ignored --nocapture

Baseline and profiling evidence

The 40-TU baseline persisted 28,375 definitions, 28,847 uses, 62,693 lineage relationships, and 729,601 source facts in 33.008 s and 33.797 s, a 33.403 s midpoint, at approximately 2.0-2.1 GiB peak RSS.

Three interleaved candidate runs completed in 22.774 s, 22.293 s, and 21.414 s. The 22.293 s median is 33.3% below the baseline midpoint, with identical semantic and persistence counts and approximately 1.2 GiB peak RSS. A final exact-tree run with the 4096 MiB compatibility default completed in 23.567 s at a 1.1 GiB cgroup peak with no swap.

A controlled 200-TU comparison produced identical 169,045 definitions, 180,182 uses, 400,939 lineage relationships, and 4,738,745 source facts:

Revision	Persistence	Peak memory
Baseline `32959f5`	217.743 s	5.7 GiB
Typed-COPY candidate	135.751 s	5.1 GiB

This is a 37.7% persistence-time reduction and approximately 10.5% lower peak memory.

A final 128 MiB buffer-pool run was excluded: LadybugDB returned Unable to allocate memory! The buffer pool is full and no memory could be freed! after a 1.1 GiB process peak. This confirms that the buffer-pool setting is not a total RSS limit and that undersized pools may fail publication, as documented by #39.

Root cause

Baseline source-fact node writes use one prepared parameter statement per node family, including SymbolObservation, SymbolReference, and Operation: source node writers.
High-cardinality ownership and symbol relationships also use prepared parameter batches: source relationship writers.
Operation inputs and definition-site relationships branch per endpoint type before repeatedly invoking the parameter writer: operation relationship writers.
The generic batching path materializes LadybugDB LIST<STRUCT> parameters for every configured batch: parameter batching.

The causal mechanism is repeated construction, conversion, and query execution for wide native parameter structures. Bounded typed CSV COPY removes that conversion path while preserving explicit schema targets and releasing each temporary import file after one chunk.

Proposed implementation

Add closed typed COPY targets for SymbolObservation, SymbolReference, and Operation nodes.
Add explicit typed targets for the high-volume source-fact relationship families, including ownership, symbol, operation input, source-anchor, result-type, and definition-site edges.
Project typed iterators directly into bounded COPY chunks without collecting a second full relationship graph.
Preserve exact property order, null-versus-empty behavior, enum encodings, endpoint direction, checkpoints, staging rollback, read-write validation, and read-only reopen validation.
Keep generic parameter writers for lower-volume families where conversion work is not a measured constraint.
Update CLI/rustdoc and database architecture documentation so source-fact batch settings explicitly cover typed COPY chunks.

Alternatives considered:

Tuning the LadybugDB buffer pool does not remove parameter conversion and can make publication fail when undersized.
Converting every low-volume table increases implementation and schema-drift surface without a measured material benefit.
Removing checkpoint or validation guarantees was rejected because durability and correctness take priority over benchmark gains.

Expected benefit, confidence, and risks

Expected: at least 25% lower persistence time on representative medium and large C workloads, with lower peak memory; confidence high for the pinned corpus and medium across other projects.
Risks: schema column-order drift, reversed same-type endpoints, enum/null encoding changes, CSV escaping defects, and native COPY failures.
Compatibility: no graph-schema, database-format, or frontend contract change.
Rollback: revert these families to the existing parameter writers; persisted databases remain compatible.

Acceptance criteria

The listed high-volume node and relationship families use bounded typed COPY.
Exact read-back tests cover migrated node properties, operation-input role/position nullability, relationship cardinality, and endpoint direction after read-only reopen.
Tiny configured chunks preserve complete mapping counts and append behavior.
The pinned 40-TU and 200-TU workloads preserve semantic and persisted counts.
Three comparable 40-TU runs improve median persistence by at least 25% without increasing peak memory.
The 200-TU workload confirms the improvement direction under a no-swap cgroup.
CLI help and database documentation describe the broader typed-COPY batch semantics.
Nix formatting, Clippy, all-target/all-feature tests, rustdoc, and Windows-target cross-compilation pass.

Related: #38 owns bounded persistence mapping, #39 owns buffer-pool configuration, and #40 introduced typed COPY for source anchors and semantic value nodes.

## Problem and impact After typed COPY was introduced for source anchors and semantic value nodes in #40, ClariTree still publishes the highest-volume source-fact nodes and structural relationships through prepared `UNWIND` parameter batches. On representative 40- and 200-translation-unit workloads these remaining writes dominate LadybugDB publication time. Index creation and updates therefore spend avoidable CPU and native-allocation work converting typed Rust records into wide parameter structures. ## Environment and workload - ClariTree baseline: `32959f55abed0b7bd92d94670fcf9c9abb452a1e` - LadybugDB crate: `lbug` 0.20.2 - Linux 6.18.46 x86_64; Rust/Cargo 1.93.0; Clang/libclang 21.1.8 - Corpus: `melee-event-match` at `6b6f8a8a987c2e871b8b96baa863aaa99e406f33` - Deterministic first 40 and 200 compilation-plan entries - Release harness: `tests/profile_lineage.rs` - Buffer pool: 4096 MiB for normal comparable runs - Transient user cgroups with `MemorySwapMax=0`, explicit memory limits, and runtime limits Reproduction after building the release profile test: ```sh nix develop --command cargo test --release --test profile_lineage --no-run systemd-run --user --wait --collect --pipe \ --property=MemoryMax=32G --property=MemorySwapMax=0 \ --property=TimeoutStartSec=900 \ nix develop /path/to/ClariTree --command env \ TMPDIR=/path/to/ClariTree/target/profile-tmp \ CLARITREE_PROFILE_PROJECT=/path/to/melee-event-match \ CLARITREE_PROFILE_LIMIT=40 CLARITREE_PROFILE_PERSISTENCE=1 \ CLARITREE_PROFILE_BUFFER_POOL_MIB=4096 \ cargo test --manifest-path /path/to/ClariTree/Cargo.toml \ --release --test profile_lineage -- --ignored --nocapture ``` ## Baseline and profiling evidence The 40-TU baseline persisted 28,375 definitions, 28,847 uses, 62,693 lineage relationships, and 729,601 source facts in 33.008 s and 33.797 s, a 33.403 s midpoint, at approximately 2.0-2.1 GiB peak RSS. Three interleaved candidate runs completed in 22.774 s, 22.293 s, and 21.414 s. The 22.293 s median is 33.3% below the baseline midpoint, with identical semantic and persistence counts and approximately 1.2 GiB peak RSS. A final exact-tree run with the 4096 MiB compatibility default completed in 23.567 s at a 1.1 GiB cgroup peak with no swap. A controlled 200-TU comparison produced identical 169,045 definitions, 180,182 uses, 400,939 lineage relationships, and 4,738,745 source facts: | Revision | Persistence | Peak memory | |---|---:|---:| | Baseline `32959f5` | 217.743 s | 5.7 GiB | | Typed-COPY candidate | 135.751 s | 5.1 GiB | This is a 37.7% persistence-time reduction and approximately 10.5% lower peak memory. A final 128 MiB buffer-pool run was excluded: LadybugDB returned `Unable to allocate memory! The buffer pool is full and no memory could be freed!` after a 1.1 GiB process peak. This confirms that the buffer-pool setting is not a total RSS limit and that undersized pools may fail publication, as documented by #39. ## Root cause - Baseline source-fact node writes use one prepared parameter statement per node family, including `SymbolObservation`, `SymbolReference`, and `Operation`: [source node writers](https://git.jacoby6000.com/JayKobe6k/ClariTree/src/commit/32959f55abed0b7bd92d94670fcf9c9abb452a1e/src/persistence/ladybug_source.rs#L275-L379). - High-cardinality ownership and symbol relationships also use prepared parameter batches: [source relationship writers](https://git.jacoby6000.com/JayKobe6k/ClariTree/src/commit/32959f55abed0b7bd92d94670fcf9c9abb452a1e/src/persistence/ladybug_source.rs#L538-L593). - Operation inputs and definition-site relationships branch per endpoint type before repeatedly invoking the parameter writer: [operation relationship writers](https://git.jacoby6000.com/JayKobe6k/ClariTree/src/commit/32959f55abed0b7bd92d94670fcf9c9abb452a1e/src/persistence/ladybug_source.rs#L1114-L1245). - The generic batching path materializes LadybugDB `LIST<STRUCT>` parameters for every configured batch: [parameter batching](https://git.jacoby6000.com/JayKobe6k/ClariTree/src/commit/32959f55abed0b7bd92d94670fcf9c9abb452a1e/src/persistence/ladybug_source.rs#L1320-L1390). The causal mechanism is repeated construction, conversion, and query execution for wide native parameter structures. Bounded typed CSV COPY removes that conversion path while preserving explicit schema targets and releasing each temporary import file after one chunk. ## Proposed implementation 1. Add closed typed COPY targets for `SymbolObservation`, `SymbolReference`, and `Operation` nodes. 2. Add explicit typed targets for the high-volume source-fact relationship families, including ownership, symbol, operation input, source-anchor, result-type, and definition-site edges. 3. Project typed iterators directly into bounded COPY chunks without collecting a second full relationship graph. 4. Preserve exact property order, null-versus-empty behavior, enum encodings, endpoint direction, checkpoints, staging rollback, read-write validation, and read-only reopen validation. 5. Keep generic parameter writers for lower-volume families where conversion work is not a measured constraint. 6. Update CLI/rustdoc and database architecture documentation so source-fact batch settings explicitly cover typed COPY chunks. Alternatives considered: - Tuning the LadybugDB buffer pool does not remove parameter conversion and can make publication fail when undersized. - Converting every low-volume table increases implementation and schema-drift surface without a measured material benefit. - Removing checkpoint or validation guarantees was rejected because durability and correctness take priority over benchmark gains. ## Expected benefit, confidence, and risks - Expected: at least 25% lower persistence time on representative medium and large C workloads, with lower peak memory; confidence high for the pinned corpus and medium across other projects. - Risks: schema column-order drift, reversed same-type endpoints, enum/null encoding changes, CSV escaping defects, and native COPY failures. - Compatibility: no graph-schema, database-format, or frontend contract change. - Rollback: revert these families to the existing parameter writers; persisted databases remain compatible. ## Acceptance criteria - [ ] The listed high-volume node and relationship families use bounded typed COPY. - [ ] Exact read-back tests cover migrated node properties, operation-input role/position nullability, relationship cardinality, and endpoint direction after read-only reopen. - [ ] Tiny configured chunks preserve complete mapping counts and append behavior. - [ ] The pinned 40-TU and 200-TU workloads preserve semantic and persisted counts. - [ ] Three comparable 40-TU runs improve median persistence by at least 25% without increasing peak memory. - [ ] The 200-TU workload confirms the improvement direction under a no-swap cgroup. - [ ] CLI help and database documentation describe the broader typed-COPY batch semantics. - [ ] Nix formatting, Clippy, all-target/all-feature tests, rustdoc, and Windows-target cross-compilation pass. Related: #38 owns bounded persistence mapping, #39 owns buffer-pool configuration, and #40 introduced typed COPY for source anchors and semantic value nodes.

~~JayKobe6k referenced this issue 2026-09-11 12:55:49 +00:00~~

Reduce index-update preflight cost with typed dependency discovery #42

JayKobe6k referenced this issue

2026-09-11 12:56:53 +00:00

Reduce index-update preflight cost with typed dependency discovery #42

JayKobe6k commented

2026-09-11 12:57:47 +00:00

Author

Owner

Implemented and pushed in b9354026f4. The pinned 40-TU workload improved from the 33.403 s baseline midpoint to a 22.293 s candidate median (33.3% lower), and the 200-TU workload improved from 217.743 s to 135.751 s (37.7% lower), with identical source-fact and semantic counts. Peak memory also decreased. Full Nix formatting, Clippy, all-target/all-feature tests, rustdoc, and Windows cross-compilation pass.

Implemented and pushed in b9354026f484072e9a7a4d786548045324ec00c0. The pinned 40-TU workload improved from the 33.403 s baseline midpoint to a 22.293 s candidate median (33.3% lower), and the 200-TU workload improved from 217.743 s to 135.751 s (37.7% lower), with identical source-fact and semantic counts. Peak memory also decreased. Full Nix formatting, Clippy, all-target/all-feature tests, rustdoc, and Windows cross-compilation pass.

JayKobe6k closed this issue

2026-09-11 12:57:47 +00:00