Bound indexing peak memory by streaming typed persistence mapping #38

New issue

Closed

opened 2026-09-10 03:38:33 +00:00 by JayKobe6k · 5 comments

JayKobe6k commented 2026-09-10 03:38:33 +00:00

Owner

Copy link

Problem and impact

Index publication currently materializes a complete persistence MetadataGraph while retaining the complete application-owned ProjectIndex. On a bounded 200-translation-unit workload, this nearly doubled process peak RSS before any LadybugDB write occurred. This amplification increases OOM and swap-pressure risk for large projects and prevents safely validating the full 1,103-TU melee workload on a busy workstation.

Environment and workload

• ClariTree source baseline: eccc45a8aed2fd9ebe6334b4975d83cdb0751def plus analyzer-only performance changes from #37; persistence mapping itself is unchanged
• Host: Linux 6.18.46-1-lts, x86_64
• Rust/Cargo: 1.93.0
• libclang/Clang: 21.1.8
• Corpus: melee-event-match at 6b6f8a8a987c2e871b8b96baa863aaa99e406f33
• Workload: first 200 deterministic compilation-plan entries
• Release-mode ignored harness: tests/profile_lineage.rs, with optional CLARITREE_PROFILE_METADATA_MAPPING=1
• Safety envelope: transient cgroup with at most 3 GB RAM, 512 MB swap, and a 240-second timeout
• The mapping measurement calls MetadataGraph::from_project_index but does not open or write a database

Baseline measurements

Same candidate binary and corpus, sequential runs:

Work performed	Parse	Analysis	Mapping	Peak RSS
Parse + lineage	29.825 s	28.155 s	n/a	1,140,592 KiB
Parse + lineage + metadata mapping	29.646 s	28.570 s	4.563 s	2,136,228 KiB

Metadata mapping added 995,636 KiB to peak RSS, an 87.3% increase over the parse-and-lineage peak. Both runs produced exactly 169,045 definitions, 180,182 uses, and 400,939 relationships; mapped definitions also totaled 169,045. Global host swap remained below 30 GB of 110.71 GB total and decreased during the experiment.

Root cause

• ProjectIndex owns every indexed compilation unit and the complete analyzed lineage.
• MetadataGraph::from_project_index maps all compilation metadata, source facts, value nodes, and relationships into additional owned vectors.
• publish_index_with_directory_sync builds that complete graph from a borrowed ProjectIndex at line 517, then retains both while opening staging storage and writing at line 536.
• write_metadata writes in bounded database batches, but those batches do not bound the already-materialized graph.

The causal mechanism is duplicated ownership of large source and lineage representations, not LadybugDB cache allocation: the measured peak occurs when only in-memory mapping is enabled and no database is opened.

Proposed implementation

Preserve ProjectIndex as the application-domain input and move bounded record generation behind the LadybugDB adapter:

1. Build only the compact global mapping context needed for stable typed IDs, such as translation-unit IDs and deduplicated source-path IDs.
2. Open the isolated staging database before materializing bulk records. Atomic publication semantics remain unchanged because failures still discard the staging directory.
3. Map and write each typed node or relationship family in bounded chunks, dropping each chunk after LadybugDB accepts it. Source facts should be processed per translation unit where endpoint ordering allows it; lineage definitions, uses, and relationships should use bounded typed iterators or chunk builders.
4. Accumulate typed counts during streaming and validate persisted counts after writing and read-only reopen. Preserve exact endpoint and source-fact validation without retaining the full graph.
5. Keep raw Ladybug values, Cypher, and CSV serialization inside the adapter. Do not introduce untyped records or expose persistence details through application interfaces.

Alternatives considered:

• Larger persistence batch settings do not help because the complete graph already exists before batching.
• Arc sharing across domain and persistence records was prototyped during #37 and did not reduce measured peak RSS; it also expanded API risk.
• Consuming ProjectIndex during mapping could reduce coexistence but would ripple through the borrowed repository port and still materialize a second complete graph. Streaming typed records offers a bounded-memory contract and retains adapter ownership.

Expected benefit, confidence, and risks

• Expected benefit: remove most of the measured 995,636 KiB mapping increment at 200 TUs and make peak mapping memory proportional to configured chunk size rather than total graph size.
• Confidence: high that complete graph materialization causes the measured increment; medium for the exact savings until streaming is implemented.
• Runtime: mapping is only 4.6 seconds in this sample, so optimize memory without materially regressing throughput.
• Risks: endpoint ordering, duplicate-ID detection, count validation, and atomic rollback can regress if validation is deferred incorrectly.
• Compatibility: no schema or public frontend contract change should be required.
• Rollback: retain the existing complete-graph writer behind the same repository port until streaming passes parity tests; reverting requires no data migration.

Acceptance criteria

• Add a bounded benchmark that measures parse, analysis, mapping/write, peak RSS, and output counts independently.
• On the same 200-TU workload, keep persistence-mapping peak RSS at or below 1,425,740 KiB (25% above the 1,140,592 KiB parse-and-lineage baseline), over at least three runs.
• Preserve exactly 169,045 definitions, 180,182 uses, and 400,939 lineage relationships for the measured sample.
• Keep transient mapped-record memory bounded by configured chunk sizes rather than total project size.
• Preserve atomic staging/publication, durability reporting, manifest behavior, duplicate detection, source-fact validation, read-only reopen validation, and typed error context.
• Add mapping parity tests for every node and relationship family and a regression test proving multiple chunks produce the same database as one chunk.
• Do not regress bounded end-to-end publication wall time by more than 10% at the same persistence settings.
• Pass formatting, Clippy, tests, rustdoc, and the Windows-target cross-compilation check through the pinned Nix environment.
• Do not attempt the full 1,103-TU workload until a cgroup limit and rollback-safe output location are documented.

Related: #37 reduces semantic-analysis CPU time; this issue independently owns persistence-mapping peak memory.

## Problem and impact Index publication currently materializes a complete persistence `MetadataGraph` while retaining the complete application-owned `ProjectIndex`. On a bounded 200-translation-unit workload, this nearly doubled process peak RSS before any LadybugDB write occurred. This amplification increases OOM and swap-pressure risk for large projects and prevents safely validating the full 1,103-TU melee workload on a busy workstation. ## Environment and workload - ClariTree source baseline: `eccc45a8aed2fd9ebe6334b4975d83cdb0751def` plus analyzer-only performance changes from #37; persistence mapping itself is unchanged - Host: Linux 6.18.46-1-lts, x86_64 - Rust/Cargo: 1.93.0 - libclang/Clang: 21.1.8 - Corpus: `melee-event-match` at `6b6f8a8a987c2e871b8b96baa863aaa99e406f33` - Workload: first 200 deterministic compilation-plan entries - Release-mode ignored harness: `tests/profile_lineage.rs`, with optional `CLARITREE_PROFILE_METADATA_MAPPING=1` - Safety envelope: transient cgroup with at most 3 GB RAM, 512 MB swap, and a 240-second timeout - The mapping measurement calls `MetadataGraph::from_project_index` but does not open or write a database ## Baseline measurements Same candidate binary and corpus, sequential runs: | Work performed | Parse | Analysis | Mapping | Peak RSS | |---|---:|---:|---:|---:| | Parse + lineage | 29.825 s | 28.155 s | n/a | 1,140,592 KiB | | Parse + lineage + metadata mapping | 29.646 s | 28.570 s | 4.563 s | 2,136,228 KiB | Metadata mapping added **995,636 KiB** to peak RSS, an **87.3% increase** over the parse-and-lineage peak. Both runs produced exactly 169,045 definitions, 180,182 uses, and 400,939 relationships; mapped definitions also totaled 169,045. Global host swap remained below 30 GB of 110.71 GB total and decreased during the experiment. ## Root cause - [`ProjectIndex`](https://git.jacoby6000.com/JayKobe6k/ClariTree/src/commit/eccc45a8aed2fd9ebe6334b4975d83cdb0751def/src/application/mod.rs#L2146-L2177) owns every indexed compilation unit and the complete analyzed lineage. - [`MetadataGraph::from_project_index`](https://git.jacoby6000.com/JayKobe6k/ClariTree/src/commit/eccc45a8aed2fd9ebe6334b4975d83cdb0751def/src/persistence/mod.rs#L824-L931) maps all compilation metadata, source facts, value nodes, and relationships into additional owned vectors. - [`publish_index_with_directory_sync`](https://git.jacoby6000.com/JayKobe6k/ClariTree/src/commit/eccc45a8aed2fd9ebe6334b4975d83cdb0751def/src/persistence/ladybug.rs#L477-L537) builds that complete graph from a borrowed `ProjectIndex` at line 517, then retains both while opening staging storage and writing at line 536. - [`write_metadata`](https://git.jacoby6000/JayKobe6k/ClariTree/src/commit/eccc45a8aed2fd9ebe6334b4975d83cdb0751def/src/persistence/ladybug.rs#L308-L333) writes in bounded database batches, but those batches do not bound the already-materialized graph. The causal mechanism is duplicated ownership of large source and lineage representations, not LadybugDB cache allocation: the measured peak occurs when only in-memory mapping is enabled and no database is opened. ## Proposed implementation Preserve `ProjectIndex` as the application-domain input and move bounded record generation behind the LadybugDB adapter: 1. Build only the compact global mapping context needed for stable typed IDs, such as translation-unit IDs and deduplicated source-path IDs. 2. Open the isolated staging database before materializing bulk records. Atomic publication semantics remain unchanged because failures still discard the staging directory. 3. Map and write each typed node or relationship family in bounded chunks, dropping each chunk after LadybugDB accepts it. Source facts should be processed per translation unit where endpoint ordering allows it; lineage definitions, uses, and relationships should use bounded typed iterators or chunk builders. 4. Accumulate typed counts during streaming and validate persisted counts after writing and read-only reopen. Preserve exact endpoint and source-fact validation without retaining the full graph. 5. Keep raw Ladybug values, Cypher, and CSV serialization inside the adapter. Do not introduce untyped records or expose persistence details through application interfaces. Alternatives considered: - Larger persistence batch settings do not help because the complete graph already exists before batching. - `Arc` sharing across domain and persistence records was prototyped during #37 and did not reduce measured peak RSS; it also expanded API risk. - Consuming `ProjectIndex` during mapping could reduce coexistence but would ripple through the borrowed repository port and still materialize a second complete graph. Streaming typed records offers a bounded-memory contract and retains adapter ownership. ## Expected benefit, confidence, and risks - Expected benefit: remove most of the measured 995,636 KiB mapping increment at 200 TUs and make peak mapping memory proportional to configured chunk size rather than total graph size. - Confidence: high that complete graph materialization causes the measured increment; medium for the exact savings until streaming is implemented. - Runtime: mapping is only 4.6 seconds in this sample, so optimize memory without materially regressing throughput. - Risks: endpoint ordering, duplicate-ID detection, count validation, and atomic rollback can regress if validation is deferred incorrectly. - Compatibility: no schema or public frontend contract change should be required. - Rollback: retain the existing complete-graph writer behind the same repository port until streaming passes parity tests; reverting requires no data migration. ## Acceptance criteria - [ ] Add a bounded benchmark that measures parse, analysis, mapping/write, peak RSS, and output counts independently. - [ ] On the same 200-TU workload, keep persistence-mapping peak RSS at or below 1,425,740 KiB (25% above the 1,140,592 KiB parse-and-lineage baseline), over at least three runs. - [ ] Preserve exactly 169,045 definitions, 180,182 uses, and 400,939 lineage relationships for the measured sample. - [ ] Keep transient mapped-record memory bounded by configured chunk sizes rather than total project size. - [ ] Preserve atomic staging/publication, durability reporting, manifest behavior, duplicate detection, source-fact validation, read-only reopen validation, and typed error context. - [ ] Add mapping parity tests for every node and relationship family and a regression test proving multiple chunks produce the same database as one chunk. - [ ] Do not regress bounded end-to-end publication wall time by more than 10% at the same persistence settings. - [ ] Pass formatting, Clippy, tests, rustdoc, and the Windows-target cross-compilation check through the pinned Nix environment. - [ ] Do not attempt the full 1,103-TU workload until a cgroup limit and rollback-safe output location are documented. Related: #37 reduces semantic-analysis CPU time; this issue independently owns persistence-mapping peak memory.

JayKobe6k commented 2026-09-10 03:39:02 +00:00

Author

Owner

Copy link

Correction to the write_metadata source link in the issue body: LadybugMetadataStore::write_metadata. The original URL omitted .com.

Correction to the `write_metadata` source link in the issue body: [`LadybugMetadataStore::write_metadata`](https://git.jacoby6000.com/JayKobe6k/ClariTree/src/commit/eccc45a8aed2fd9ebe6334b4975d83cdb0751def/src/persistence/ladybug.rs#L308-L333). The original URL omitted `.com`.

JayKobe6k commented 2026-09-10 17:41:05 +00:00

Author

Owner

Copy link

Full-corpus memory evidence after analyzer CPU remediation

The analyzer changes tracked by #37 made it possible to complete the full 1,103-translation-unit in-memory workload under a bounded cgroup. Persistence mapping itself is unchanged.

Environment and procedure

• ClariTree base: eccc45a8aed2fd9ebe6334b4975d83cdb0751def plus the uncommitted analyzer candidate
• Corpus: melee-event-match 6b6f8a8a987c2e871b8b96baa863aaa99e406f33
• Exact workload: all 1,103 compilation-plan entries
• Harness: tests/profile_lineage.rs with CLARITREE_PROFILE_METADATA_MAPPING=1
• No LadybugDB was opened or written
• Cgroup: MemoryHigh=24G, MemoryMax=32G, MemorySwapMax=8G; command timeout 1,700 s

Result

parse_ms=334796
analysis_ms=419326
mapping_ms=46784
total_elapsed_s=806.04
definitions=1144989
uses=1370981
relationships=3597637
mapped_definitions=1144989
max_rss_kb=15315736
cgroup_memory_peak=14.5 GiB

No cgroup swap was observed during polling. Global swap remained below half of host capacity and decreased over the experiment.

This run does not isolate the full-corpus mapping increment because /usr/bin/time reports one process-wide maximum, but it establishes the operational scale of the current complete-graph design: ProjectIndex and a MetadataGraph containing 1.145 million definitions, 1.371 million uses, and 3.598 million relationships coexist, producing a 15,315,736 KiB process peak. The earlier paired 200-TU experiment remains the controlled attribution evidence: mapping added 995,636 KiB (+87.3%) before any database open.

Implication

The full workload now fits the approved 32 GiB envelope, but a 14.5 GiB peak for in-memory mapping alone remains too large for routine indexing and leaves database staging/write memory unmeasured. Streaming typed persistence remains justified; capacity expansion is not a substitute. The next validation should compare complete publication before/after streaming with bounded chunks, preserving the existing 1,103-TU counts and atomic rollback behavior.

The profiler now emits parse and analysis completion records before mapping, so a future timeout can be localized without another blind run. This issue should remain open.

## Full-corpus memory evidence after analyzer CPU remediation The analyzer changes tracked by #37 made it possible to complete the full 1,103-translation-unit in-memory workload under a bounded cgroup. Persistence mapping itself is unchanged. ### Environment and procedure - ClariTree base: `eccc45a8aed2fd9ebe6334b4975d83cdb0751def` plus the uncommitted analyzer candidate - Corpus: `melee-event-match` `6b6f8a8a987c2e871b8b96baa863aaa99e406f33` - Exact workload: all 1,103 compilation-plan entries - Harness: `tests/profile_lineage.rs` with `CLARITREE_PROFILE_METADATA_MAPPING=1` - No LadybugDB was opened or written - Cgroup: `MemoryHigh=24G`, `MemoryMax=32G`, `MemorySwapMax=8G`; command timeout 1,700 s ### Result ```text parse_ms=334796 analysis_ms=419326 mapping_ms=46784 total_elapsed_s=806.04 definitions=1144989 uses=1370981 relationships=3597637 mapped_definitions=1144989 max_rss_kb=15315736 cgroup_memory_peak=14.5 GiB ``` No cgroup swap was observed during polling. Global swap remained below half of host capacity and decreased over the experiment. This run does not isolate the full-corpus mapping increment because `/usr/bin/time` reports one process-wide maximum, but it establishes the operational scale of the current complete-graph design: `ProjectIndex` and a `MetadataGraph` containing 1.145 million definitions, 1.371 million uses, and 3.598 million relationships coexist, producing a 15,315,736 KiB process peak. The earlier paired 200-TU experiment remains the controlled attribution evidence: mapping added 995,636 KiB (+87.3%) before any database open. ### Implication The full workload now fits the approved 32 GiB envelope, but a 14.5 GiB peak for in-memory mapping alone remains too large for routine indexing and leaves database staging/write memory unmeasured. Streaming typed persistence remains justified; capacity expansion is not a substitute. The next validation should compare complete publication before/after streaming with bounded chunks, preserving the existing 1,103-TU counts and atomic rollback behavior. The profiler now emits parse and analysis completion records before mapping, so a future timeout can be localized without another blind run. This issue should remain open.

JayKobe6k commented 2026-09-10 19:57:43 +00:00

Author

Owner

Copy link

Candidate implementation and validation update (uncommitted working tree based on 912603680d):

Implementation:

• Production LadybugIndexRepository publication no longer calls MetadataGraph::from_project_index.
• RevisionMetadataGraph retains root/source metadata once; LineageMappingSession maps and writes value definitions, uses, declaration/ownership edges, and all six lineage families in configured bounded chunks.
• MetadataGraph::from_project_index remains the public full-graph compatibility/parity oracle.
• Expected MetadataCounts are accumulated by typed family and reused for writable and read-only reopen validation.
• Exact declaration and lineage read-back validation now uses constant-space, domain-separated SHA-512 multiset fingerprints covering endpoints, certainty, and uncertainty reason. Same-count endpoint and certainty corruption tests fail as expected.
• ValueHandle canonical text now uses Arc, eliminating repeated canonical-string allocations when handles are cloned into typed records.

200-TU mapping results:

• Existing full-graph baseline: 2,136,228 KiB peak RSS versus 1,140,592 KiB parse+analysis, a 995,636 KiB mapping increment.
• New production revision/source mapping, measured before opening LadybugDB via the profile harness post-mapping filesystem failpoint, had peak RSS of 1,236,764 / 1,243,324 / 1,240,200 KiB across three runs (median 1,240,200 KiB).
• The median is 185,540 KiB below the 1,425,740 KiB acceptance ceiling and 896,028 KiB (41.9%) below the old full-graph peak. Against the current 1,192,580 KiB parse+analysis run, retained revision mapping added 47,620 KiB (4.0%).
• The retained public full-graph oracle also fell to a 1,839,900 KiB median from targeted handle sharing, a 303,748 KiB (14.2%) total-process reduction.
• Every run preserved 169,045 definitions, 180,182 uses, and 400,939 lineage relationships.

Comparable end-to-end publication, 12 GiB MemoryMax, 11 GiB MemoryHigh, MemorySwapMax=0:

• Legacy 9126036 writer: persistence 375,207 ms; 8,001,724 KiB peak RSS.
• Streaming candidate: persistence 318,381 ms; 7,597,436 KiB peak RSS.
• Result: 15.1% faster and 5.1% lower end-to-end process peak despite independently reading and fingerprinting all eight relationship families twice.
• Both persisted exactly 169,045 definitions, 180,182 uses, 400,939 lineage relationships, and 4,738,745 source facts.

Validation completed through Nix:

• cargo fmt --all -- --check
• cargo lint
• cargo test --all-targets --all-features
• cargo doc --no-deps --all-features
• cargo check --target x86_64-pc-windows-gnu --all-features
• git diff --check

Tests include one-row chunk parity against the full-graph writer, repeated COPY calls, all relationship families, exact fingerprint parity, same-count corruption rejection in writable and read-only modes, source-fact validation, and staging rollback.

Remaining bounded state is the complete SourceFactGraph plus compact lineage endpoint/type/projection indexes. Transient typed lineage database records and relationship CSVs are bounded by configured batch sizes. End-to-end peak is now dominated by the configured 4 GiB LadybugDB buffer pool and persisted source-fact workload rather than full MetadataGraph coexistence.

Candidate implementation and validation update (uncommitted working tree based on 912603680d78db19b00c0df2e8a62038193afea0): Implementation: - Production LadybugIndexRepository publication no longer calls MetadataGraph::from_project_index. - RevisionMetadataGraph retains root/source metadata once; LineageMappingSession maps and writes value definitions, uses, declaration/ownership edges, and all six lineage families in configured bounded chunks. - MetadataGraph::from_project_index remains the public full-graph compatibility/parity oracle. - Expected MetadataCounts are accumulated by typed family and reused for writable and read-only reopen validation. - Exact declaration and lineage read-back validation now uses constant-space, domain-separated SHA-512 multiset fingerprints covering endpoints, certainty, and uncertainty reason. Same-count endpoint and certainty corruption tests fail as expected. - ValueHandle canonical text now uses Arc<str>, eliminating repeated canonical-string allocations when handles are cloned into typed records. 200-TU mapping results: - Existing full-graph baseline: 2,136,228 KiB peak RSS versus 1,140,592 KiB parse+analysis, a 995,636 KiB mapping increment. - New production revision/source mapping, measured before opening LadybugDB via the profile harness post-mapping filesystem failpoint, had peak RSS of 1,236,764 / 1,243,324 / 1,240,200 KiB across three runs (median 1,240,200 KiB). - The median is 185,540 KiB below the 1,425,740 KiB acceptance ceiling and 896,028 KiB (41.9%) below the old full-graph peak. Against the current 1,192,580 KiB parse+analysis run, retained revision mapping added 47,620 KiB (4.0%). - The retained public full-graph oracle also fell to a 1,839,900 KiB median from targeted handle sharing, a 303,748 KiB (14.2%) total-process reduction. - Every run preserved 169,045 definitions, 180,182 uses, and 400,939 lineage relationships. Comparable end-to-end publication, 12 GiB MemoryMax, 11 GiB MemoryHigh, MemorySwapMax=0: - Legacy 9126036 writer: persistence 375,207 ms; 8,001,724 KiB peak RSS. - Streaming candidate: persistence 318,381 ms; 7,597,436 KiB peak RSS. - Result: 15.1% faster and 5.1% lower end-to-end process peak despite independently reading and fingerprinting all eight relationship families twice. - Both persisted exactly 169,045 definitions, 180,182 uses, 400,939 lineage relationships, and 4,738,745 source facts. Validation completed through Nix: - cargo fmt --all -- --check - cargo lint - cargo test --all-targets --all-features - cargo doc --no-deps --all-features - cargo check --target x86_64-pc-windows-gnu --all-features - git diff --check Tests include one-row chunk parity against the full-graph writer, repeated COPY calls, all relationship families, exact fingerprint parity, same-count corruption rejection in writable and read-only modes, source-fact validation, and staging rollback. Remaining bounded state is the complete SourceFactGraph plus compact lineage endpoint/type/projection indexes. Transient typed lineage database records and relationship CSVs are bounded by configured batch sizes. End-to-end peak is now dominated by the configured 4 GiB LadybugDB buffer pool and persisted source-fact workload rather than full MetadataGraph coexistence.

JayKobe6k referenced this issue 2026-09-10 21:53:43 +00:00

Allow per-publication LadybugDB buffer-pool tuning #39

JayKobe6k commented 2026-09-10 22:08:23 +00:00

Author

Owner

Copy link

Implementation candidate status (uncommitted)

The production publication path no longer calls MetadataGraph::from_project_index. It maps revision and source metadata once, then maps and writes value definitions, uses, declaration/ownership relationships, and all six lineage families in configured typed chunks. Expected counts and order-independent, domain-separated SHA-512 multiset fingerprints are accumulated while mapping and independently validated in writable staging and after read-only reopen. Counts remain a separate cardinality check. Tiny-chunk parity, same-count endpoint/certainty corruption, rollback, publication, and complete source-fact tests pass.

Measured first-200-TU results on the pinned melee-event-match corpus:

• Legacy full in-memory mapping peak: 2,136,228 KiB.
• Production revision/source mapping peaks: 1,236,764, 1,243,324, and 1,240,200 KiB; median 1,240,200 KiB, 41.9% below legacy and below the 1,425,740 KiB acceptance threshold.
• Comparable end-to-end publication improved from 375.207 s and 8,001,724 KiB to 318.381 s and 7,597,436 KiB: 15.1% faster and 5.1% lower peak RSS.
• Output remained exactly 169,045 definitions, 180,182 uses, 400,939 lineage relationships, and 4,738,745 source facts.

A review identified that RevisionMetadataGraph still owns the project-sized SourceFactGraph and source identity indexes. This is intentional current scope: the measured persistence duplication has been removed, transient lineage records and COPY files are chunk-bounded, and experiments that skipped source-ID validation did not materially change peak RSS. Further source-fact redesign is not justified without a profile locating material retained cost.

The complete required Nix validation passes: format, Clippy, 273 unit tests plus all integration suites, rustdoc, Windows GNU cross-check, and git diff --check. A compatibility review also removed the new public LadybugAdapterError enum variant; fingerprint mismatches now use the existing typed UnexpectedRow/ValidateMetadata surface.

Implementation candidate status (uncommitted) The production publication path no longer calls MetadataGraph::from_project_index. It maps revision and source metadata once, then maps and writes value definitions, uses, declaration/ownership relationships, and all six lineage families in configured typed chunks. Expected counts and order-independent, domain-separated SHA-512 multiset fingerprints are accumulated while mapping and independently validated in writable staging and after read-only reopen. Counts remain a separate cardinality check. Tiny-chunk parity, same-count endpoint/certainty corruption, rollback, publication, and complete source-fact tests pass. Measured first-200-TU results on the pinned melee-event-match corpus: - Legacy full in-memory mapping peak: 2,136,228 KiB. - Production revision/source mapping peaks: 1,236,764, 1,243,324, and 1,240,200 KiB; median 1,240,200 KiB, 41.9% below legacy and below the 1,425,740 KiB acceptance threshold. - Comparable end-to-end publication improved from 375.207 s and 8,001,724 KiB to 318.381 s and 7,597,436 KiB: 15.1% faster and 5.1% lower peak RSS. - Output remained exactly 169,045 definitions, 180,182 uses, 400,939 lineage relationships, and 4,738,745 source facts. A review identified that RevisionMetadataGraph still owns the project-sized SourceFactGraph and source identity indexes. This is intentional current scope: the measured persistence duplication has been removed, transient lineage records and COPY files are chunk-bounded, and experiments that skipped source-ID validation did not materially change peak RSS. Further source-fact redesign is not justified without a profile locating material retained cost. The complete required Nix validation passes: format, Clippy, 273 unit tests plus all integration suites, rustdoc, Windows GNU cross-check, and git diff --check. A compatibility review also removed the new public LadybugAdapterError enum variant; fingerprint mismatches now use the existing typed UnexpectedRow/ValidateMetadata surface.

JayKobe6k referenced this issue 2026-09-11 00:44:28 +00:00

Reduce LadybugDB publication time and memory with typed node COPY #40

JayKobe6k commented 2026-09-11 01:03:16 +00:00

Author

Owner

Copy link

Implemented and pushed to main as 654b2fbcf8e3ccc7ce7428b637d8b2736f67de19. The follow-up typed node COPY optimization is delivered as 32959f55abed0b7bd92d94670fcf9c9abb452a1e and tracked by #40.

Streaming mapping reduced the bounded 200-TU mapping peak from 2,136,228 KiB to a 1,240,200 KiB median while preserving 169,045 definitions, 180,182 uses, and 400,939 relationships. Publication, rollback, fingerprints, read-only reopen validation, tests, documentation, and Windows cross-compilation all passed through Nix.

Implemented and pushed to `main` as [`654b2fbcf8e3ccc7ce7428b637d8b2736f67de19`](https://git.jacoby6000.com/JayKobe6k/ClariTree/commit/654b2fbcf8e3ccc7ce7428b637d8b2736f67de19). The follow-up typed node COPY optimization is delivered as [`32959f55abed0b7bd92d94670fcf9c9abb452a1e`](https://git.jacoby6000.com/JayKobe6k/ClariTree/commit/32959f55abed0b7bd92d94670fcf9c9abb452a1e) and tracked by #40. Streaming mapping reduced the bounded 200-TU mapping peak from 2,136,228 KiB to a 1,240,200 KiB median while preserving 169,045 definitions, 180,182 uses, and 400,939 relationships. Publication, rollback, fingerprints, read-only reopen validation, tests, documentation, and Windows cross-compilation all passed through Nix.

JayKobe6k closed this issue 2026-09-11 01:03:16 +00:00

JayKobe6k referenced this issue 2026-09-11 02:25:27 +00:00

Reduce high-volume source-fact publication with typed COPY #41

JayKobe6k referenced this issue 2026-09-12 18:18:23 +00:00

Bound full-project indexing memory below the corpus working set #53

JayKobe6k referenced this issue 2026-09-12 18:19:07 +00:00

Bound full-project indexing memory below the corpus working set #53

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

issue-18-cli-workflows

issue-16-search-struct

issue-17-search-enum

issue-14-search-value

issue-15-search-function

issue-9-index-update

issue-12-lineage-query

issue-8-index-new

issue-13-shared-search-pipeline

issue-11-lineage-analysis

issue-10-durable-value-handles

issue-7-c-facts

issue-6-project-sources

issue-5-ladybug-adapter

issue-4-index-catalog

issue-3-cli-contract

issue-2-application-services

No results found.

Labels

Clear labels   

area/cli

Command-line adapter and output

  

area/core

Backend-independent domain and application logic

  

area/database

LadybugDB schema and adapter

  

area/indexing

Project indexing and update lifecycle

  

area/query

Search and lineage query behavior

  

type/epic

Tracks a coordinated body of work

  

type/feature

User-facing capability

  

type/foundation

Foundational architecture or infrastructure

No labels

area/cli

area/core

area/database

area/indexing

area/query

type/epic

type/feature

type/foundation

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

JayKobe6k/ClariTree#38

No description provided.