Bound full-project indexing memory below the corpus working set #53

Open
opened 2026-09-12 18:18:23 +00:00 by JayKobe6k · 2 comments
Owner

Problem and impact

The current production-path profiler cannot complete indexing of the pinned 1,103-translation-unit melee-event-match corpus inside the reported 32 GiB physical-memory cgroup with cgroup swap disabled. Parsing completes, but the process is OOM-killed after semantic analysis starts and before it reports completion; persistence therefore never starts. This prevents indexing a representative full project even though bounded 200-TU runs succeed. No successful current full end-to-end duration exists.

Environment and reproduction

  • ClariTree: 5d04beabeb4bcd00d27e69dd318e5834b097cf30
  • Corpus: melee-event-match commit 6b6f8a8a987c2e871b8b96baa863aaa99e406f33
  • compile_commands.json SHA-256: 0c025066b2f5ed7940a668d4fc97b912d8eb5d467fb7bb33ba0e72ab7d50ea16
  • Linux/Nix release build
  • Reported launch envelope: four logical CPUs (AllowedCPUs=0-3, CPUQuota=400%, taskset -c 0-3), MemoryMax=32G, MemorySwapMax=0, and RuntimeMaxSec=7200
  • Production persistence requested with the compatible 4,096 MiB LadybugDB buffer pool

The sanitized retained log independently verifies the parser output, OOM result, service runtime, CPU time, 32 GiB cgroup peak, and zero cgroup swap. It does not retain the environment variables or transient-unit properties, so the launch envelope and persistence request above come from the reproduction record rather than the log itself.

Build the ignored release profiler through Nix, then run all 1,103 units with:

CLARITREE_PROFILE_PROJECT=/path/to/melee-event-match
CLARITREE_PROFILE_LIMIT=1103
CLARITREE_PROFILE_PERSISTENCE=1
CLARITREE_PROFILE_BUFFER_POOL_MIB=4096

Run the profiler in a transient user service carrying the CPU, memory, swap, and runtime properties above. Do not use ulimit -v: LadybugDB reserves a large sparse virtual mapping, so a virtual-address limit produces an unrelated database-open failure.

Baseline and profiling evidence

The full run reported:

profile_phase=parse units=1103 elapsed_ms=1372076
profile_facts ...
Finished with result: oom-kill
Service runtime: 23min 27.735s
CPU time consumed: 21min 13.741s
Memory peak: 32G (swap: 0B)

The sum of the 21 emitted parser cardinalities is 80,835,220, including 17,047,327 declarations, 16,828,426 symbols, 9,973,010 references, 6,220,895 aggregate fields, 5,525,983 operation inputs, and 4,435,322 operations. Subtracting the 1,372.076-second parse marker from the 1,407.735-second whole-service runtime gives 35.659 seconds after that marker. That is a derived upper bound that includes post-parse test overhead, not a direct analyzer timestamp. The missing analysis-completion marker, OOM result, and source execution order localize the kill to semantic analysis before LadybugDB publication, but not to a specific analyzer allocation.

The sanitized raw full-run log is retained locally as target/full-melee-4cpu-current.log, SHA-256 aedc41d38b8999e04a36928d54c1ece7cd672419359e595d19abba742c07218a. Local ignored artifacts are not durable; the measurements above are included here in full.

For comparison, the delivered first-200 profile documented in closed issue #52 completed with 3.9 GiB peak and zero cgroup swap, producing a sum of 6,823,854 across the same parser cardinalities, 377,067 definitions, 412,330 uses, and 801,507 relationships. The remaining 903 units account for 74,011,366 reported facts and average 81,961.646 per TU versus 34,119.270 for the first 200, a 2.402x density ratio.

Related prior work:

  • #37 reduced semantic-analysis CPU amplification but explicitly did not bound the complete working set.
  • #38 bounded lineage persistence mapping, while retaining the complete ProjectIndex and source-fact graph.
  • #54 separately tracks bounded process-level translation-unit parallelism. Establish memory headroom here before enabling multiple workers by default because concurrent native ASTs can increase the working set.

Root cause

The application retains every complete owned translation unit before starting analysis:

  • index_compilation_plan appends all IndexedCompilationUnit values to one Vec.
  • TranslationUnit owns all source, symbol, declaration, type, reference, operation, CFG, and preprocessing vectors. Shared headers are represented independently for each compilation context.
  • Analyzer::new eagerly constructs a UnitAnalysisIndex for every retained unit. UnitAnalysisIndex::new duplicates unit collections into maps and sets before lineage output is produced.
  • Complete definitions, uses, relationships, block reaching states, and validation identifier sets add further project-sized working sets during analysis.
  • If analysis succeeds, ProjectIndex retains both all units and complete lineage. Production persistence then maps a project-sized SourceFactGraph before stage_revision opens staging storage.

The log and source order localize the OOM to semantic analysis, but the precise contribution of analyzer indexes, lineage identities, reaching states, and allocator/native retention has not yet been separated with phase-level memory counters.

Proposed implementation

  1. Add manual-profiler RSS/cgroup checkpoints after parsing, Analyzer::new, definition/use construction, call binding, control-flow analysis, aggregate validation, source-fact mapping, and database open. Use prefix runs to quantify each layer before changing representation.
  2. Split semantic-analysis data into compact project-wide summaries and unit-local lookup indexes. Retain globally only facts required for cross-unit ownership, call effects, seeds, and targets; construct and release operation/declaration/reference/CFG indexes one unit at a time where dependencies permit.
  3. Shorten large temporary lifetimes, including the first reaching solution and analysis-only maps before aggregate validation, where measurement shows meaningful overlap.
  4. Continue the streaming boundary through source-fact mapping and lineage endpoint lookup so persistence does not require another complete project-shaped graph.
  5. If exact analysis dependencies prevent one-pass unit processing, spill typed intermediate facts behind a ClariTree-owned repository interface rather than introducing LadybugDB or Cypher into core analysis.

Keep symbols distinct from value definitions/uses, preserve producer-to-consumer relationship direction, and retain exact spelling/expansion anchors and occurrence-key hashing.

Alternatives considered

  • Raising MemoryMax: operational workaround only; it leaves working-set growth unbounded.
  • Reducing the LadybugDB buffer pool: cannot fix this failure because persistence never starts, and undersized pools can fail publication.
  • Deduplicating complete shared-header facts: unsafe across different compiler arguments, macros, and source contexts.
  • Parallel parsing first: can multiply active native AST memory and should not precede a bounded working set.
  • Low-level allocation tuning: not justified until phase-level measurements identify its maximum contribution.

Expected benefit, confidence, and risks

  • Benefit: enable full-corpus indexing within bounded physical memory and create headroom for later concurrency.
  • Confidence: high that complete-unit plus eager analyzer-index coexistence is a material architectural constraint; medium that it is the dominant allocation and medium on the best decomposition until phase-level checkpoints quantify each layer.
  • Risks: changing analysis order can alter stable IDs, first-owner selection, diagnostic order, relationship ordering, certainty, or source anchors; spill/stream failure handling must preserve atomic publication.

Acceptance criteria

  • The pinned 1,103-TU workload completes parsing, semantic analysis, staging, validation, checkpoint, read-only reopen, and publication under four logical CPUs, MemoryMax=32G, and MemorySwapMax=0; report phase and full end-to-end durations rather than extrapolating from the current failed run.
  • Peak cgroup memory is at most 28 GiB, leaving at least 12.5% headroom below the enforced limit.
  • The existing first-200 oracle remains deeply equal for complete owned IndexedTranslationUnit outputs, including diagnostics, dependencies, source anchors, identities, and order, and for semantic definitions, uses, relationships, certainty, uncertainty reasons, endpoint fingerprints, and persisted source-fact families; count-only parity is insufficient.
  • All 21 full-corpus emitted parser cardinalities remain exactly the values in the retained hashed log. Full-corpus semantic and persistence counts/fingerprints are recorded as a new durable baseline because current HEAD cannot complete those phases.
  • Failure injection proves staging cleanup and no partial publication when parsing, analysis, mapping, write, checkpoint, or reopen validation fails.
  • Linux full-runtime validation, Windows cross-compilation, formatting, lint, tests, and docs pass through the pinned Nix environment.

Rollback

Keep the existing eager analyzer path as a temporary test oracle while implementing the bounded path. If exact-output parity, failure atomicity, or the 32 GiB target fails, retain the measurements, remove the candidate path, and leave current production behavior unchanged.

## Problem and impact The current production-path profiler cannot complete indexing of the pinned 1,103-translation-unit `melee-event-match` corpus inside the reported 32 GiB physical-memory cgroup with cgroup swap disabled. Parsing completes, but the process is OOM-killed after semantic analysis starts and before it reports completion; persistence therefore never starts. This prevents indexing a representative full project even though bounded 200-TU runs succeed. No successful current full end-to-end duration exists. ## Environment and reproduction - ClariTree: [`5d04beabeb4bcd00d27e69dd318e5834b097cf30`](https://git.jacoby6000.com/JayKobe6k/ClariTree/commit/5d04beabeb4bcd00d27e69dd318e5834b097cf30) - Corpus: `melee-event-match` commit `6b6f8a8a987c2e871b8b96baa863aaa99e406f33` - `compile_commands.json` SHA-256: `0c025066b2f5ed7940a668d4fc97b912d8eb5d467fb7bb33ba0e72ab7d50ea16` - Linux/Nix release build - Reported launch envelope: four logical CPUs (`AllowedCPUs=0-3`, `CPUQuota=400%`, `taskset -c 0-3`), `MemoryMax=32G`, `MemorySwapMax=0`, and `RuntimeMaxSec=7200` - Production persistence requested with the compatible 4,096 MiB LadybugDB buffer pool The sanitized retained log independently verifies the parser output, OOM result, service runtime, CPU time, 32 GiB cgroup peak, and zero cgroup swap. It does not retain the environment variables or transient-unit properties, so the launch envelope and persistence request above come from the reproduction record rather than the log itself. Build the ignored release profiler through Nix, then run all 1,103 units with: ```text CLARITREE_PROFILE_PROJECT=/path/to/melee-event-match CLARITREE_PROFILE_LIMIT=1103 CLARITREE_PROFILE_PERSISTENCE=1 CLARITREE_PROFILE_BUFFER_POOL_MIB=4096 ``` Run the profiler in a transient user service carrying the CPU, memory, swap, and runtime properties above. Do not use `ulimit -v`: LadybugDB reserves a large sparse virtual mapping, so a virtual-address limit produces an unrelated database-open failure. ## Baseline and profiling evidence The full run reported: ```text profile_phase=parse units=1103 elapsed_ms=1372076 profile_facts ... Finished with result: oom-kill Service runtime: 23min 27.735s CPU time consumed: 21min 13.741s Memory peak: 32G (swap: 0B) ``` The sum of the 21 emitted parser cardinalities is 80,835,220, including 17,047,327 declarations, 16,828,426 symbols, 9,973,010 references, 6,220,895 aggregate fields, 5,525,983 operation inputs, and 4,435,322 operations. Subtracting the 1,372.076-second parse marker from the 1,407.735-second whole-service runtime gives 35.659 seconds after that marker. That is a derived upper bound that includes post-parse test overhead, not a direct analyzer timestamp. The missing analysis-completion marker, OOM result, and source execution order localize the kill to semantic analysis before LadybugDB publication, but not to a specific analyzer allocation. The sanitized raw full-run log is retained locally as `target/full-melee-4cpu-current.log`, SHA-256 `aedc41d38b8999e04a36928d54c1ece7cd672419359e595d19abba742c07218a`. Local ignored artifacts are not durable; the measurements above are included here in full. For comparison, the delivered first-200 profile documented in closed [issue #52](https://git.jacoby6000.com/JayKobe6k/ClariTree/issues/52) completed with 3.9 GiB peak and zero cgroup swap, producing a sum of 6,823,854 across the same parser cardinalities, 377,067 definitions, 412,330 uses, and 801,507 relationships. The remaining 903 units account for 74,011,366 reported facts and average 81,961.646 per TU versus 34,119.270 for the first 200, a 2.402x density ratio. Related prior work: - [#37](https://git.jacoby6000.com/JayKobe6k/ClariTree/issues/37) reduced semantic-analysis CPU amplification but explicitly did not bound the complete working set. - [#38](https://git.jacoby6000.com/JayKobe6k/ClariTree/issues/38) bounded lineage persistence mapping, while retaining the complete `ProjectIndex` and source-fact graph. - [#54](https://git.jacoby6000.com/JayKobe6k/ClariTree/issues/54) separately tracks bounded process-level translation-unit parallelism. Establish memory headroom here before enabling multiple workers by default because concurrent native ASTs can increase the working set. ## Root cause The application retains every complete owned translation unit before starting analysis: - [`index_compilation_plan`](https://git.jacoby6000.com/JayKobe6k/ClariTree/src/commit/5d04beabeb4bcd00d27e69dd318e5834b097cf30/src/application/mod.rs#L2626-L2644) appends all `IndexedCompilationUnit` values to one `Vec`. - [`TranslationUnit`](https://git.jacoby6000.com/JayKobe6k/ClariTree/src/commit/5d04beabeb4bcd00d27e69dd318e5834b097cf30/src/ir/mod.rs#L2065-L2117) owns all source, symbol, declaration, type, reference, operation, CFG, and preprocessing vectors. Shared headers are represented independently for each compilation context. - [`Analyzer::new`](https://git.jacoby6000.com/JayKobe6k/ClariTree/src/commit/5d04beabeb4bcd00d27e69dd318e5834b097cf30/src/lineage/analyzer.rs#L574-L600) eagerly constructs a `UnitAnalysisIndex` for every retained unit. [`UnitAnalysisIndex::new`](https://git.jacoby6000.com/JayKobe6k/ClariTree/src/commit/5d04beabeb4bcd00d27e69dd318e5834b097cf30/src/lineage/analyzer.rs#L318-L415) duplicates unit collections into maps and sets before lineage output is produced. - Complete definitions, uses, relationships, block reaching states, and validation identifier sets add further project-sized working sets during analysis. - If analysis succeeds, [`ProjectIndex`](https://git.jacoby6000.com/JayKobe6k/ClariTree/src/commit/5d04beabeb4bcd00d27e69dd318e5834b097cf30/src/application/mod.rs#L2204-L2235) retains both all units and complete lineage. Production persistence then maps a project-sized [`SourceFactGraph`](https://git.jacoby6000.com/JayKobe6k/ClariTree/src/commit/5d04beabeb4bcd00d27e69dd318e5834b097cf30/src/persistence/mod.rs#L829-L895) before [`stage_revision`](https://git.jacoby6000.com/JayKobe6k/ClariTree/src/commit/5d04beabeb4bcd00d27e69dd318e5834b097cf30/src/persistence/ladybug.rs#L825-L835) opens staging storage. The log and source order localize the OOM to semantic analysis, but the precise contribution of analyzer indexes, lineage identities, reaching states, and allocator/native retention has not yet been separated with phase-level memory counters. ## Proposed implementation 1. Add manual-profiler RSS/cgroup checkpoints after parsing, `Analyzer::new`, definition/use construction, call binding, control-flow analysis, aggregate validation, source-fact mapping, and database open. Use prefix runs to quantify each layer before changing representation. 2. Split semantic-analysis data into compact project-wide summaries and unit-local lookup indexes. Retain globally only facts required for cross-unit ownership, call effects, seeds, and targets; construct and release operation/declaration/reference/CFG indexes one unit at a time where dependencies permit. 3. Shorten large temporary lifetimes, including the first reaching solution and analysis-only maps before aggregate validation, where measurement shows meaningful overlap. 4. Continue the streaming boundary through source-fact mapping and lineage endpoint lookup so persistence does not require another complete project-shaped graph. 5. If exact analysis dependencies prevent one-pass unit processing, spill typed intermediate facts behind a ClariTree-owned repository interface rather than introducing LadybugDB or Cypher into core analysis. Keep symbols distinct from value definitions/uses, preserve producer-to-consumer relationship direction, and retain exact spelling/expansion anchors and occurrence-key hashing. ## Alternatives considered - Raising `MemoryMax`: operational workaround only; it leaves working-set growth unbounded. - Reducing the LadybugDB buffer pool: cannot fix this failure because persistence never starts, and undersized pools can fail publication. - Deduplicating complete shared-header facts: unsafe across different compiler arguments, macros, and source contexts. - Parallel parsing first: can multiply active native AST memory and should not precede a bounded working set. - Low-level allocation tuning: not justified until phase-level measurements identify its maximum contribution. ## Expected benefit, confidence, and risks - Benefit: enable full-corpus indexing within bounded physical memory and create headroom for later concurrency. - Confidence: high that complete-unit plus eager analyzer-index coexistence is a material architectural constraint; medium that it is the dominant allocation and medium on the best decomposition until phase-level checkpoints quantify each layer. - Risks: changing analysis order can alter stable IDs, first-owner selection, diagnostic order, relationship ordering, certainty, or source anchors; spill/stream failure handling must preserve atomic publication. ## Acceptance criteria - The pinned 1,103-TU workload completes parsing, semantic analysis, staging, validation, checkpoint, read-only reopen, and publication under four logical CPUs, `MemoryMax=32G`, and `MemorySwapMax=0`; report phase and full end-to-end durations rather than extrapolating from the current failed run. - Peak cgroup memory is at most 28 GiB, leaving at least 12.5% headroom below the enforced limit. - The existing first-200 oracle remains deeply equal for complete owned `IndexedTranslationUnit` outputs, including diagnostics, dependencies, source anchors, identities, and order, and for semantic definitions, uses, relationships, certainty, uncertainty reasons, endpoint fingerprints, and persisted source-fact families; count-only parity is insufficient. - All 21 full-corpus emitted parser cardinalities remain exactly the values in the retained hashed log. Full-corpus semantic and persistence counts/fingerprints are recorded as a new durable baseline because current HEAD cannot complete those phases. - Failure injection proves staging cleanup and no partial publication when parsing, analysis, mapping, write, checkpoint, or reopen validation fails. - Linux full-runtime validation, Windows cross-compilation, formatting, lint, tests, and docs pass through the pinned Nix environment. ## Rollback Keep the existing eager analyzer path as a temporary test oracle while implementing the bounded path. If exact-output parity, failure atomicity, or the 32 GiB target fails, retain the measurements, remove the candidate path, and leave current production behavior unchanged.
Author
Owner

Delivered the independently validated semantic-memory remediation to remote main.

Measured result on the exact API-compatible candidate:

  • Paired first 200 TUs: semantic analysis 23.685s -> 19.527s (-17.56%); rounded cgroup peak 3.9 GiB -> 1.8 GiB (-53.85%).
  • Exact ordered complete-output digest remained fcbb1a390e51d5e6dc87ae13633e2a7c3c929e8c4c5bc1c1f24516fba16d39c7, with identical 363,167 definitions, 396,913 uses, and 772,961 relationships.
  • Full 1,103 TUs: every semantic stage completed; authoritative whole-service peak through production mapping was 26,812,751,872 bytes (24.97 GiB), with zero swap and no OOM kill. This is below the 28-GiB target.
  • Full semantic output: 4,224,058 definitions, 4,973,123 uses, and 9,551,602 relationships.
  • Required Nix formatting, lint, all-target/all-feature tests (289 unit tests plus all integration suites), docs, and Windows GNU cross-compilation passed.

The candidate borrows immutable analyzer facts, uses typed dense lookup with a sparse first-insertion-preserving fallback, shares validated ProjectPath/SymbolKey text, and shares scope chains behind a private Owned/Shared representation. Public const APIs and prior Debug shape remain intact.

The run then stopped in source-fact mapping with the independently pre-existing ConflictingCanonicalSymbol("fallback:FieldDecl:src/melee/ft/types.h:48867"); it did not reach staging, checkpoint, reopen, or publication. This blocker is now tracked separately in #55. I am keeping #53 open until #55 is resolved and the complete production workflow verifies the final persistence-stage peak and fingerprints.

Residual evidence limitation: current full parser cardinalities differ from the older retained issue oracle despite matching corpus and compilation-database hashes; paired current-environment parser counts and semantic digest do match exactly.

Delivered the independently validated semantic-memory remediation to remote `main`. - Implementation: [`ec7131294b37ba95d81fd71afe46081f65daeafd`](https://git.jacoby6000.com/JayKobe6k/ClariTree/commit/ec7131294b37ba95d81fd71afe46081f65daeafd) - Evidence record: [`dfa4f39818c3c9efb04cacd5e88ee72d9a279ca7`](https://git.jacoby6000.com/JayKobe6k/ClariTree/commit/dfa4f39818c3c9efb04cacd5e88ee72d9a279ca7) - Remote verification follow-up: [`7bcb186accb315752961c7188dc7b074cced9cd0`](https://git.jacoby6000.com/JayKobe6k/ClariTree/commit/7bcb186accb315752961c7188dc7b074cced9cd0) Measured result on the exact API-compatible candidate: - Paired first 200 TUs: semantic analysis 23.685s -> 19.527s (-17.56%); rounded cgroup peak 3.9 GiB -> 1.8 GiB (-53.85%). - Exact ordered complete-output digest remained `fcbb1a390e51d5e6dc87ae13633e2a7c3c929e8c4c5bc1c1f24516fba16d39c7`, with identical 363,167 definitions, 396,913 uses, and 772,961 relationships. - Full 1,103 TUs: every semantic stage completed; authoritative whole-service peak through production mapping was 26,812,751,872 bytes (24.97 GiB), with zero swap and no OOM kill. This is below the 28-GiB target. - Full semantic output: 4,224,058 definitions, 4,973,123 uses, and 9,551,602 relationships. - Required Nix formatting, lint, all-target/all-feature tests (289 unit tests plus all integration suites), docs, and Windows GNU cross-compilation passed. The candidate borrows immutable analyzer facts, uses typed dense lookup with a sparse first-insertion-preserving fallback, shares validated `ProjectPath`/`SymbolKey` text, and shares scope chains behind a private Owned/Shared representation. Public `const` APIs and prior Debug shape remain intact. The run then stopped in source-fact mapping with the independently pre-existing `ConflictingCanonicalSymbol("fallback:FieldDecl:src/melee/ft/types.h:48867")`; it did not reach staging, checkpoint, reopen, or publication. This blocker is now tracked separately in [#55](https://git.jacoby6000.com/JayKobe6k/ClariTree/issues/55). I am keeping #53 open until #55 is resolved and the complete production workflow verifies the final persistence-stage peak and fingerprints. Residual evidence limitation: current full parser cardinalities differ from the older retained issue oracle despite matching corpus and compilation-database hashes; paired current-environment parser counts and semantic digest do match exactly.
Author
Owner

Full-corpus follow-up after isolating issue #55:

The validated uncommitted parser normalization removes the anonymous-field canonical-symbol conflict through the smallest known reproducing prefix: 547 TUs source-mapped successfully with a 21.7-GiB peak and zero swap. Exact first-200 semantic counts and ordered digest remain unchanged.

The newly reachable next constraint is source-fact mapping memory. A 1,103-TU source-only MetadataGraph run parsed all units in 1,247,552 ms, then was OOM-killed during mapping at MemoryMax=32G, MemorySwapMax=0; staging, checkpoint, reopen validation, and publication were not reached. This is not a successful end-to-end run and issue #53 should remain open.

The issue #55 root cause, candidate, validation, and this bounded failure are recorded in annals/2026-09-12-anonymous-record-field-symbol-normalization.md. The next optimization target is the O(source facts) SourceFactGraph coexistence with all retained translation units, not semantic analysis or the LadybugDB buffer pool.

Full-corpus follow-up after isolating issue #55: The validated uncommitted parser normalization removes the anonymous-field canonical-symbol conflict through the smallest known reproducing prefix: 547 TUs source-mapped successfully with a 21.7-GiB peak and zero swap. Exact first-200 semantic counts and ordered digest remain unchanged. The newly reachable next constraint is source-fact mapping memory. A 1,103-TU source-only `MetadataGraph` run parsed all units in 1,247,552 ms, then was OOM-killed during mapping at `MemoryMax=32G`, `MemorySwapMax=0`; staging, checkpoint, reopen validation, and publication were not reached. This is not a successful end-to-end run and issue #53 should remain open. The issue #55 root cause, candidate, validation, and this bounded failure are recorded in `annals/2026-09-12-anonymous-record-field-symbol-normalization.md`. The next optimization target is the O(source facts) `SourceFactGraph` coexistence with all retained translation units, not semantic analysis or the LadybugDB buffer pool.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
JayKobe6k/ClariTree#53
No description provided.