Bound full-project indexing memory below the corpus working set #53
Labels
No labels
area/cli
area/core
area/database
area/indexing
area/query
type/epic
type/feature
type/foundation
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
JayKobe6k/ClariTree#53
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem and impact
The current production-path profiler cannot complete indexing of the pinned 1,103-translation-unit
melee-event-matchcorpus inside the reported 32 GiB physical-memory cgroup with cgroup swap disabled. Parsing completes, but the process is OOM-killed after semantic analysis starts and before it reports completion; persistence therefore never starts. This prevents indexing a representative full project even though bounded 200-TU runs succeed. No successful current full end-to-end duration exists.Environment and reproduction
5d04beabeb4bcd00d27e69dd318e5834b097cf30melee-event-matchcommit6b6f8a8a987c2e871b8b96baa863aaa99e406f33compile_commands.jsonSHA-256:0c025066b2f5ed7940a668d4fc97b912d8eb5d467fb7bb33ba0e72ab7d50ea16AllowedCPUs=0-3,CPUQuota=400%,taskset -c 0-3),MemoryMax=32G,MemorySwapMax=0, andRuntimeMaxSec=7200The sanitized retained log independently verifies the parser output, OOM result, service runtime, CPU time, 32 GiB cgroup peak, and zero cgroup swap. It does not retain the environment variables or transient-unit properties, so the launch envelope and persistence request above come from the reproduction record rather than the log itself.
Build the ignored release profiler through Nix, then run all 1,103 units with:
Run the profiler in a transient user service carrying the CPU, memory, swap, and runtime properties above. Do not use
ulimit -v: LadybugDB reserves a large sparse virtual mapping, so a virtual-address limit produces an unrelated database-open failure.Baseline and profiling evidence
The full run reported:
The sum of the 21 emitted parser cardinalities is 80,835,220, including 17,047,327 declarations, 16,828,426 symbols, 9,973,010 references, 6,220,895 aggregate fields, 5,525,983 operation inputs, and 4,435,322 operations. Subtracting the 1,372.076-second parse marker from the 1,407.735-second whole-service runtime gives 35.659 seconds after that marker. That is a derived upper bound that includes post-parse test overhead, not a direct analyzer timestamp. The missing analysis-completion marker, OOM result, and source execution order localize the kill to semantic analysis before LadybugDB publication, but not to a specific analyzer allocation.
The sanitized raw full-run log is retained locally as
target/full-melee-4cpu-current.log, SHA-256aedc41d38b8999e04a36928d54c1ece7cd672419359e595d19abba742c07218a. Local ignored artifacts are not durable; the measurements above are included here in full.For comparison, the delivered first-200 profile documented in closed issue #52 completed with 3.9 GiB peak and zero cgroup swap, producing a sum of 6,823,854 across the same parser cardinalities, 377,067 definitions, 412,330 uses, and 801,507 relationships. The remaining 903 units account for 74,011,366 reported facts and average 81,961.646 per TU versus 34,119.270 for the first 200, a 2.402x density ratio.
Related prior work:
ProjectIndexand source-fact graph.Root cause
The application retains every complete owned translation unit before starting analysis:
index_compilation_planappends allIndexedCompilationUnitvalues to oneVec.TranslationUnitowns all source, symbol, declaration, type, reference, operation, CFG, and preprocessing vectors. Shared headers are represented independently for each compilation context.Analyzer::neweagerly constructs aUnitAnalysisIndexfor every retained unit.UnitAnalysisIndex::newduplicates unit collections into maps and sets before lineage output is produced.ProjectIndexretains both all units and complete lineage. Production persistence then maps a project-sizedSourceFactGraphbeforestage_revisionopens staging storage.The log and source order localize the OOM to semantic analysis, but the precise contribution of analyzer indexes, lineage identities, reaching states, and allocator/native retention has not yet been separated with phase-level memory counters.
Proposed implementation
Analyzer::new, definition/use construction, call binding, control-flow analysis, aggregate validation, source-fact mapping, and database open. Use prefix runs to quantify each layer before changing representation.Keep symbols distinct from value definitions/uses, preserve producer-to-consumer relationship direction, and retain exact spelling/expansion anchors and occurrence-key hashing.
Alternatives considered
MemoryMax: operational workaround only; it leaves working-set growth unbounded.Expected benefit, confidence, and risks
Acceptance criteria
MemoryMax=32G, andMemorySwapMax=0; report phase and full end-to-end durations rather than extrapolating from the current failed run.IndexedTranslationUnitoutputs, including diagnostics, dependencies, source anchors, identities, and order, and for semantic definitions, uses, relationships, certainty, uncertainty reasons, endpoint fingerprints, and persisted source-fact families; count-only parity is insufficient.Rollback
Keep the existing eager analyzer path as a temporary test oracle while implementing the bounded path. If exact-output parity, failure atomicity, or the 32 GiB target fails, retain the measurements, remove the candidate path, and leave current production behavior unchanged.
Delivered the independently validated semantic-memory remediation to remote
main.ec7131294b37ba95d81fd71afe46081f65daeafddfa4f39818c3c9efb04cacd5e88ee72d9a279ca77bcb186accb315752961c7188dc7b074cced9cd0Measured result on the exact API-compatible candidate:
fcbb1a390e51d5e6dc87ae13633e2a7c3c929e8c4c5bc1c1f24516fba16d39c7, with identical 363,167 definitions, 396,913 uses, and 772,961 relationships.The candidate borrows immutable analyzer facts, uses typed dense lookup with a sparse first-insertion-preserving fallback, shares validated
ProjectPath/SymbolKeytext, and shares scope chains behind a private Owned/Shared representation. PublicconstAPIs and prior Debug shape remain intact.The run then stopped in source-fact mapping with the independently pre-existing
ConflictingCanonicalSymbol("fallback:FieldDecl:src/melee/ft/types.h:48867"); it did not reach staging, checkpoint, reopen, or publication. This blocker is now tracked separately in #55. I am keeping #53 open until #55 is resolved and the complete production workflow verifies the final persistence-stage peak and fingerprints.Residual evidence limitation: current full parser cardinalities differ from the older retained issue oracle despite matching corpus and compilation-database hashes; paired current-environment parser counts and semantic digest do match exactly.
Full-corpus follow-up after isolating issue #55:
The validated uncommitted parser normalization removes the anonymous-field canonical-symbol conflict through the smallest known reproducing prefix: 547 TUs source-mapped successfully with a 21.7-GiB peak and zero swap. Exact first-200 semantic counts and ordered digest remain unchanged.
The newly reachable next constraint is source-fact mapping memory. A 1,103-TU source-only
MetadataGraphrun parsed all units in 1,247,552 ms, then was OOM-killed during mapping atMemoryMax=32G,MemorySwapMax=0; staging, checkpoint, reopen validation, and publication were not reached. This is not a successful end-to-end run and issue #53 should remain open.The issue #55 root cause, candidate, validation, and this bounded failure are recorded in
annals/2026-09-12-anonymous-record-field-symbol-normalization.md. The next optimization target is the O(source facts)SourceFactGraphcoexistence with all retained translation units, not semantic analysis or the LadybugDB buffer pool.