Make libclang fallback symbol keys collision-safe across translation units #55

New issue

Open

opened 2026-09-13 00:16:22 +00:00 by JayKobe6k · 1 comment

JayKobe6k commented

2026-09-13 00:16:22 +00:00

Owner

Problem and impact

Full-corpus indexing now completes semantic analysis inside the issue #53 memory bound, but production source-fact mapping rejects the pinned 1,103-TU corpus with ConflictingCanonicalSymbol("fallback:FieldDecl:src/melee/ft/types.h:48867"). Staging, checkpoint, read-only reopen, and publication therefore never run. This independently blocks the complete workflow and issue #53 closure.

Do not weaken or suppress canonical-symbol conflict validation: two observations assigned one key but carrying incompatible canonical metadata would otherwise be silently merged.

Environment and reproduction

ClariTree baseline/current committed HEAD: 51a57a890d2e1c06ebb3964b71227800a40fa2c4
Corpus: melee-event-match 6b6f8a8a987c2e871b8b96baa863aaa99e406f33
compile_commands.json SHA-256: 0c025066b2f5ed7940a668d4fc97b912d8eb5d467fb7bb33ba0e72ab7d50ea16
Linux/Nix release profile; 1,103 units; four logical CPUs
MemoryMax=32G, MemorySwapMax=0, RuntimeMaxSec=7200
CLARITREE_PROFILE_PERSISTENCE=1, 4,096 MiB LadybugDB buffer pool

Build and run the ignored profile_lineage release test through Nix with CLARITREE_PROFILE_PROJECT pointing to the pinned corpus and CLARITREE_PROFILE_LIMIT=1103. The current memory-remediation candidate is not required to explain the collision: it changes identity storage and analyzer indexes, not fallback symbol construction or persistence canonicalization. Earlier baseline runs died before mapping and therefore could not expose this later failure.

Baseline evidence

The final bounded run completed semantic analysis with 4,224,058 definitions, 4,973,123 uses, and 9,551,602 relationships, then emitted:

Error: Mapping { source: ConflictingCanonicalSymbol("fallback:FieldDecl:src/melee/ft/types.h:48867") }
MemoryPeak=26812751872
MemorySwapPeak=0

The process exited 101 from the typed mapping error and was not OOM-killed. The sanitized local artifact is target/issue53-shared-scopes-full.log; its measured facts and hashes are preserved in annals/2026-09-12-borrowed-dense-analysis-indexes.md.

The referenced byte offset is at the anonymous union beginning near src/melee/ft/types.h:1586, containing throw_flags and an anonymous bit-field struct.

Source-backed root-cause boundary

At symbol_key, a cursor without a usable USR receives a fallback key containing only entity kind, portable path, and spelling start byte offset. At map_symbols, observations with the same key are canonicalized project-wide. merge_canonical_symbol correctly rejects equal keys whose ID, key, name, kind, or linkage differs.

The verified failure therefore proves that the fallback key is not unique enough for these observations, or that metadata expected to be canonical varies across compilation contexts. The exact disagreeing fields and translation units have not yet been captured, so the precise parser remedy remains to be measured rather than guessed.

Proposed implementation

Extend the typed conflict diagnostic or a feature-gated profiler to report sanitized existing/candidate metadata and translation-unit identities for this key.
Add the smallest multi-TU libclang regression fixture reproducing the anonymous-union/field observation under the relevant compilation-context difference.
Correct parser fallback identity using stable libclang/source provenance sufficient to distinguish genuinely distinct declarations while continuing to merge repeated observations of the same declaration. Prefer spelling and expansion range endpoints plus validated enclosing declaration/scope identity where USR is unavailable; do not use traversal ordinal alone.
Retain strict persistence conflict validation as the independent oracle.

Alternatives rejected: ignoring the conflict, selecting the first observation, or weakening metadata comparison can silently conflate distinct symbols; including compilation fingerprint unconditionally would prevent legitimate cross-TU canonicalization.

Expected benefit, risks, and validation

Expected benefit is restoring full-corpus publication after issue #53 creates sufficient memory headroom. Confidence is high in the collision boundary and medium in the final discriminator until both observations are captured. Risks are fragmenting one real symbol across TUs, making keys traversal-dependent, or changing search/lineage handles.

Acceptance criteria:

A deterministic multi-TU regression reproduces the current collision before the fix and maps successfully after it.
Diagnostic evidence identifies both observations and every disagreeing canonical field.
Repeated declarations of the same fallback symbol still merge across translation units and traversal order.
Distinct anonymous/nested declarations at the same apparent start location retain distinct stable keys.
The pinned 1,103-TU workflow completes mapping, staging, validation, checkpoint, read-only reopen, and publication with exact persisted count/fingerprint validation.
Issue #53 is rerun under its 32-GiB zero-swap envelope and remains at or below 28 GiB.
Required Nix format, lint, all-target/all-feature tests, docs, and Windows GNU cross-compilation pass.

Rollback: revert the fallback-key change if canonical handles become unstable or same-declaration observations stop merging; retain the diagnostic and strict conflict test for the next design.

## Problem and impact Full-corpus indexing now completes semantic analysis inside the issue #53 memory bound, but production source-fact mapping rejects the pinned 1,103-TU corpus with `ConflictingCanonicalSymbol("fallback:FieldDecl:src/melee/ft/types.h:48867")`. Staging, checkpoint, read-only reopen, and publication therefore never run. This independently blocks the complete workflow and issue #53 closure. Do not weaken or suppress canonical-symbol conflict validation: two observations assigned one key but carrying incompatible canonical metadata would otherwise be silently merged. ## Environment and reproduction - ClariTree baseline/current committed HEAD: `51a57a890d2e1c06ebb3964b71227800a40fa2c4` - Corpus: `melee-event-match` `6b6f8a8a987c2e871b8b96baa863aaa99e406f33` - `compile_commands.json` SHA-256: `0c025066b2f5ed7940a668d4fc97b912d8eb5d467fb7bb33ba0e72ab7d50ea16` - Linux/Nix release profile; 1,103 units; four logical CPUs - `MemoryMax=32G`, `MemorySwapMax=0`, `RuntimeMaxSec=7200` - `CLARITREE_PROFILE_PERSISTENCE=1`, 4,096 MiB LadybugDB buffer pool Build and run the ignored `profile_lineage` release test through Nix with `CLARITREE_PROFILE_PROJECT` pointing to the pinned corpus and `CLARITREE_PROFILE_LIMIT=1103`. The current memory-remediation candidate is not required to explain the collision: it changes identity storage and analyzer indexes, not fallback symbol construction or persistence canonicalization. Earlier baseline runs died before mapping and therefore could not expose this later failure. ## Baseline evidence The final bounded run completed semantic analysis with 4,224,058 definitions, 4,973,123 uses, and 9,551,602 relationships, then emitted: ```text Error: Mapping { source: ConflictingCanonicalSymbol("fallback:FieldDecl:src/melee/ft/types.h:48867") } MemoryPeak=26812751872 MemorySwapPeak=0 ``` The process exited 101 from the typed mapping error and was not OOM-killed. The sanitized local artifact is `target/issue53-shared-scopes-full.log`; its measured facts and hashes are preserved in `annals/2026-09-12-borrowed-dense-analysis-indexes.md`. The referenced byte offset is at the anonymous union beginning near `src/melee/ft/types.h:1586`, containing `throw_flags` and an anonymous bit-field struct. ## Source-backed root-cause boundary At [`symbol_key`](https://git.jacoby6000.com/JayKobe6k/ClariTree/src/commit/51a57a890d2e1c06ebb3964b71227800a40fa2c4/src/parser/libclang.rs#L2427-L2468), a cursor without a usable USR receives a fallback key containing only entity kind, portable path, and spelling start byte offset. At [`map_symbols`](https://git.jacoby6000.com/JayKobe6k/ClariTree/src/commit/51a57a890d2e1c06ebb3964b71227800a40fa2c4/src/persistence/source.rs#L1672-L1713), observations with the same key are canonicalized project-wide. [`merge_canonical_symbol`](https://git.jacoby6000.com/JayKobe6k/ClariTree/src/commit/51a57a890d2e1c06ebb3964b71227800a40fa2c4/src/persistence/source.rs#L1716-L1739) correctly rejects equal keys whose ID, key, name, kind, or linkage differs. The verified failure therefore proves that the fallback key is not unique enough for these observations, or that metadata expected to be canonical varies across compilation contexts. The exact disagreeing fields and translation units have not yet been captured, so the precise parser remedy remains to be measured rather than guessed. ## Proposed implementation 1. Extend the typed conflict diagnostic or a feature-gated profiler to report sanitized existing/candidate metadata and translation-unit identities for this key. 2. Add the smallest multi-TU libclang regression fixture reproducing the anonymous-union/field observation under the relevant compilation-context difference. 3. Correct parser fallback identity using stable libclang/source provenance sufficient to distinguish genuinely distinct declarations while continuing to merge repeated observations of the same declaration. Prefer spelling and expansion range endpoints plus validated enclosing declaration/scope identity where USR is unavailable; do not use traversal ordinal alone. 4. Retain strict persistence conflict validation as the independent oracle. Alternatives rejected: ignoring the conflict, selecting the first observation, or weakening metadata comparison can silently conflate distinct symbols; including compilation fingerprint unconditionally would prevent legitimate cross-TU canonicalization. ## Expected benefit, risks, and validation Expected benefit is restoring full-corpus publication after issue #53 creates sufficient memory headroom. Confidence is high in the collision boundary and medium in the final discriminator until both observations are captured. Risks are fragmenting one real symbol across TUs, making keys traversal-dependent, or changing search/lineage handles. Acceptance criteria: - A deterministic multi-TU regression reproduces the current collision before the fix and maps successfully after it. - Diagnostic evidence identifies both observations and every disagreeing canonical field. - Repeated declarations of the same fallback symbol still merge across translation units and traversal order. - Distinct anonymous/nested declarations at the same apparent start location retain distinct stable keys. - The pinned 1,103-TU workflow completes mapping, staging, validation, checkpoint, read-only reopen, and publication with exact persisted count/fingerprint validation. - Issue #53 is rerun under its 32-GiB zero-swap envelope and remains at or below 28 GiB. - Required Nix format, lint, all-target/all-feature tests, docs, and Windows GNU cross-compilation pass. Rollback: revert the fallback-key change if canonical handles become unstable or same-declaration observations stop merging; retain the diagnostic and strict conflict test for the next design.

JayKobe6k added the

area/core

area/indexing

type/foundation

labels

2026-09-13 00:17:11 +00:00

JayKobe6k referenced this issue

2026-09-13 01:12:49 +00:00

Bound full-project indexing memory below the corpus working set #53

JayKobe6k commented

2026-09-13 02:57:13 +00:00

Author

Owner

Investigation update (validated, uncommitted):

The conflicting observations are the same implicit anonymous-record FieldDecl, not two declarations that need different fallback keys. libclang generated either union Fighter::(anonymous at src/melee/ft/types.h:1586:19) or the equivalent absolute-path spelling while both observations retained fallback:FieldDecl:src/melee/ft/types.h:48867. The unstable path is presentation metadata. Strict merge_canonical_symbol validation remains unchanged.

The candidate now recognizes only a FieldDecl whose generated name contains (anonymous at and whose type declaration is an anonymous record, then emits name=None, display <anonymous>, and qualified component <anonymous>. A focused libclang regression proves the implicit field becomes Fighter::<anonymous> while an explicitly named field of anonymous-record type remains Fighter::named.

Validation evidence on melee-event-match 6b6f8a8a987c2e871b8b96baa863aaa99e406f33:

First 547 TUs (the smallest known prefix containing the outlier at unit 546) completed source mapping: parse 264,858 ms, mapping 105,386 ms, 21.7-GiB cgroup peak, zero swap.
First 200 TUs retained 363,167 definitions, 396,913 uses, 772,961 relationships and exact ordered digest fcbb1a390e51d5e6dc87ae13633e2a7c3c929e8c4c5bc1c1f24516fba16d39c7; peak 1.9 GiB, zero swap.
Nix formatting, lint, all-target/all-feature tests (289 unit tests plus all integration suites), rustdoc, and Windows GNU all-target/all-feature checking pass.

A full 1,103-TU source-only mapping attempt parsed all units but was then OOM-killed at the 32-GiB zero-swap bound during MetadataGraph construction. Thus this candidate removes the issue #55 correctness failure, but it does not satisfy the issue #53 full-publication acceptance criterion. Evidence and limitations are recorded in annals/2026-09-12-anonymous-record-field-symbol-normalization.md. Issue #55 remains open pending delivery/acceptance review.

Investigation update (validated, uncommitted): The conflicting observations are the same implicit anonymous-record `FieldDecl`, not two declarations that need different fallback keys. libclang generated either `union Fighter::(anonymous at src/melee/ft/types.h:1586:19)` or the equivalent absolute-path spelling while both observations retained `fallback:FieldDecl:src/melee/ft/types.h:48867`. The unstable path is presentation metadata. Strict `merge_canonical_symbol` validation remains unchanged. The candidate now recognizes only a `FieldDecl` whose generated name contains `(anonymous at ` and whose type declaration is an anonymous record, then emits `name=None`, display `<anonymous>`, and qualified component `<anonymous>`. A focused libclang regression proves the implicit field becomes `Fighter::<anonymous>` while an explicitly named field of anonymous-record type remains `Fighter::named`. Validation evidence on `melee-event-match` `6b6f8a8a987c2e871b8b96baa863aaa99e406f33`: - First 547 TUs (the smallest known prefix containing the outlier at unit 546) completed source mapping: parse 264,858 ms, mapping 105,386 ms, 21.7-GiB cgroup peak, zero swap. - First 200 TUs retained 363,167 definitions, 396,913 uses, 772,961 relationships and exact ordered digest `fcbb1a390e51d5e6dc87ae13633e2a7c3c929e8c4c5bc1c1f24516fba16d39c7`; peak 1.9 GiB, zero swap. - Nix formatting, lint, all-target/all-feature tests (289 unit tests plus all integration suites), rustdoc, and Windows GNU all-target/all-feature checking pass. A full 1,103-TU source-only mapping attempt parsed all units but was then OOM-killed at the 32-GiB zero-swap bound during `MetadataGraph` construction. Thus this candidate removes the issue #55 correctness failure, but it does not satisfy the issue #53 full-publication acceptance criterion. Evidence and limitations are recorded in `annals/2026-09-12-anonymous-record-field-symbol-normalization.md`. Issue #55 remains open pending delivery/acceptance review.

JayKobe6k referenced this issue

2026-09-13 02:57:13 +00:00

Bound full-project indexing memory below the corpus working set #53