ClariTree

ClariTree is an early-stage system for exploring value lineage in obfuscated C source code. It will use libclang to process C abstract syntax trees and expose the relationships that explain where a selected value came from and what was derived from it.

The intended workflow lets a user select a value and:

• locate its declaration and references;
• inspect the values and expressions that contributed to it;
• traverse lineage across declarations, assignments, and function calls; and
• follow derived values in the opposite direction.

Status

The repository contains the foundational language-neutral indexed-fact and lineage models plus a safe libclang adapter for complete C declaration types, scopes, aggregates, enumerations, recursive structural operations, direct and indirect calls, deterministic control flow, preprocessing facts, diagnostics, and exact spelling/expansion anchors. Reaching-definition analysis, semantic lineage persistence, durable value selection, and bounded lineage queries are implemented. claritree index new is connected to the production source, compilation-plan, libclang, semantic-analysis, LadybugDB, and catalog adapters for local directories and Git origins, publishing complete schema-version-2 source, lineage, and search graphs. Incremental and forced claritree index update operations are also composed through those production adapters with immutable revision publication, deterministic fingerprint comparison, and rollback. The production CLI composes catalog, value-resolution, application traversal, and LadybugDB query adapters for query lineage, query search value, query search function, query search struct, and query search enum. Struct search uses explicit record facts and supports canonical tagged, typedef-named, anonymous, nested, forward-declared, and multiply declared record identity.

Structure

src/
├── application/ Typed use cases and frontend/backend ports
├── cli/       Command parsing, dispatch, and text/JSON rendering
├── index/     Durable index identity, catalog, and selector resolution
├── ir/        Language-neutral indexed source facts
├── lineage/   Semantic lineage graph construction
├── parser/    libclang integration and translation-unit processing
├── persistence/ Typed graph records and the LadybugDB adapter
├── lib.rs     Library module boundaries
└── main.rs    Command-line adapter entry point

Keeping libclang-specific processing in parser prevents its cursor lifetimes and API details from leaking into the rest of the application. The ir module holds durable, language-neutral indexed facts, while lineage owns semantic graph construction. The application module defines the typed operations used by every frontend, implements their shared orchestration, and owns the ports implemented by source, analysis, catalog, persistence, and query adapters. Create and update operations acquire a project, plan compilation, index each translation unit, analyze lineage, persist an immutable revision, and finally activate it through a catalog compare-and-swap. Managed Git acquisition returns a non-cloneable lease whose exclusive checkout lock remains held throughout that source-consuming sequence. Search and lineage operations resolve the active revision through that same catalog before invoking typed query ports. Durable value occurrence identity, canonical handles, exact readable selectors, typed resolution outcomes, and the value repository boundary are documented in docs/value-handles.md. The shared typed search request, regex, ordering, result-limit, nested-grouping, and backend-independent repository contracts are documented in docs/search.md. Bounded directional traversal policy, deterministic breadth-first behavior, typed result fields, truncation semantics, and the one-hop repository boundary are documented in docs/lineage-queries.md. Incremental change detection, forced rebuilds, immutable publication, rollback, result fields, and revision retention are documented in docs/index-updates.md.

Libclang is the authoritative C source index. It provides syntax, symbols, declarations, references, caller/callee relationships, preprocessing details, and exact source locations. ClariTree owns the higher-level semantic analysis that turns those facts into distinct value definitions and uses, reaching definitions, alias effects, and producer-to-consumer lineage. Symbol-reference and call edges are therefore inputs to lineage analysis, not substitutes for value-flow edges.

ClariTree will not depend on a JVM or an external code-property-graph engine. The core analysis remains a native Rust component, with libclang confined to the C parser adapter and LadybugDB confined to the persistence adapter.

The LadybugDB graph schema is documented in docs/database-schema.md. Raw Cypher, native values, and row decoding remain private to the LadybugDB adapter.

Create An Index

Provide a local project directory or Git origin containing a compilation database. By default ClariTree reads <project>/compile_commands.json, stores immutable revision databases under the platform user cache, and derives the index name from the project:

claritree index new ./project
claritree --format json index new https://example.test/team/project.git \
  --name project --git-revision main

Use --compile-commands <path> to select another database, --build-program <program> plus repeated --build-arg options to generate it, and repeated --extra-compiler-arg options to extend every libclang invocation. An explicit --index-path must be absolute and its parent must already exist. Names and canonical project identities are unique. A collision fails without publishing; --replace creates and atomically activates the next immutable revision when both the name and project match the active index.

Both index commands accept persistence tuning options. The validated compatibility defaults are --source-fact-write-batch-size 65536, --value-node-write-batch-size 16384, --pending-relationship-batch-size 16384, and --relationship-copy-batch-size 65536. Batch sizes must be non-zero; override them only when profiling a specific project or system. The source-fact limit bounds both parameter batches and typed CSV imports; iterator-projected relationship imports retain at most one such batch.

Successful text and JSON output includes a selector equal to the index name, the database path, stable index/project IDs, source identity and Git commit, schema/revision, complete fact counts, diagnostics, durability status, and elapsed time. Pass the selector directly as query --index <selector> when using a frontend with query services enabled.

Search Enumerations

Search indexed enum declarations and definitions through the production application service:

claritree query --index project search enum --name '^Status$'
claritree --format json query --index project search enum \
  --translation-unit '^src/' --struct '^Packet$' --order-by name

Results retain canonical identity, all declaration occurrences, exact source anchors, enclosing scopes, underlying and canonical type spellings, typed enumerator values, and a durable revision-scoped enum handle. See docs/search.md for identity, filters, grouping, ordering, text, and JSON details.

The owned structural fact model, relationship directions, validation rules, unsupported-construct policy, and current control-flow limits are documented in docs/c-indexed-facts.md.

Project acquisition, managed Git refresh/reuse behavior, compilation database discovery, path normalization, command tokenization, ordering, deduplication, and fingerprinting are documented in docs/project-sources.md.

Update An Index

Update the cataloged index for a local or managed Git project, or force every translation unit through the same immutable publication path:

claritree index update ./project
claritree index update https://example.test/project.git --git-checkout-policy refresh
claritree --format json index update ./project --reindex

When the canonical project has no catalog entry, pass the existing database as --index-path <path>. Unchanged incremental input returns a deterministic no-op without creating a revision. Changed updates report old/new revisions, changed and affected translation units, lifecycle categories, old/new and net-added/net-removed fact cardinalities, parser diagnostics, and durability. Prior revisions remain immutable and retained. See docs/index-updates.md for the complete contract.

Prerequisites

• Nix with flakes enabled

The Nix development environment provides the pinned Rust toolchain, Clippy, rustfmt, libclang, native build tools, pre-commit, and the Windows GNU cross-toolchain. The same environment is used by local checks and Forgejo CI.

Development

nix develop
pre-commit install
cargo lint
cargo test --all-targets --all-features

Checks can also be run without opening an interactive shell:

nix develop --command pre-commit run --all-files
nix develop --command cargo test --all-targets --all-features

The pre-commit hooks run formatting and Clippy through nix develop, ensuring they use the same dependencies and compiler as CI. Forgejo also checks that the project compiles for x86_64-pc-windows-gnu; executable tests run on Linux because Nix does not provide a native Windows execution environment.

Forgejo caches the Nix store using the pinned flake inputs and separately caches Cargo dependencies and build artifacts. The Nix cache is garbage collected to a maximum size of 2 GiB before new cache entries are saved, and stale cache entries are purged automatically.

Index Identity And Catalog

Projects, logical indexes, and immutable index revisions have separate typed identifiers. A logical index keeps its project identifier, name, and canonical project identity across revisions. Each newer revision must use a database path that is unique across every logical index and historical revision, plus a monotonically increasing revision identifier; Git-backed revisions also record the resolved 40- or 64-character source commit. Names and canonical project identities have one active logical owner and remain stable across replacement revisions. The catalog preserves and validates graph schema version 2. The LadybugDB adapter initializes the complete schema only for a new empty database. It does not modify existing schemas: a nonempty database with any other managed schema version is incompatible. Existing version 1 databases must be recreated as fresh version 2 databases from authoritative source and compilation inputs; in-place migrations are not supported or needed. In version 2, semantic value nodes retain occurrence anchors and denormalized declaration search fields, while typed value-to-declaration relationships lead to exact declaration source provenance.

Local projects are identified by canonical directory paths, so separate worktrees remain distinct. Git projects are identified by normalized origins; transport usernames, URL fragments, query strings, trailing .git, and trailing slashes do not change project identity. Application source requests retain the exact Git fetch location separately so identity normalization does not discard transport usernames. A create collision is rejected; explicit replacement requires both project identity and index name to match one active logical index.

The catalog lives under the platform cache directory selected by the operating system (XDG_CACHE_HOME on Linux when set, and the standard local application data cache on Windows). Catalog records use lossless platform-specific path encoding. Writers serialize through a cross-process lock, sync a temporary record, and publish it as an identifier-named immutable file with an atomic hard link. Readers select the newest complete revision. Missing historical database or project paths are treated as stale metadata rather than catalog corruption; the descriptor remains name-selectable so a later operation can report or repair the missing resource. The hard link is the publication commit point. A subsequent directory-flush failure is reported as a successful, committed publication with unconfirmed directory durability, never as an error that could prompt a caller to delete the indexed database.

An unresolved selector is classified in this order: an existing directory is a canonical local project, a recognized URL or scp-like value is a normalized Git origin, and every other Unicode value is an index name. Existing regular files are not inferred as database selectors. Frontends that intentionally support direct database selection construct the typed database-path selector explicitly.

Validation Corpus

Focused unit and public-boundary coverage indexes handwritten C under tests/fixtures/indexing, including declarations and storage, shadowing, nested aggregates, enums, qualifiers, initialization and mutation, pointers and arrays, macros, calls and returns, branches and loops, unsupported constructs, parser diagnostics, relationship endpoints, and deterministic repeated indexing. The end-to-end test indexes all 15 production translation units from zlib 1.3.1 at commit 51b7f2abdade71cd9bb0e7a373ef2610ec6f9daf. Nix fetches and pins that source tree through flake.lock, so the test does not clone a moving branch or depend on a pre-existing local checkout.

The zlib corpus is approximately 9,500 lines of C and exercises multiple translation units, project headers, macros, conditional compilation, globals, structures, callbacks, and direct calls. The test requires clean parsing and stable lower bounds for indexed function definitions, references, and calls.