Allow per-publication LadybugDB buffer-pool tuning #39

New issue

Closed

opened 2026-09-10 21:53:43 +00:00 by JayKobe6k · 2 comments

JayKobe6k commented

2026-09-10 21:53:43 +00:00

Owner

Problem and impact

ClariTree currently hard-codes a 4 GiB LadybugDB buffer pool for every database open. Publication workloads and user memory budgets differ, so one fixed cache limit cannot balance memory headroom, I/O, and throughput for every project. Users cannot currently lower the native cache allowance on constrained systems or raise it for deliberately provisioned indexing jobs.

This is an operational tuning control, not a promise that process RSS equals the selected pool size. Measurements below show substantial non-buffer-pool memory in LadybugDB publication.

Environment and workload

ClariTree base: 912603680d plus the uncommitted issue #38 streaming candidate
LadybugDB Rust crate: lbug 0.20.2
Linux 6.18.46-1-lts, x86_64
Rust/Cargo 1.93.0
Corpus: melee-event-match at 6b6f8a8a987c2e871b8b96baa863aaa99e406f33
Workload: first 40 deterministic compilation-plan entries
Output: 28,375 definitions, 28,847 uses, 62,693 lineage relationships, and 729,601 source facts
Release ignored harness: tests/profile_lineage.rs
Safety envelope: MemoryMax=8G, MemoryHigh=7G, MemorySwapMax=0, 420-second runtime limit

Baseline and evidence

The fixed value is defined at src/persistence/ladybug.rs:44 and passed to every SystemConfig database open. Upstream documents buffer_pool_size as a byte-valued maximum controlling cached database pages: https://docs.rs/lbug/0.20.2/lbug/struct.SystemConfig.html#method.buffer_pool_size

Controlled candidate runs after plumbing a configurable value:

Pool	Persistence	Peak RSS	Result
4096 MiB	47.739 s	5,825,968 KiB	success
1024 MiB	52.059 s	5,840,036 KiB	success
512 MiB	49.091 s	5,841,484 KiB	success
256 MiB	47.599 s	5,804,488 KiB	success
128 MiB	48.482 s	5,696,180 KiB	success
64 MiB	failed after 3.2 s persistence	2,059,984 KiB	LadybugDB buffer pool full

A 128 MiB pool reduced peak RSS by 129,788 KiB (2.2%) versus 4096 MiB at a 0.743 s (1.6%) persistence cost in these single runs. Values from 256 to 4096 MiB had effectively identical peak RSS, proving that the pool is not the dominant retained-memory source for this workload. The 64 MiB failure also proves that undersizing has a workload-dependent correctness/availability cost that must remain a visible typed database error.

Root cause

LadybugMetadataStore::open always constructs SystemConfig with one process-wide constant. IndexPersistenceConfiguration already carries the other per-publication memory and batching controls from both index new and index update, but it has no typed buffer-pool field.

Proposed implementation

Add a typed DatabaseBufferPoolSize measured in non-zero mebibytes, with the existing 4096 MiB behavior as the default.
Carry it in IndexPersistenceConfiguration without changing existing constructor callers; provide a typed builder and byte conversion at the Ladybug adapter boundary.
Add --database-buffer-pool-size-mib MEBIBYTES to index new and index update.
Use the selected value for both writable staging and read-only reopen validation of that publication.
Keep ordinary query opens on the existing default until there is evidence and a separate design for persistent or per-query tuning.
Document that this is a native cache limit, not a total process-memory cap, and preserve native typed failures for undersized values.

Alternatives considered:

Native auto-detection would make memory use depend on total host RAM and defeat predictable indexing envelopes.
Persisting this operational setting in the database or catalog would couple future query behavior to creation-time hardware. The immediate request concerns publication-time tuning.
Lowering the global default is not justified: the smallest successful value is workload dependent, and only one bounded corpus was measured.

Expected benefit and risks

Benefit: users can select a publication cache budget appropriate to available memory and I/O performance.
Confidence: high that the CLI value reaches LadybugDB; benchmark success/failure changes across values verify the native setting. Low confidence in universal RSS savings because non-pool allocations dominate the measured sample.
Risks: too-small pools fail writes; lower values may increase I/O and wall time; MiB/byte conversion must not overflow.
Compatibility: additive CLI and typed API change; default remains 4096 MiB; no schema or data migration.
Rollback: remove the additive option and builder, restoring the constant without touching persisted databases.

Acceptance criteria

Both index new and index update expose a non-zero --database-buffer-pool-size-mib option with a 4096 MiB default.
The typed value reaches both writable staging and read-only reopen SystemConfig initialization.
Zero is rejected by CLI parsing before application dispatch.
Existing programmatic IndexPersistenceConfiguration construction retains the 4 GiB default.
Help and architecture documentation explain scope and units.
A bounded benchmark reports selected pool, persistence time, peak RSS, counts, and typed failure behavior.
Publication parity, rollback, durability, and validation behavior remain unchanged.
Formatting, Clippy, all tests, rustdoc, and Windows cross-compilation pass through Nix.

## Problem and impact ClariTree currently hard-codes a 4 GiB LadybugDB buffer pool for every database open. Publication workloads and user memory budgets differ, so one fixed cache limit cannot balance memory headroom, I/O, and throughput for every project. Users cannot currently lower the native cache allowance on constrained systems or raise it for deliberately provisioned indexing jobs. This is an operational tuning control, not a promise that process RSS equals the selected pool size. Measurements below show substantial non-buffer-pool memory in LadybugDB publication. ## Environment and workload - ClariTree base: 912603680d78db19b00c0df2e8a62038193afea0 plus the uncommitted issue #38 streaming candidate - LadybugDB Rust crate: lbug 0.20.2 - Linux 6.18.46-1-lts, x86_64 - Rust/Cargo 1.93.0 - Corpus: melee-event-match at 6b6f8a8a987c2e871b8b96baa863aaa99e406f33 - Workload: first 40 deterministic compilation-plan entries - Output: 28,375 definitions, 28,847 uses, 62,693 lineage relationships, and 729,601 source facts - Release ignored harness: tests/profile_lineage.rs - Safety envelope: MemoryMax=8G, MemoryHigh=7G, MemorySwapMax=0, 420-second runtime limit ## Baseline and evidence The fixed value is defined at [src/persistence/ladybug.rs:44](https://git.jacoby6000.com/JayKobe6k/ClariTree/src/commit/912603680d78db19b00c0df2e8a62038193afea0/src/persistence/ladybug.rs#L44) and passed to every SystemConfig database open. Upstream documents buffer_pool_size as a byte-valued maximum controlling cached database pages: https://docs.rs/lbug/0.20.2/lbug/struct.SystemConfig.html#method.buffer_pool_size Controlled candidate runs after plumbing a configurable value: | Pool | Persistence | Peak RSS | Result | |---:|---:|---:|---| | 4096 MiB | 47.739 s | 5,825,968 KiB | success | | 1024 MiB | 52.059 s | 5,840,036 KiB | success | | 512 MiB | 49.091 s | 5,841,484 KiB | success | | 256 MiB | 47.599 s | 5,804,488 KiB | success | | 128 MiB | 48.482 s | 5,696,180 KiB | success | | 64 MiB | failed after 3.2 s persistence | 2,059,984 KiB | LadybugDB buffer pool full | A 128 MiB pool reduced peak RSS by 129,788 KiB (2.2%) versus 4096 MiB at a 0.743 s (1.6%) persistence cost in these single runs. Values from 256 to 4096 MiB had effectively identical peak RSS, proving that the pool is not the dominant retained-memory source for this workload. The 64 MiB failure also proves that undersizing has a workload-dependent correctness/availability cost that must remain a visible typed database error. ## Root cause LadybugMetadataStore::open always constructs SystemConfig with one process-wide constant. IndexPersistenceConfiguration already carries the other per-publication memory and batching controls from both index new and index update, but it has no typed buffer-pool field. ## Proposed implementation - Add a typed DatabaseBufferPoolSize measured in non-zero mebibytes, with the existing 4096 MiB behavior as the default. - Carry it in IndexPersistenceConfiguration without changing existing constructor callers; provide a typed builder and byte conversion at the Ladybug adapter boundary. - Add --database-buffer-pool-size-mib MEBIBYTES to index new and index update. - Use the selected value for both writable staging and read-only reopen validation of that publication. - Keep ordinary query opens on the existing default until there is evidence and a separate design for persistent or per-query tuning. - Document that this is a native cache limit, not a total process-memory cap, and preserve native typed failures for undersized values. Alternatives considered: - Native auto-detection would make memory use depend on total host RAM and defeat predictable indexing envelopes. - Persisting this operational setting in the database or catalog would couple future query behavior to creation-time hardware. The immediate request concerns publication-time tuning. - Lowering the global default is not justified: the smallest successful value is workload dependent, and only one bounded corpus was measured. ## Expected benefit and risks - Benefit: users can select a publication cache budget appropriate to available memory and I/O performance. - Confidence: high that the CLI value reaches LadybugDB; benchmark success/failure changes across values verify the native setting. Low confidence in universal RSS savings because non-pool allocations dominate the measured sample. - Risks: too-small pools fail writes; lower values may increase I/O and wall time; MiB/byte conversion must not overflow. - Compatibility: additive CLI and typed API change; default remains 4096 MiB; no schema or data migration. - Rollback: remove the additive option and builder, restoring the constant without touching persisted databases. ## Acceptance criteria - [ ] Both index new and index update expose a non-zero --database-buffer-pool-size-mib option with a 4096 MiB default. - [ ] The typed value reaches both writable staging and read-only reopen SystemConfig initialization. - [ ] Zero is rejected by CLI parsing before application dispatch. - [ ] Existing programmatic IndexPersistenceConfiguration construction retains the 4 GiB default. - [ ] Help and architecture documentation explain scope and units. - [ ] A bounded benchmark reports selected pool, persistence time, peak RSS, counts, and typed failure behavior. - [ ] Publication parity, rollback, durability, and validation behavior remain unchanged. - [ ] Formatting, Clippy, all tests, rustdoc, and Windows cross-compilation pass through Nix.

JayKobe6k commented

2026-09-10 22:08:23 +00:00

Author

Owner

Implementation candidate status (uncommitted)

Added typed non-zero DatabaseBufferPoolSize and carried it through IndexPersistenceConfiguration.
Added --database-buffer-pool-size-mib to both index new and index update, retaining the 4096 MiB compatibility default.
The selected value configures writable staging and read-only reopen validation.
Added default, propagation, CLI translation, help, zero-rejection, and configured-open tests.
Documentation now states explicitly that this limits only the LadybugDB page cache, not total process RSS, and that undersized pools can fail publication.
Review removed a misleading benchmark-tuned claim for the 4096 MiB value; measurements support compatibility, not optimality.
The complete required Nix validation passes: format, Clippy, 273 unit tests plus all integration suites, rustdoc, Windows GNU cross-check, and git diff --check.

The bounded 40-TU measurements and 64 MiB failure in the issue body remain the validation evidence. A follow-up 4-CPU-affinity experiment at 4096 MiB took 44.957 s persistence and still peaked at approximately 5.6 GiB, versus 47.739 s and approximately 5.6 GiB with all 16 CPUs. This did not identify thread-count tuning as a meaningful memory improvement, so no additional tuning issue is justified.

Implementation candidate status (uncommitted) - Added typed non-zero DatabaseBufferPoolSize and carried it through IndexPersistenceConfiguration. - Added --database-buffer-pool-size-mib to both index new and index update, retaining the 4096 MiB compatibility default. - The selected value configures writable staging and read-only reopen validation. - Added default, propagation, CLI translation, help, zero-rejection, and configured-open tests. - Documentation now states explicitly that this limits only the LadybugDB page cache, not total process RSS, and that undersized pools can fail publication. - Review removed a misleading benchmark-tuned claim for the 4096 MiB value; measurements support compatibility, not optimality. - The complete required Nix validation passes: format, Clippy, 273 unit tests plus all integration suites, rustdoc, Windows GNU cross-check, and git diff --check. The bounded 40-TU measurements and 64 MiB failure in the issue body remain the validation evidence. A follow-up 4-CPU-affinity experiment at 4096 MiB took 44.957 s persistence and still peaked at approximately 5.6 GiB, versus 47.739 s and approximately 5.6 GiB with all 16 CPUs. This did not identify thread-count tuning as a meaningful memory improvement, so no additional tuning issue is justified.

JayKobe6k referenced this issue

2026-09-11 00:44:28 +00:00

Reduce LadybugDB publication time and memory with typed node COPY #40

JayKobe6k commented

2026-09-11 01:03:16 +00:00

Author

Owner

Implemented and pushed to main as 654b2fbcf8e3ccc7ce7428b637d8b2736f67de19.

Both index new and index update now expose the typed non-zero --database-buffer-pool-size-mib setting, preserve the 4096 MiB compatibility default, and apply it to writable staging and read-only validation. CLI rejection/help, application defaults, publication behavior, documentation, full tests, rustdoc, and Windows cross-compilation passed through Nix.

Implemented and pushed to `main` as [`654b2fbcf8e3ccc7ce7428b637d8b2736f67de19`](https://git.jacoby6000.com/JayKobe6k/ClariTree/commit/654b2fbcf8e3ccc7ce7428b637d8b2736f67de19). Both `index new` and `index update` now expose the typed non-zero `--database-buffer-pool-size-mib` setting, preserve the 4096 MiB compatibility default, and apply it to writable staging and read-only validation. CLI rejection/help, application defaults, publication behavior, documentation, full tests, rustdoc, and Windows cross-compilation passed through Nix.

JayKobe6k closed this issue

2026-09-11 01:03:17 +00:00

JayKobe6k referenced this issue

2026-09-11 02:25:27 +00:00

Reduce high-volume source-fact publication with typed COPY #41