Skip to content

block store implementation (littDB)#3600

Open
cody-littley wants to merge 28 commits into
mainfrom
cjl/block-store-impl
Open

block store implementation (littDB)#3600
cody-littley wants to merge 28 commits into
mainfrom
cjl/block-store-impl

Conversation

@cody-littley

Copy link
Copy Markdown
Contributor

Describe your changes and provide context

Implements the blockDB using littDB. Small tweaks to blockDB API.

Testing performed to validate your change

unit tests

@cody-littley cody-littley self-assigned this Jun 16, 2026
@cody-littley cody-littley changed the base branch from main to cjl/litt-iteration June 16, 2026 17:46
@github-actions

github-actions Bot commented Jun 16, 2026

Copy link
Copy Markdown

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedJun 18, 2026, 5:34 PM

@codecov

codecov Bot commented Jun 16, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 61.88925% with 234 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.21%. Comparing base (c1eab8e) to head (813881a).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
sei-db/ledger_db/block/blocksim/blocksim.go 20.71% 104 Missing and 7 partials ⚠️
sei-db/ledger_db/block/littblock/litt_block_db.go 69.37% 27 Missing and 22 partials ⚠️
...ei-db/ledger_db/block/blocksim/blocksim_metrics.go 0.00% 27 Missing ⚠️
...b/ledger_db/block/littblock/litt_block_iterator.go 64.44% 8 Missing and 8 partials ⚠️
sei-db/ledger_db/block/blocksim/blocksim_config.go 25.00% 12 Missing ⚠️
sei-db/ledger_db/block/blocksim/block_generator.go 89.28% 9 Missing ⚠️
...-db/ledger_db/block/littblock/litt_block_config.go 50.00% 4 Missing and 4 partials ⚠️
sei-db/ledger_db/block/memblock/mem_block_db.go 98.03% 1 Missing and 1 partial ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3600      +/-   ##
==========================================
- Coverage   59.02%   58.21%   -0.81%     
==========================================
  Files        2215     2146      -69     
  Lines      182530   174482    -8048     
==========================================
- Hits       107741   101581    -6160     
+ Misses      65092    63865    -1227     
+ Partials     9697     9036     -661     
Flag Coverage Δ
sei-chain-pr 69.88% <61.88%> (?)
sei-db 70.62% <ø> (+0.21%) ⬆️
sei-db-state-db ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
sei-db/db_engine/litt/table.go 100.00% <ø> (ø)
sei-db/db_engine/litt/table_config.go 100.00% <ø> (ø)
sei-db/ledger_db/block/littblock/codec.go 100.00% <100.00%> (ø)
sei-db/ledger_db/block/memblock/mem_block_db.go 98.03% <98.03%> (ø)
...-db/ledger_db/block/littblock/litt_block_config.go 50.00% <50.00%> (ø)
sei-db/ledger_db/block/blocksim/block_generator.go 80.64% <89.28%> (+80.64%) ⬆️
sei-db/ledger_db/block/blocksim/blocksim_config.go 41.53% <25.00%> (+41.53%) ⬆️
...b/ledger_db/block/littblock/litt_block_iterator.go 64.44% <64.44%> (ø)
...ei-db/ledger_db/block/blocksim/blocksim_metrics.go 0.00% <0.00%> (ø)
sei-db/ledger_db/block/littblock/litt_block_db.go 69.37% <69.37%> (ø)
... and 1 more

... and 79 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@cody-littley cody-littley changed the base branch from cjl/litt-iteration to main June 17, 2026 14:46
@cody-littley cody-littley marked this pull request as ready for review June 17, 2026 19:43
@cody-littley cody-littley requested review from arajasek and pompon0 June 17, 2026 19:43
@cody-littley cody-littley requested a review from wen-coding June 17, 2026 19:43
@cursor

cursor Bot commented Jun 17, 2026

Copy link
Copy Markdown

PR Summary

High Risk
Introduces the primary durable ledger path for finalized blocks and QCs with strict ordering and crash invariants; regressions could corrupt recovery or violate QC-before-block guarantees.

Overview
Replaces the legacy binary BlockDB (BinaryBlock, height/hash/tx lookups) with a consensus-oriented types.BlockDB in autobahn: ordered WriteQC then WriteBlock, iterators, PruneBefore, and documented crash rules (no gaps in write order; a persisted block always has a covering QC).

Adds a durable littblock implementation on LittDB: one ledger table with sharding factor 1, prefixed keys for blocks/QCs/hash aliases, write-order cursors recovered on reopen, TTL + GC watermark pruning, and ForceGC for tests. An in-memory memblock impl mirrors the same contract.

blocksim is rewired to generate FullCommitQC + block batches, supports mem and litt backends, resume after restart (reverse iterators + optional QC-ahead backfill), and batch-at-a-time shutdown semantics.

Litt Put/PutBatch docs now state that with sharding factor 1, crash durability preserves issue order across writes; a new segment test locks the single-shard contiguous prefix recovery property that littblock relies on.

Reviewed by Cursor Bugbot for commit 813881a. Bugbot is set up for automated code reviews on this repo. Configure here.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 4 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 1b0c8c8. Configure here.

Comment thread sei-db/ledger_db/block/littblock/litt_block_db.go
Comment thread sei-db/ledger_db/block/littblock/litt_block_db.go
Comment thread sei-db/ledger_db/block/blocksim/blocksim.go Outdated
Comment thread sei-db/ledger_db/block/memblock/mem_block_db.go

@wen-coding wen-coding left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mainly checked the block/QC contract and it looks good to me. I'll leave the review of implementation details to other reviewers.

// writes landed): drop the tail of its value file, then flip its metadata
// back to unsealed so LoadSegment re-runs recovery on reopen.
valPath, metaPath := largestValueSegmentFiles(t, dir)
truncateFileBy(t, valPath, 16)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I assume this assumes one block is always > 16 bytes (which is almost always true)

}
if qcsErr != nil {
return fmt.Errorf("failed to flush qcs table: %w", qcsErr)
if err := s.table.Flush(); err != nil {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we have a test which does: write -> Flush -> kill process -> reopen -> assert data is present?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants