🥮 mooncakes.io
README

#core/persistence/wal

Write-Ahead Log (WAL) for crash recovery.

#Format

Header (12 bytes):
  - Magic: "VCDBMBT\0" (8 bytes)
  - Type: 1 byte
  - Version: 1 byte
  - Reserved: 2 bytes

Records:
  - Type: 1 byte (Upsert/Remove/SetAttrs)
  - Reserved: 1 byte
  - ID: 8 bytes
  - AttrsLen: 4 bytes
  - VectorLen: 4 bytes
  - Attrs: JSON UTF-8
  - Vector: Float32 array

Footer (8 bytes, optional):
  - Magic: "WCRC" (4 bytes)
  - CRC32: 4 bytes

#Record Types

  • Upsert: Insert or update vector with attrs
  • Remove: Delete vector
  • SetAttrs: Update attrs only

#Usage

let writer = WalWriter::new()
writer.write_upsert(id, vector, attrs)
let bytes = writer.finalize()

#
AsyncWalRuntime

source
pub struct AsyncWalRuntime[S] {
  storage : S
  path : String
  kind : 
StorageKind
  data : Bytes
  record_count : Int
  loaded : Bool
}

#
AsyncWalRuntime::append

source

Append records to the WAL. Merges in-memory (no read from storage), then writes to storage. Delegates merge logic to format.mbt's merge_wal.

#
AsyncWalRuntime::async_exists

source

Check if WAL exists on storage.

#
AsyncWalRuntime::byte_size

source
fn[S] AsyncWalRuntime::byte_size(self : AsyncWalRuntime[S]) -> Int

Current WAL data size in bytes (for byte-size checkpoint decisions).

#
AsyncWalRuntime::is_loaded

source
fn[S] AsyncWalRuntime::is_loaded(self : AsyncWalRuntime[S]) -> Bool

Whether load() has been called.

#
AsyncWalRuntime::load

source

Load WAL data from storage into the in-memory buffer. Must be called once before append/truncate. Idempotent. Returns the WAL data bytes (for callers that need it, e.g., replay).

#
AsyncWalRuntime::new

source
fn[S] AsyncWalRuntime::new(storage : S, path : String, kind? : 
StorageKind) -> AsyncWalRuntime[S]

#
AsyncWalRuntime::path

source
fn[S] AsyncWalRuntime::path(self : AsyncWalRuntime[S]) -> String

Get the WAL file path.

#
AsyncWalRuntime::record_count

source
fn[S] AsyncWalRuntime::record_count(self : AsyncWalRuntime[S]) -> Int

Number of records in the current WAL buffer (since last truncate).

#
AsyncWalRuntime::replay_into

source

Replay WAL records into a CoreStore. Returns number of records applied. Delegates replay logic to replay_wal_data in runtime.mbt.

Uses in-memory buffer if loaded, otherwise reads from storage.

#
AsyncWalRuntime::truncate

source

Truncate the WAL — reset in-memory buffer and write empty header.

#
WalRecord

source
pub struct WalRecord {
  record_type : WalRecordType
  id : 
VectorId
  attrs : 
Attrs?
  vector : Array[Double]?
  timestamp : Int64
} derive(
Debug)

WAL record with timestamp
impl Show for WalRecord

#
WalRecord::remove

source
fn WalRecord::remove(id : 
VectorId, timestamp? : Int64) -> WalRecord

Create a remove record with timestamp

#
WalRecord::set_attrs

source

Create a set_attrs record with timestamp

#
WalRecord::upsert

source
fn WalRecord::upsert(id : 
VectorId, vector : Array[Double], attrs : 
Attrs, timestamp? : Int64) -> WalRecord

Create an upsert record with timestamp

#
WalRecordType

source
pub enum WalRecordType {
  Upsert
  Remove
  SetAttrs
} derive(Eq, 
Debug)

WAL record types

#
WalRecordType::from_byte

source
fn WalRecordType::from_byte(b : Byte) -> WalRecordType?

#
WalRecordType::to_byte

source
fn WalRecordType::to_byte(self : WalRecordType) -> Byte

#
decode_wal_record

source

Decode a WAL record from reader (public entry point — assumes v3 format). For reading v1/v2 WAL files, use decode_wal_records which handles versioning.

#
decode_wal_records

source
fn decode_wal_records(data : Bytes) -> Array[WalRecord]

Decode all records from WAL data (supports v1, v2, and v3)
fn encode_wal_footer(body : Bytes) -> Bytes

Encode WAL footer with CRC32

#
encode_wal_header

source
fn encode_wal_header() -> Bytes

Encode WAL header (12 bytes)

#
encode_wal_record

source
fn encode_wal_record(record : WalRecord) -> Bytes

Encode a WAL record to bytes (v2 format with timestamp)

#
encode_wal_segment

source
fn encode_wal_segment(records : Array[WalRecord]) -> Bytes

Encode multiple records to a complete WAL segment

#
filter_wal_by_timestamp

source
fn filter_wal_by_timestamp(data : Bytes, cutoff_ts : Int64) -> Array[WalRecord]

Filter WAL records by timestamp — returns only records with timestamp <= cutoff.

#
max_wal_timestamp

source
fn max_wal_timestamp(data : Bytes) -> Int64

Get the maximum timestamp across all records in a WAL. Returns 0 if no timestamped records.

#
merge_wal

source
fn merge_wal(existing : Bytes, new_segment : Bytes) -> Bytes

Merge an existing WAL with a new WAL segment.

#
replay_wal_data

source
fn replay_wal_data(data : Bytes, store : 
CoreStore, cutoff_ts? : Int64) -> Int

Replay WAL binary data into a CoreStore (no Storage dependency). Used by:
  • AsyncWalRuntime::replay_into (reads from AsyncStorage, then delegates here)
  • JS persistent exports (passes WAL bytes directly)

Returns the number of records applied. Returns 0 for empty/invalid data.

#
verify_wal_checksum

source
fn verify_wal_checksum(data : Bytes) -> Bool

Check if WAL has valid footer and verify CRC32

#
verify_wal_header

source
fn verify_wal_header(r : 
BinaryReader) -> Bool

Verify WAL header (accepts v1, v2, and v3). Returns the version byte on success.
let wal_footer_magic : UInt

WAL Footer magic: "WCRC" in little-endian

#
wal_records_for_replay

source
fn wal_records_for_replay(data : Bytes, cutoff_ts? : Int64) -> Array[WalRecord]

Decode replayable WAL records after validating header and checksum.

#
wal_version

source
let wal_version : Byte

Current WAL format version

Source Files

Powered by MoonBit

Site sourceReport issuePackagesBuild queueSkillsStatistics

© 2026 mooncakes.io