🥮 mooncakes.io

moonkv

An append-only key-value storage engine in MoonBit based on Bitcask architecture.

kv
database
storage
bitcask
moon add wallll-wal6/moonkv@0.1.3
Download zip
Version
0.1.3
License
Apache-2.0
Repository
Last updated
3 days ago
Downloads
16

Dependencies

README

#MoonKV package guide

MoonKV is an embedded append-only key-value store implemented in MoonBit. It provides a small Bitcask-style storage engine with an in-memory key directory, checksummed records, logical TTL, atomic write batches, segment rolling, and hint-file assisted compaction recovery.

#Installation

From another MoonBit module:

moon add wallll-wal6/moonkv

Then import the root package:

import {
  "wallll-wal6/moonkv" @moonkv,
}

The package is published as wallll-wal6/moonkv, uses Apache-2.0, and is tested against MoonBit 0.10.3 in CI.

#Core API

let db = @moonkv.DB::open("my_db", 10 * 1024 * 1024)
db.put("user_name", @moonkv.string_to_bytes("Alice"))
let value = db.get("user_name")
println(@moonkv.bytes_to_string(value))
db.close()

The max_file_size argument must be positive. put, get, delete, put_ttl, merge, and WriteBatch::commit may raise MoonKVError.

#TTL and atomic batches

TTL is a logical write clock: a positive TTL expires after the corresponding number of later logical write steps, rather than after wall-clock seconds.

db.put_ttl("temporary", @moonkv.string_to_bytes("value"), 5L)
let batch = db.new_write_batch()
batch.put("balance", @moonkv.string_to_bytes("100"))
batch.delete("old_balance")
batch.commit()

put_ttl requires a positive TTL. A batch validates every operation before it writes the first transaction record, so an invalid TTL or a reserved key does not create a partial batch.

#Ordered queries and diagnostics

The query API is intended for cache indexes, local queues, and metadata tools:

///|
let page = db.scan_prefix("user:", 100)

///|
let next_page = db.scan_after("user:", page[page.length() - 1].0, 100)

///|
let range = db.scan_range("user:", "user;", 100)

///|
let page = db.scan_page("user:", "", 100)

///|
let stats = db.stats()

///|
let verified = db.verify()

///|
let fingerprint = db.fingerprint()

Scans are lexicographically ordered and omit deleted or logically expired keys. scan_after takes an exclusive cursor, while scan_range uses a half-open interval. scan_page returns rows, has_more, and an optional continuation cursor. Limits must be positive. delete_prefix uses one atomic write batch and is bounded by its limit. verify reads every visible value through the normal corruption checks. fingerprint is a deterministic FNV-1a smoke-check value, not a cryptographic digest.

#CLI

moon run cmd/main -- ./data_dir put mykey "Hello MoonBit"
moon run cmd/main -- ./data_dir get mykey
moon run cmd/main -- ./data_dir put-ttl token "temporary" 2
moon run cmd/main -- ./data_dir delete mykey
moon run cmd/main -- ./data_dir merge
moon run cmd/main -- ./data_dir scan user: 100
moon run cmd/main -- ./data_dir delete-prefix session: 100
moon run cmd/main -- ./data_dir stats
moon run cmd/main -- ./data_dir verify
moon run cmd/main -- ./data_dir fingerprint

The standalone benchmark uses one native process to write and read a deterministic workload:

bash scripts/benchmark.sh 10000
# Windows PowerShell:
.\scripts\benchmark.ps1 -Records 10000

#Key and value boundaries

Values are immutable Bytes; string_to_bytes and bytes_to_string are convenience helpers for simple text examples. Empty keys and empty values are valid. Keys beginning with __tx__: and __tx_commit__: are reserved for the transaction journal and are rejected. A database directory should have one writer at a time.

#Recovery and file format

Records contain a 28-byte header followed by the key and value. The checksum is verified while reading. A compaction writes live records to a data segment and a hint segment, removes obsolete files, and creates the next empty active data file before returning. This ensures that a new process writes after the hint segment rather than appending into it. The independent-process regression is:

bash scripts/cross-process-recovery.sh

#Test and quality gates

moon fmt --check
moon check --target all --deny-warn
moon build --target all
moon test --target all --deny-warn
moon info
git diff --exit-code

The CI matrix covers Linux, macOS, and Windows with MoonBit 0.10.3. See README.md, CHANGELOG.md, and CONTRIBUTING.md for the repository-level acceptance and maintenance information.

#
MoonKVError

source
pub suberror MoonKVError {
  IOError(String)
  KeyNotFound(String)
  CorruptedDatabase(String)
  InvalidRecord(String)
}

Custom error types for moonkv database operations.
pub struct DB {
  dir : String
  keydir : Keydir
  active_file_id : Int
  active_file_offset : Int
  max_file_size : Int
  logical_clock : Int64
}

Main database handle.

#
DB::close

source
fn DB::close(self : DB) -> Unit

Helper to close/clean resources (currently a no-op).

#
DB::contains

source
fn DB::contains(self : DB, key : String) -> Bool

Returns whether a key is currently visible.

#
DB::delete

source
fn DB::delete(self : DB, key : String) -> Unit raise MoonKVError

Deletes a key-value pair from the database by writing a tombstone.

#
DB::delete_prefix

source
fn DB::delete_prefix(self : DB, prefix : String, limit : Int) -> Int raise MoonKVError

Deletes up to limit visible keys with the given prefix atomically.

The prefix may be empty to clear the visible keyspace. The operation first collects matching keys, then commits one write batch, so an invalid limit cannot leave a partial deletion.

Callers can repeat the operation with the same prefix to drain large sets.

#
DB::fingerprint

source
fn DB::fingerprint(self : DB) -> Int raise MoonKVError

Computes a deterministic FNV-1a fingerprint of the visible keyspace.

The fingerprint is useful for smoke-checking replicas or detecting an unexpected change between maintenance runs. It is not a cryptographic digest and must not be used as an integrity or authentication boundary.

#
DB::get

source
fn DB::get(self : DB, key : String) -> Bytes raise MoonKVError

Retrieves a value from the database by its key.

#
DB::key_count

source
fn DB::key_count(self : DB) -> Int

Returns the number of currently indexed keys.

Expired entries remain lazily represented until accessed, so use verify or a scan when a fully materialized visible count is required.

#
DB::live_value_bytes

source
fn DB::live_value_bytes(self : DB) -> Int64

Returns the approximate bytes occupied by currently visible values.

This is metadata from the key directory, so it excludes record headers and stale log records. It is intended for admission control and dashboards.

Values are measured in bytes and remain exact for the current keydir view.

#
DB::merge

source
fn DB::merge(self : DB) -> Unit raise MoonKVError

Writes consolidated active records to a new data and hint file, and deletes older files.

#
DB::new_write_batch

source
fn DB::new_write_batch(self : DB) -> WriteBatch

Creates a new WriteBatch for the database.

#
DB::open

source
fn DB::open(dir : String, max_file_size : Int) -> DB raise MoonKVError

Rebuilds the in-memory index from existing data files.

#
DB::put

source
fn DB::put(self : DB, key : String, value : Bytes) -> Unit raise MoonKVError

Writes a key-value pair to the database.

#
DB::put_ttl

source
fn DB::put_ttl(self : DB, key : String, value : Bytes, ttl : Int64) -> Unit raise MoonKVError

Writes a key-value pair to the database with a logical Time-To-Live (TTL).

#
DB::scan_after

source
fn DB::scan_after(self : DB, prefix : String, cursor : String, limit : Int) -> Array[(String, Bytes)] raise MoonKVError

Returns at most limit keys with prefix after an exclusive cursor.

The returned keys are sorted lexicographically. Pass the last returned key as cursor to fetch the next page. An empty cursor starts at the beginning.

#
DB::scan_page

source
fn DB::scan_page(self : DB, prefix : String, cursor : String, limit : Int) -> ScanPage raise MoonKVError

Returns one page and an exclusive cursor for the following page.

Unlike inferring pagination from a short result array, has_more is determined by reading one sentinel row. This makes an exact page at the end of the keyspace unambiguous to HTTP, CLI, and replication adapters.

#
DB::scan_prefix

source
fn DB::scan_prefix(self : DB, prefix : String, limit : Int) -> Array[(String, Bytes)] raise MoonKVError

A stable, ordered view of the keys currently visible in the database.

Scans are built from the in-memory key directory and then read through the normal record validation path. They therefore never expose transaction markers, tombstones, or partially written records.

#
DB::scan_range

source
fn DB::scan_range(self : DB, start_key : String, end_key : String, limit : Int) -> Array[(String, Bytes)] raise MoonKVError

Returns keys in the half-open range [start_key, end_key).

#
DB::stats

source
fn DB::stats(self : DB) -> DBStats

Returns storage metadata without reading every value from disk. This is safe to call from periodic monitoring loops. The values describe the current process-local view. Reopen the database to refresh it after an external writer.

#
DB::verify

source
fn DB::verify(self : DB) -> Int raise MoonKVError

Reads every visible key and returns the number successfully verified.

This is intended for startup diagnostics and maintenance tooling. It uses the same bounds and checksum validation as get, so a corrupted value is reported instead of being silently skipped.

#
DBStats

source
pub struct DBStats {
  key_count : Int
  live_value_bytes : Int64
  active_file_id : Int
  active_file_offset : Int
  max_file_size : Int
  logical_clock : Int64
}

A compact health and capacity snapshot for monitoring integrations.

#
HintRecord

source
pub struct HintRecord {
  timestamp : Int64
  expires_at : Int64
  key_sz : Int
  val_sz : Int
  value_pos : Int64
  key : Bytes
}

Container for metadata read from a Hint File.

#
KeyEntry

source
pub struct KeyEntry {
  file_id : Int
  value_sz : Int
  value_pos : Int64
  timestamp : Int64
  expires_at : Int64
}

KeyEntry metadata representing a record's location on disk.

#
KeyEntry::new

source
fn KeyEntry::new(file_id : Int, value_sz : Int, value_pos : Int64, timestamp : Int64, expires_at : Int64) -> KeyEntry

Creates a new KeyEntry.

#
Keydir

source
pub struct Keydir {
  entries : Map[String, KeyEntry]
}

In-memory key directory mapping keys to their metadata.

#
Keydir::clear

source
fn Keydir::clear(self : Keydir) -> Unit

Clears all entries from the directory.

#
Keydir::contains

source
fn Keydir::contains(self : Keydir, key : String) -> Bool

Checks if the directory contains a key.

#
Keydir::delete

source
fn Keydir::delete(self : Keydir, key : String) -> Unit

Removes a key from the directory.

#
Keydir::get

source
fn Keydir::get(self : Keydir, key : String) -> KeyEntry?

Retrieves a KeyEntry from the directory.

#
Keydir::keys

source
fn Keydir::keys(self : Keydir) -> Array[String]

Returns all keys in the directory.

#
Keydir::new

source
fn Keydir::new() -> Keydir

Creates a new Keydir.

#
Keydir::put

source
fn Keydir::put(self : Keydir, key : String, entry : KeyEntry) -> Unit

Inserts a KeyEntry into the directory.

#
LogRecord

source
pub struct LogRecord {
  record : Record
  offset : Int
  size : Int
}

Container for a record retrieved from a WAL data file.

#
Record

source
pub struct Record {
  checksum : Int
  timestamp : Int64
  expires_at : Int64
  key : Bytes
  value : Bytes
  is_tombstone : Bool
}

Represents a single key-value record stored on disk.

#
Record::new

source
fn Record::new(key : Bytes, value : Bytes, timestamp : Int64) -> Record

Creates a new active Record.

#
Record::new_tombstone

source
fn Record::new_tombstone(key : Bytes, timestamp : Int64) -> Record

Creates a new tombstone (deleted) Record.

#
Record::new_ttl

source
fn Record::new_ttl(key : Bytes, value : Bytes, timestamp : Int64, ttl : Int64) -> Record

Creates a new active Record with a Time-To-Live (expires at timestamp + ttl).

#
Record::serialize

source
fn Record::serialize(self : Record) -> Bytes

Serializes the Record to its on-disk binary representation.

#
ScanPage

source
pub struct ScanPage {
  rows : Array[(String, Bytes)]
  next_cursor : String?
  has_more : Bool
}

A page of ordered scan results for clients that need explicit continuation.

#
WriteBatch

source
pub struct WriteBatch {
  db : DB
  ops : Array[(String, Bytes?, Int64, Bool)]
}

WriteBatch stores a sequence of database operations to be committed atomically.

#
WriteBatch::commit

source
fn WriteBatch::commit(self : WriteBatch) -> Unit raise MoonKVError

Commits all operations in the batch atomically to the WAL log.

#
WriteBatch::delete

source
fn WriteBatch::delete(self : WriteBatch, key : String) -> Unit

Adds a Delete operation to the batch.

#
WriteBatch::put

source
fn WriteBatch::put(self : WriteBatch, key : String, value : Bytes) -> Unit

Adds a Put operation to the batch.

#
WriteBatch::put_ttl

source
fn WriteBatch::put_ttl(self : WriteBatch, key : String, value : Bytes, ttl : Int64) -> Unit

Adds a Put operation with TTL to the batch.

#
append_bytes_to_file

source
fn append_bytes_to_file(path : String, content : Bytes) -> Unit raise MoonKVError

Appends a Bytes payload to the file at path for Native target.

#
bytes_to_string

source
fn bytes_to_string(bytes : Bytes) -> String

Converts UTF-8/ASCII Bytes to a String.

#
decode_header

source
fn decode_header(header : Bytes) -> (Int, Int64, Int64, Int, Int) raise MoonKVError

Returns (checksum, timestamp, expires_at, key_sz, val_sz).

#
deserialize_record

source
fn deserialize_record(bytes : Bytes) -> Record raise MoonKVError

Deserializes a complete Record from raw bytes, verifying its checksum.

#
fnv1a

source
fn fnv1a(bytes : Array[Byte], offset : Int, len : Int) -> Int

Computes the hash for a slice of bytes from offset to offset + len.

#
fnv1a_bytes

source
fn fnv1a_bytes(bytes : Bytes, offset : Int, len : Int) -> Int

Computes the hash for a slice of bytes from offset to offset + len.

#
get_int32

source
fn get_int32(bytes : Bytes, offset : Int) -> Int

Reads a 32-bit signed integer from Bytes starting at offset in Big-Endian format.

#
get_int64

source
fn get_int64(bytes : Bytes, offset : Int) -> Int64

Reads a 64-bit signed integer from Bytes starting at offset in Big-Endian format.

#
merge_bytes

source
fn merge_bytes(b1 : Bytes, b2 : Bytes) -> Bytes

Merges two Bytes objects into a single new Bytes object.

#
read_hint_records_from_file

source
fn read_hint_records_from_file(path : String) -> Array[HintRecord] raise MoonKVError

Reads and parses all hint records from a Hint File.

#
read_records_from_file

source
fn read_records_from_file(path : String) -> Array[LogRecord] raise MoonKVError

Sequentially reads and parses all records from a WAL data file.

#
serialize_hint_record

source
fn serialize_hint_record(timestamp : Int64, expires_at : Int64, key_sz : Int, val_sz : Int, value_pos : Int64, key : Bytes) -> Bytes

Layout: timestamp (8B) | expires_at (8B) | key_sz (4B) | val_sz (4B) | value_pos (8B) | key

#
set_int32

source
fn set_int32(bytes : Array[Byte], offset : Int, value : Int) -> Unit

Writes a 32-bit signed integer into an Array[Byte] starting at offset in Big-Endian format.

#
set_int64

source
fn set_int64(bytes : Array[Byte], offset : Int, value : Int64) -> Unit

Writes a 64-bit signed integer into an Array[Byte] starting at offset in Big-Endian format.

#
string_to_bytes

source
fn string_to_bytes(s : String) -> Bytes

Converts a String to UTF-8/ASCII Bytes.

Source Files

Powered by MoonBit

Site sourceReport issuePackagesBuild queueSkillsStatistics

© 2026 mooncakes.io