🥮 mooncakes.io

vcdb

High-performance vector database with multiple ANN algorithms

vector-database
ann
hnsw
ivf
similarity-search
moon add trkbt10/vcdb@0.3.2
Download zip
Author
Version
0.3.2
License
Apache-2.0
Repository
Last updated
28 days ago
Downloads
552

Dependencies

README

#vcdb

High-performance vector database with multiple ANN algorithms for MoonBit.

#Features

  • Multiple ANN Algorithms: HNSW, IVF, and Brute-force search
  • Flexible Storage: Pluggable backends (memory, S3)
  • Persistence: WAL-based durability with segment management
  • Attribute Filtering: Metadata-based vector filtering
  • Gateway Execution Layer: API handling reusable across native and JS runtimes

#Overview

vcdb is structured as a layered architecture:

#Core Layers

PackagePurpose
core/annANN algorithms (HNSW, IVF, Bruteforce)
core/storeVector data storage
core/storageAbstract storage interface
core/persistenceWAL and segment management
core/attrAttribute indexing and filtering

#API Layers

PackagePurpose
gatewayTransport-agnostic API execution core
httpHTTP transport adapter over gateway
jsJS runtime adapter, npm distribution, and HTTP server
cmd/native-gatewayNative gateway execution entrypoint
cmd/native-serveNative HTTP server over gateway
cliCommand-line interface
libLibrary entry point

#Getting Started

// Create a collection with HNSW index
let store = @vcdb.CoreStore::new()
let collection = store.create_collection("my_vectors", dim=128, ann_type=HNSW)

// Add vectors with optional attributes
collection.upsert([
  { id: "vec1", vector: [...], attrs: { "category": "A" } },
  { id: "vec2", vector: [...], attrs: { "category": "B" } },
])

// Search with filtering
let results = collection.search(
  query_vector,
  top_k=10,
  filter={ "category": "A" }
)

#Usage

#Native Gateway Execution

Execute gateway requests directly from MoonBit:

moon run cmd/native-gateway -- healthz
moon run cmd/native-gateway -- collections create demo --dim 3

#HTTP Serving via JS Adapter

Start the JavaScript transport adapter:

cd js
npm run build
node dist/server.js --host 127.0.0.1 --port 6333 --storage ../.local-storage

#HTTP Serving via Native Adapter

Start the MoonBit native transport adapter:

moon run cmd/native-serve -- --host 127.0.0.1 --port 6333

#Create Collection

curl -X POST http://localhost:8080/collections/my_collection \
  -H "Content-Type: application/json" \
  -d '{"dim": 128, "ann_type": "hnsw"}'

#Upsert Vectors

curl -X PUT http://localhost:8080/collections/my_collection/points \
  -H "Content-Type: application/json" \
  -d '{"points": [{"id": "1", "vector": [...]}]}'

#Search

curl -X POST http://localhost:8080/collections/my_collection/search \
  -H "Content-Type: application/json" \
  -d '{"vector": [...], "top_k": 10}'

#Installation

#From Package Manager

moon add trkbt10/vcdb

#From Source

git clone https://github.com/trkbt10/vcdb_mbt
cd vcdb_mbt
moon build

#Requirements

  • MoonBit toolchain (moon >= 0.1.0)

gateway is the API execution core. Long-lived HTTP serving is now available in both the JS adapter and the native async/http adapter.

#License

See LICENSE for details.

#
PersistentDB

source
pub struct PersistentDB[W, S] {
  engine : VectorDB
  wal : 
AsyncWalRuntime[W]
  snapshot_storage : S
  base_path : String
  name : String
  checkpoint_threshold : Int
  checkpoint_bytes : Int
}

#
PersistentDB::add

source

Add a single vector. Fails if the ID already exists (not tombstoned).

This is the WAL-backed equivalent of VectorDB::add. Existence check is performed BEFORE WAL append to prevent orphaned WAL records when the ID already exists.

#
PersistentDB::checkpoint

source

Checkpoint: snapshot to snapshot_storage, then truncate WAL. Crash safe: snapshot-first, WAL truncate second.

#
PersistentDB::compact

source

Compact HNSW index (removes tombstones). Returns removed count. Triggers checkpoint after compaction.

#
PersistentDB::count_filtered

source
fn[W, S] PersistentDB::count_filtered(self : PersistentDB[W, S], expr? : 
FilterExpr?) -> Int

Count vectors matching filter.

#
PersistentDB::deserialize

source

Deserialize from snapshot bytes, wrapping with in-memory storage.

Used for loading a serialized database without persistence backends.

#
PersistentDB::dim

source
fn[W, S] PersistentDB::dim(self : PersistentDB[W, S]) -> Int

Vector dimension.

#
PersistentDB::find

source

Find the single best match.

#
PersistentDB::from_snapshot

source
fn[S] PersistentDB::from_snapshot(data : Bytes, storage : S, base_path : String, name : String, checkpoint_threshold? : Int, checkpoint_bytes? : Int) -> PersistentDB[S, S]

Load from snapshot bytes with explicit storage backends.

Used by gateway's load_collection to restore from a serialized snapshot and attach real storage backends for subsequent WAL operations.

#
PersistentDB::get

source

Get a single vector by ID.

#
PersistentDB::has

source
fn[W, S] PersistentDB::has(self : PersistentDB[W, S], id : 
VectorId) -> Bool

Check if a vector exists.

#
PersistentDB::in_memory

source

Create an in-memory PersistentDB (no durable persistence).

Uses MemoryStorage for both WAL and snapshot backends. Since MemoryStorage resolves all I/O synchronously, initialization completes immediately — but the function is async to maintain a uniform interface.

#
PersistentDB::in_memory_bruteforce

source

Create an in-memory PersistentDB with Bruteforce strategy.

#
PersistentDB::in_memory_hnsw

source

Create an in-memory PersistentDB with HNSW strategy.

#
PersistentDB::in_memory_ivf

source

Create an in-memory PersistentDB with IVF strategy.

#
PersistentDB::init

source
async fn[W : 
AsyncStorage, S : 
AsyncStorage] PersistentDB::init(wal_storage : W, snapshot_storage : S, base_path : String, name : String, dim : Int, capacity : Int, metric? : 
Metric, strategy? : 
Strategy, checkpoint_threshold? : Int, checkpoint_bytes? : Int) -> PersistentDB[W, S]

Initialize from storage: load WAL + snapshot, replay, build engine.

Call sequence:
  1. Load snapshot from snapshot_storage
  2. Load WAL from wal_storage into in-memory buffer
  3. Deserialize snapshot -> CoreStore (or create empty)
  4. Replay WAL records onto CoreStore
  5. Build VectorDB from CoreStore

#
PersistentDB::metric

source

Similarity metric.

#
PersistentDB::raw_size

source
fn[W, S] PersistentDB::raw_size(self : PersistentDB[W, S]) -> Int

Raw size including tombstones.

#
PersistentDB::remove

source

Remove a vector with WAL-before-state guarantee.

#
PersistentDB::scroll

source

Scroll through vectors in ascending ID order.

#
PersistentDB::scroll_filtered

source

Scroll with filter expression.

#
PersistentDB::search

source

Search by vector similarity with optional filter expression.

#
PersistentDB::search_with_expr

source

Search with explicit filter expression, index override, and strategy.

#
PersistentDB::search_with_filter

source

Search by vector similarity with callback filter.

#
PersistentDB::serialize_snapshot

source
fn[W, S] PersistentDB::serialize_snapshot(self : PersistentDB[W, S]) -> Bytes

Serialize current state as snapshot bytes.

#
PersistentDB::size

source
fn[W, S] PersistentDB::size(self : PersistentDB[W, S]) -> Int

Database size (excluding tombstones).

#
PersistentDB::store

source

Access the underlying CoreStore (for advanced/distributed use).

#
PersistentDB::strategy

source

ANN strategy.

#
PersistentDB::train

source
fn[W, S] PersistentDB::train(self : PersistentDB[W, S], iterations? : Int) -> Unit

Train IVF index (IVF strategy only).

#
PersistentDB::update_attrs

source

Update attributes with WAL-before-state guarantee.

#
PersistentDB::upsert

source

Upsert points with WAL-before-state guarantee.

Order:
  1. Encode WAL records (sync)
  2. Persist WAL to storage (async, awaited)
  3. Update engine in-memory (sync)
  4. Auto-checkpoint if thresholds exceeded

#
PersistentDB::wal_byte_size

source
fn[W, S] PersistentDB::wal_byte_size(self : PersistentDB[W, S]) -> Int

Current WAL byte size (for diagnostics).

#
PersistentDB::wal_record_count

source
fn[W, S] PersistentDB::wal_record_count(self : PersistentDB[W, S]) -> Int

Current WAL record count (for diagnostics).

#
VectorDB

source

VectorDB - Internal ANN index engine.

Provides unified access to all ANN strategies: Bruteforce, HNSW, and IVF. The engine automatically handles:
  • Vector normalization for cosine similarity
  • Index updates on add/remove operations
  • Tombstone management for HNSW deletions

This is an internal implementation detail. The public API is VectorDB[W, S].

#
VectorDB::add

source

Add a new vector to the database.

For HNSW mode, allows re-adding tombstoned IDs (clears tombstone and re-indexes). Fails if the ID already exists and is not tombstoned.

Parameters:
  • id: Unique identifier for the vector
  • vector: The embedding vector (must match database dimension)
  • attrs: Metadata attributes for filtering

#
VectorDB::compact

source
fn VectorDB::compact(self : VectorDB) -> (VectorDB, Int)

Compact the database by removing tombstoned vectors (HNSW only).

HNSW uses soft-delete (tombstones) for remove(). Over time, tombstones accumulate and waste memory. compact() rebuilds the HNSW graph without tombstoned entries.

For Bruteforce and IVF, this is a no-op (they physically remove).

Returns: A new VectorDB with the compacted state, and the number of removed tombstones.

#
VectorDB::count_filtered

source
fn VectorDB::count_filtered(self : VectorDB, expr? : 
FilterExpr?) -> Int

Count vectors matching an optional filter expression.

When expr is None, counts all non-tombstoned vectors. When expr is Some, resolves the expression to a candidate set via the B+ tree index, then subtracts tombstoned entries. No vector data is touched.

Parameters:
  • expr: Optional filter expression to count

Returns: Number of matching, non-tombstoned vectors

#
VectorDB::deserialize

source
fn VectorDB::deserialize(data : Bytes) -> VectorDB

Deserialize a database from bytes.

Restores all vectors, attributes, metadata index state, and ANN index state.

Parameters:
  • data: Binary data from serialize()

Returns: The restored VectorDB instance

#
VectorDB::dim

source
fn VectorDB::dim(self : VectorDB) -> Int

Get the dimension of vectors in this database.

#
VectorDB::export_jsonl

source
fn VectorDB::export_jsonl(self : VectorDB) -> Array[String]

Export all points as JSONL lines.

#
VectorDB::find

source

Find the single best match for a query.

Convenience method equivalent to search(query, 1, filter).

Parameters:
  • query: The query vector
  • filter: Optional filter predicate

Returns: The best matching SearchHit, or None if no matches

#
VectorDB::from_store

source

Create VectorDB from an existing CoreStore (for loading from distributed storage)

#
VectorDB::get

source

Get a vector and its attributes by ID.

For HNSW mode, tombstoned IDs return None.

Parameters:
  • id: The vector ID to retrieve

Returns: The vector record if found and accessible, None otherwise

#
VectorDB::has

source

Check if an id exists and is accessible.

For HNSW mode, tombstoned IDs are considered non-existent.

Parameters:
  • id: The vector ID to check

Returns: true if the vector exists and is not tombstoned

#
VectorDB::metric

source

Get the similarity metric used by this database.

#
VectorDB::new

source

Create a new VectorDB with the given options.

Parameters:
  • options: Configuration including dimension, metric, capacity, and strategy

Returns: A new VectorDB instance configured according to options

#
VectorDB::raw_size

source
fn VectorDB::raw_size(self : VectorDB) -> Int

Get the raw size including tombstoned vectors. This is primarily for internal use and serialization.

#
VectorDB::remove

source

Remove a vector by ID.

Behavior differs by strategy:
  • Bruteforce/IVF: Physically removes the vector
  • HNSW: Marks as tombstone (soft delete) to maintain graph connectivity

For HNSW, use compact() to reclaim space from tombstoned vectors.

Parameters:
  • id: The vector ID to remove

Returns: true if the vector was removed, false if not found or already deleted

#
VectorDB::scroll

source

Scroll points in ascending ID order, similar to Qdrant scroll.

Thin wrapper over scroll_filtered with no filter expression. Offset semantics: returns entries with ID strictly greater than offset. Pass the last ID from previous page as offset to get the next page.

#
VectorDB::scroll_filtered

source

Scroll points in ascending ID order with optional filter expression.

When expr is None, returns all (non-tombstoned) entries. When expr is Some, resolves candidates via the B+ tree attribute index — every expression variant is handled by the index with a single execution path.

Offset semantics: returns entries with ID strictly greater than offset (cursor = last ID seen by caller).

Parameters:
  • expr: Optional filter expression to match against attributes
  • offset: Cursor — only IDs strictly greater than this value are returned
  • limit: Maximum number of entries to return

Returns: Array of (VectorId, VectorRecord) sorted by ID ascending

#
VectorDB::search

source

Search for k nearest neighbors to a query vector.

Parameters:
  • query: The query vector (must match database dimension)
  • k: Number of results to return
  • filter: Optional predicate to filter results by ID and attributes

Returns: Array of SearchHit sorted by score (highest/most similar first)

#
VectorDB::search_with_expr

source

Search with filter expressions and metadata index support.

Supports two filtering strategies:
  • PreFilter: Use attr index to get candidate IDs, then score only those
  • PostFilter: Score all vectors, then filter results
  • Auto: Automatically choose based on estimated selectivity

PreFilter is more efficient when the filter is selective (<50% of vectors). PostFilter is better for broad filters or when no attr index is available.

Parameters:
  • query: The query vector
  • k: Number of results to return
  • expr: Optional filter expression (Must/MustNot/Should/Leaf)
  • attr_index: Optional B+ tree index override for pre-filtering
  • strategy: Filtering strategy (PreFilter/PostFilter/Auto)

Returns: Array of SearchHit matching the filter criteria

#
VectorDB::serialize

source
fn VectorDB::serialize(self : VectorDB) -> Bytes

Serialize the database to bytes for persistence.

Includes all vectors, attributes, metadata index state, and ANN index state. Use deserialize() to restore.

Returns: Binary representation of the database

#
VectorDB::size

source
fn VectorDB::size(self : VectorDB) -> Int

Get the number of accessible vectors in the database.

For HNSW, this excludes tombstoned (deleted) vectors. Use raw_size() to get the total count including tombstones.

#
VectorDB::store

source

Get access to the underlying CoreStore (for distributed storage operations)

#
VectorDB::strategy

source

Get the ANN strategy used by this database.

#
VectorDB::train

source
fn VectorDB::train(self : VectorDB, iterations? : Int) -> Unit

Train the IVF index for optimal search performance.

Only applicable for IVF strategy. Call this after adding a representative sample of vectors to build the centroid clusters.

Parameters:
  • iterations: Number of k-means iterations (default: 10)

#
VectorDB::update_attrs

source

Update only the attributes for a vector (without changing the embedding).

Parameters:
  • id: The vector ID to update
  • attrs: New attributes to set

Returns: true if updated, false if ID not found

#
VectorDB::upsert

source

Add or update a vector (upsert operation).

If the ID exists, updates the vector and reindexes. If the ID doesn't exist, adds a new vector.

Parameters:
  • id: Unique identifier for the vector
  • vector: The embedding vector
  • attrs: Metadata attributes

Returns: true if a new vector was created, false if existing was updated

#
VectorDB::with_dim

source
fn VectorDB::with_dim(dim : Int) -> VectorDB

Create a VectorDB with default options (Bruteforce strategy).

Parameters:
  • dim: Vector dimension

Returns: A VectorDB with Bruteforce strategy and Cosine metric

#
VectorDB::with_hnsw

source
fn VectorDB::with_hnsw(dim : Int, metric? : 
Metric) -> VectorDB

Create a VectorDB with HNSW strategy.

HNSW (Hierarchical Navigable Small World) is recommended for most use cases. It provides O(log n) search time with high recall.

Parameters:
  • dim: Vector dimension
  • metric: Similarity metric (default: Cosine)

Returns: A VectorDB configured with HNSW index

#
VectorDB::with_ivf

source
fn VectorDB::with_ivf(dim : Int, metric? : 
Metric) -> VectorDB

Create a VectorDB with IVF strategy.

IVF (Inverted File Index) is best for very large datasets. Requires training with train() before optimal search performance.

Parameters:
  • dim: Vector dimension
  • metric: Similarity metric (default: Cosine)

Returns: A VectorDB configured with IVF index

#
empty_attrs

source

Re-export factory functions for convenience

Source Files

Powered by MoonBit

Site sourceReport issuePackagesBuild queueSkillsStatistics

© 2026 mooncakes.io