#MoonSearch

#Installation

moon add Lucius646/MoonSearch@0.1.0

#Package architecture

core ──→ schema ──→ analysis ──→ index ──→ query ──→ store
  │         │           │           │          │         │
  │         │           │           │          │         └─ Directory,
  │         │           │           │          │            IndexReader/Writer
  │         │           │           │          └─ Query, Weight, Scorer,
  │         │           │           │             Searcher, Collector
  │         │           │           └─ Segment, postings, codec, Merge
  │         │           └─ Tokenizer, TokenStream, filters, registry
  │         └─ field definitions and SchemaBuilder
  └─ identifiers, documents, terms, shared errors

internal/codec ──→ shared binary container primitives used by index and store
analysis/chinese ──→ optional dictionary-backed Chinese analysis over analysis
tests/integration ──→ public cross-package behavior through the root facade

#M5b-1

Schema tokenizer name ──→ TokenizerManager
                               │
raw text ──→ Tokenizer ──→ TokenStream ──→ TokenFilter[] ──→ analyzed Tokens
                               │
                ┌──────────────┴──────────────┐
                ▼                             ▼
        SegmentWriter indexing       QueryParser analysis
                                              │
                                   existing Query hierarchy

• documents containing text fields;
• open Tokenizer, stateful TokenStream, and lazy TokenFilter boundaries;
• third-party Tokenizers that can return their own eager or lazy TokenStream implementations;
• composable TextAnalyzer pipelines that implement Tokenizer themselves;
• name-based TokenizerManager registration and replacement;
• built-in raw, Unicode-whitespace, punctuation-aware default, lowercase, and maximum-token-length components;
• configurable character NgramTokenizer with all-gram and prefix-only modes;
• Tantivy-compatible N-gram positions: every gram from one field value has position zero while UTF-8 byte offsets preserve its source span;
• token positions, position lengths, and UTF-8 byte offsets;
• schema-bound tokenizer names with legacy TextOptions::new retaining whitespace behavior;
• schema-aware indexing that resolves each field through TokenizerManager and consumes the resulting TokenStream directly;
• field-aware query-time analysis for Term, Boolean, and Phrase semantics;
• disjunction-by-default and optional conjunction query construction;
• phrase position-gap preservation after filtering;
• Segment format v2 persistence of tokenizer names and compatibility with complete M3/M4 Segment v1 files;
• field-qualified terms;
• posting lists containing document ID, term frequency, and positions;
• completed, read-only immutable segments;
• immutable snapshots of stored text fields;
• per-document field lengths and per-field collection statistics;
• stable field allocation through SchemaBuilder;
• independent indexed and stored text options;
• exact field-qualified term queries;
• BM25 ranking with k1=1.2 and b=0.75;
• Boolean Must, Should, and MustNot clauses;
• exact zero-slop phrase queries using positions;
• score multiplication through BoostQuery;
• multi-field weighting by composing Boolean and Boost queries;
• position gaps between repeated values of the same field;
• deterministic Top-K results;
• DocAddress values ready for later multi-segment search;
• stored-document lookup from a stable single-segment Searcher.
• a byte-oriented open Directory boundary;
• copy-isolated MemoryDirectory storage;
• filesystem-backed FsDirectory storage;
• incremental IndexWriter commits using numbered Segment files;
• a deterministic, versioned and checksummed manifest;
• stable IndexReader generations and Searcher snapshots;
• snapshot-wide document frequency, field document count, and average field length;
• one query Weight producing comparable Scorers for every Segment;
• global Top-K selection with stable Segment ordinal and DocId tie-breaking;
• IndexReader segment-count and generation inspection;
• exact field-qualified IndexWriter.delete_term(Term) operations;
• explicit commit_deletes() publication through a new manifest generation;
• tombstones stored per Segment in deterministic manifest v2 data;
• backward-compatible loading of M4a manifest v1;
• live-document filtering for query hits and stored-document lookup;
• live-only document frequency, field document count, average field length, and BM25 scoring;
• replacement semantics: pending deletes affect existing Segments while a Segment appended by the same commit remains live;
• native Segment Merge that copies postings, term frequencies, positions, stored documents, and field lengths without re-analysis;
• consecutive DocId remapping across merged live documents;
• preservation of indexed-only fields during Merge;
• stable old Reader snapshots across both deletion commits and Merge;
• a deterministic, versioned single-file Segment format;
• persistence of Schema, terms, postings, term frequencies, positions, stored documents, field lengths, field statistics, and document count;
• explicit rejection of unknown versions, truncation, checksum failures, and structurally inconsistent payloads.

en_stem = SimpleTokenizer
       -> RemoveLongFilter(40)
       -> LowerCaseFilter
       -> EnglishStemmerFilter

#M5b-4a

application word list ──→ ChineseLexicon (immutable Trie)
                                  │
                                  ▼
mixed input ──→ ChineseTokenizer ──→ RemoveLongFilter(40) ──→ LowerCaseFilter
                    │
          ┌─────────┴─────────┐
          ▼                   ▼
  longest Han match   contiguous non-Han text
          │
  one-character fallback

let lexicon = @MoonSearch.ChineseLexicon::new([
  "中文", "搜索引擎",
]) catch {
  error => abort(error.to_string())
}
let manager = @MoonSearch.TokenizerManager::with_defaults()
manager.register("zh_dict", @MoonSearch.chinese_analyzer(lexicon)) catch {
  error => abort(error.to_string())
}
let builder = @MoonSearch.SchemaBuilder::new()
let body = builder.add_text_field(
  "body",
  @MoonSearch.TextOptions::new(true, true).with_tokenizer("zh_dict"),
)

#M5b-4b

weighted immutable Trie ──→ candidates at every Han position
                                      │
                                      ▼
                            validated frequency DAG
                                      │
                       reverse dynamic programming
                                      │
                                      ▼
                       globally highest-probability route

let lexicon = @MoonSearch.ChineseLexicon::from_entries([
  @MoonSearch.ChineseLexiconEntry::new("研究", 1000),
  @MoonSearch.ChineseLexiconEntry::new("研究生", 1),
  @MoonSearch.ChineseLexiconEntry::new("生命", 1000),
]) catch {
  error => abort(error.to_string())
}
let manager = @MoonSearch.TokenizerManager::with_defaults()
manager.register("zh_dag", @MoonSearch.chinese_dag_analyzer(lexicon)) catch {
  error => abort(error.to_string())
}

#M5b-4c-1

application / Directory / embedded asset
                  │
            UTF-8 String
                  │
                  ▼
ChineseLexicon::from_dictionary_text
                  │
       incremental Trie builder
                  │
                  ▼
       immutable ChineseLexicon

///|
let dictionary_text = "研究 1000 v\n研究生 1 n\n生命 1000 n\n命 1 n\n起源 1000 n\n"

///|
let lexicon = @MoonSearch.ChineseLexicon::from_dictionary_text(dictionary_text) catch {
  error => abort(error.to_string())
}

///|
let analyzer = @MoonSearch.chinese_dag_analyzer(lexicon)

#M5b-4c-2

weighted dictionary DAG
          │
          ▼
consecutive one-character route buffer
          │
          ├─ buffer is an exact dictionary word ─→ preserve DAG singletons
          │
          └─ otherwise
                    │
                    ▼
          constrained B/M/E/S Viterbi
                    │
                    ▼
          recognized unknown-word tokens

///|
let model : &@MoonSearch.ChineseHmmModel = application_hmm_model

///|
let analyzer = @MoonSearch.chinese_dag_hmm_analyzer(lexicon, model)

#M5c-1

Token(position, position_length)
              │
              ▼
       validated DAG edges
              │
              ▼
 bounded finite-string expansion
        ┌─────┴─────┐
        ▼           ▼
 Boolean path   Phrase path
 alternatives  alternatives

#M5c-2

• words longer than two characters may emit dictionary-backed two-character subwords;
• words longer than three characters may also emit dictionary-backed three-character subwords;
• the original word is always retained;
• arbitrary N-grams and subwords absent from the dictionary are not emitted.

///|
let longest = @MoonSearch.chinese_search_analyzer(dictionary)

///|
let dag = @MoonSearch.chinese_dag_search_analyzer(dictionary)

///|
let dag_hmm = @MoonSearch.chinese_dag_hmm_search_analyzer(dictionary, model)

#M5c-3

• ChineseSearchModeFilter::new preserves the character-offset Token Graph for analyzer inspection and graph-aware query construction;
• ChineseSearchModeFilter::for_index stacks the original word and all of its dictionary subwords at one ordinal position, with position_length == 1.

• adjacent primary words remain adjacent;
• every emitted dictionary subword remains term-searchable;
• ordered subword phrases spanning different primary words remain exact;
• the Segment v2 format remains unchanged.

BooleanQuery(
  Should(BoostQuery(TermQuery(title, "moonbit"), 3.0)),
  Should(BoostQuery(TermQuery(body, "moonbit"), 1.0)),
)

#Run the example

moon run cmd/main

#Test

moon test

#License