🥮 mooncakes.io
README

Lucius646/MoonSearch/analysis/chinese does not have a README file

#
ChineseDictionary

source
pub(open) trait ChineseDictionary {
  fn matches_at(Self, ReadOnlyArray[Char], Int, Int) -> Array[ChineseWordMatch]
  fn total_frequency(Self) -> Int = _
}

Dictionary extension point consumed by ChineseTokenizer.

Implementations return every word beginning at start, bounded by end. Returning candidates instead of choosing one keeps segmentation policy out of the dictionary and leaves the boundary usable by later DAG routing.

#
ChineseHmmModel

source
pub(open) trait ChineseHmmModel {
  fn start_score(Self, ChineseHmmState) -> Double
  fn transition_score(Self, ChineseHmmState, ChineseHmmState) -> Double
  fn emission_score(Self, ChineseHmmState, Char) -> Double
}

Open HMM score provider. All values are natural-log scores.

#
ChineseDictionaryResourceError

source
pub(all) suberror ChineseDictionaryResourceError {
  InvalidFieldCount(Int, Int)
  InvalidFrequency(Int, String)
  InvalidWord(Int, String)
  DuplicateWord(Int, String)
  FrequencyOverflow(Int)
  LexiconError(ChineseLexiconError)
} derive(Eq, 
Debug)

Errors raised while parsing a UTF-8 Chinese dictionary text resource.

Line numbers are one-based. The accepted format is word frequency [tag]; an optional tag is parsed but not retained.

#
ChineseHmmModelError

source
pub(all) suberror ChineseHmmModelError {
  IllegalStart(ChineseHmmState)
  IllegalTransition(ChineseHmmState, ChineseHmmState)
  DuplicateStart(ChineseHmmState)
  DuplicateTransition(ChineseHmmState, ChineseHmmState)
  DuplicateEmission(ChineseHmmState, Char)
  MissingStart(ChineseHmmState)
  MissingTransition(ChineseHmmState, ChineseHmmState)
  InvalidStartScore(ChineseHmmState)
  InvalidTransitionScore(ChineseHmmState, ChineseHmmState)
  InvalidEmissionScore(ChineseHmmState, Char)
  InvalidUnknownEmissionScore
} derive(Eq, 
Debug)

Structural errors raised while snapshotting one table-backed HMM model.

#
ChineseLexiconError

source
pub(all) suberror ChineseLexiconError {
  EmptyWord
  NonHanWord(String)
  InvalidFrequency(String, Int)
  DuplicateWord(String)
  FrequencyOverflow
} derive(Eq, 
Debug)

Errors raised while constructing the built-in immutable Chinese lexicon.

#
ChineseHmmState

source
pub(all) enum ChineseHmmState {
  Begin
  Middle
  End
  Single
} derive(Compare, Eq, 
Debug)

B/M/E/S states used by Chinese unknown-word recognition.

#
ChineseLexicon

source
pub struct ChineseLexicon {
  labels : ReadOnlyArray[ReadOnlyArray[Char]]
  children : ReadOnlyArray[ReadOnlyArray[Int]]
  terminal_frequency : ReadOnlyArray[Int]
  total_frequency : Int
}

Immutable trie-backed snapshot of application-provided Chinese words.

#
ChineseLexicon::from_dictionary_text

source
fn ChineseLexicon::from_dictionary_text(text : String) -> ChineseLexicon raise ChineseDictionaryResourceError

Parses a caller-provided UTF-8 dictionary resource into an immutable Trie.

File-system access and caching stay outside the analysis package so the same parser works for native, JavaScript, WASM, and custom Directory data.

#
ChineseLexicon::from_entries

source

Builds an immutable weighted Trie snapshot for frequency-DAG routing.

#
ChineseLexicon::new

source
fn ChineseLexicon::new(words : Array[String]) -> ChineseLexicon raise ChineseLexiconError

#
ChineseLexiconEntry

source
pub(all) struct ChineseLexiconEntry {
  word : String
  frequency : Int
} derive(Eq, 
Debug)

One immutable Chinese lexicon entry and its corpus frequency.

#
ChineseLexiconEntry::new

source
fn ChineseLexiconEntry::new(word : String, frequency : Int) -> ChineseLexiconEntry

#
ChineseSearchModeFilter

source
pub struct ChineseSearchModeFilter {
  dictionary : &ChineseDictionary
  flatten_for_index : Bool
}

Adds dictionary-backed two-character and three-character subwords to a linear Chinese token stream.

The filter follows Jieba search-mode candidate selection while expressing overlaps as Token Graph edges. It expects strictly increasing input positions with position_length == 1, as emitted by ChineseTokenizer.

#
ChineseSearchModeFilter::for_index

source

Creates the lossy index-time position policy.

The original word and all of its subwords are stacked at one ordinal position because Segment postings do not persist graph edge lengths.

#
ChineseSearchModeFilter::new

source

#
ChineseTokenizer

source
pub struct ChineseTokenizer {
  dictionary : &ChineseDictionary
  frequency_dag : Bool
  hmm_model : &ChineseHmmModel?
}

Dictionary-backed tokenizer for mixed Chinese and non-Chinese text.

new uses deterministic longest matching; the other constructors opt into frequency-DAG routing and optional HMM recognition. Consecutive non-Han, non-punctuation text is emitted as one token; normalization belongs to the surrounding TextAnalyzer pipeline.

#
ChineseTokenizer::analyze

source

#
ChineseTokenizer::new

source

#
ChineseTokenizer::with_frequency_dag

source
fn ChineseTokenizer::with_frequency_dag(dictionary : &ChineseDictionary) -> ChineseTokenizer

Creates a tokenizer that selects a globally optimal frequency-DAG route.

#
ChineseTokenizer::with_frequency_dag_and_hmm

source
fn ChineseTokenizer::with_frequency_dag_and_hmm(dictionary : &ChineseDictionary, model : &ChineseHmmModel) -> ChineseTokenizer

Creates a frequency-DAG tokenizer with HMM unknown-word recognition.

#
ChineseWordMatch

source
pub(all) struct ChineseWordMatch {
  length : Int
  frequency : Int
} derive(Eq, 
Debug)

One dictionary word beginning at a requested character position.

#
TableChineseHmmModel

source
pub struct TableChineseHmmModel {
  start_scores : ReadOnlyArray[Double]
  transition_scores : ReadOnlyArray[Double]
  emission_scores : ReadOnlyArray[
HashMap[Char, Double]]
  unknown_emission_score : Double
}

Immutable public facade over snapshotted HMM score tables.

#
TableChineseHmmModel::new

source
fn TableChineseHmmModel::new(start_scores : Array[(ChineseHmmState, Double)], transition_scores : Array[(ChineseHmmState, ChineseHmmState, Double)], emission_scores : Array[(ChineseHmmState, Char, Double)], unknown_emission_score : Double) -> TableChineseHmmModel raise ChineseHmmModelError

Builds a strict table model.

Both legal start states and all eight legal transitions must be supplied exactly once. Emissions are sparse and fall back to unknown_emission_score.

#
chinese_analyzer

source

Creates a lowercase, maximum-length-limited analyzer around one dictionary.

#
chinese_dag_analyzer

source

Creates a lowercase analyzer using globally optimal frequency-DAG routing.

#
chinese_dag_hmm_analyzer

source
fn chinese_dag_hmm_analyzer(dictionary : &ChineseDictionary, model : &ChineseHmmModel) -> 
TextAnalyzer

Creates a lowercase frequency-DAG analyzer with injected HMM recognition.

#
chinese_dag_hmm_search_analyzer

source
fn chinese_dag_hmm_search_analyzer(dictionary : &ChineseDictionary, model : &ChineseHmmModel) -> 
TextAnalyzer

Creates a frequency-DAG/HMM analyzer with dictionary-backed search subwords.

#
chinese_dag_hmm_search_index_analyzer

source
fn chinese_dag_hmm_search_index_analyzer(dictionary : &ChineseDictionary, model : &ChineseHmmModel) -> 
TextAnalyzer

Creates the DAG/HMM search expansion intended for indexing.

#
chinese_dag_search_analyzer

source
fn chinese_dag_search_analyzer(dictionary : &ChineseDictionary) -> 
TextAnalyzer

Creates a frequency-DAG analyzer with dictionary-backed search subwords.

#
chinese_dag_search_index_analyzer

source
fn chinese_dag_search_index_analyzer(dictionary : &ChineseDictionary) -> 
TextAnalyzer

Creates the frequency-DAG search expansion intended for indexing.

#
chinese_search_analyzer

source

Creates a longest-match analyzer with dictionary-backed search subwords.

#
chinese_search_index_analyzer

source
fn chinese_search_index_analyzer(dictionary : &ChineseDictionary) -> 
TextAnalyzer

Creates the longest-match search expansion intended for indexing.

Source Files

Powered by MoonBit

Site sourceReport issuePackagesBuild queueSkillsStatistics

© 2026 mooncakes.io