#MoonNLP

#Scope and features

• core: Trie and Double Array Trie data structures with checked prefix lookup.
• segment: dictionary FMM/BMM/BiMM segmentation, trainable BMES HMM segmentation, hybrid segmentation, and POS tagging.
• analysis: sentence splitting, TF-IDF, TextRank, similarity, classification, metrics, configurable extractive summaries, and a document pipeline.
• cmd, examples, and benchmarks: deterministic executable demonstrations and a workload smoke harness.

#Requirements

• Git
• MoonBit toolchain with moon and moonc
• A C compiler is needed only for the native target on machines where MoonBit requires one.

#Clone, build, and test

git clone https://github.com/ppyq882/moonnlp.git
Set-Location moonnlp
moon version --all
moon update
moon fmt --check
moon check --deny-warn --target all
moon build --target all
moon test --deny-warn --target all
moon info --target all

powershell -ExecutionPolicy Bypass -File scripts/verify_acceptance.ps1 -SkipUpdate

#Quick start

moon run cmd/main

#HMM training example

let corpus = [["alpha", "beta"], ["alpha", "gamma"]]
match @segment.HMMModel::train(corpus) {
  Ok(model) => {
    let segmenter = @segment.HMMSegmenter::from_model(model)
    println(segmenter.segment("alphabeta"))
  }
  Err(message) => println("HMM training error: " + message)
}

moon run examples/train_hmm

#Document analysis example

let classifier = @analysis.NaiveBayesClassifier::new()
classifier.train("technology", ["MoonBit", "compiler", "tokenizer"])
let pipeline = @analysis.DocumentPipeline::with_classifier(classifier)
match pipeline.analyze("MoonBit compiler tokenizer", 3, 1) {
  Ok(result) => println(result.summary)
  Err(message) => println("analysis error: " + message)
}

moon run examples/document_pipeline

#Public API and architecture

import {
  "ppyq882/moonnlp/core",
  "ppyq882/moonnlp/segment",
  "ppyq882/moonnlp/analysis",
}

• core owns checked prefix data structures and Unicode character helpers.
• segment consumes the shared token policy. Dictionary lookup uses the Double Array Trie; HMM training computes initial, transition, and emission probabilities with smoothing; hybrid segmentation combines dictionary and HMM paths.
• analysis composes tokenization with statistical scoring and classification. DocumentPipeline is the convenience boundary for sentences, tokens, keywords, summaries, and an optional label.

#Benchmark scope

moon run benchmarks

#Mooncakes

moon add ppyq882/moonnlp --dry-run

moon add ppyq882/moonnlp

#CI

• moon version --all
• moon fmt --check
• moon check --deny-warn --target all
• moon build --target wasm,wasm-gc,js
• moon info --target all
• generated-interface drift check
• moon test --deny-warn --target wasm,wasm-gc,js
• native build and test when the runner has a C compiler

#Development, contribution, and release

moon fmt --check
moon check --deny-warn --target all
moon build --target all
moon test --deny-warn --target all
moon info --target all
git diff --check

#License, references, and acknowledgement

#Known boundaries

• Segmentation, tagging, and summary quality depend on deterministic algorithms and the included data; evaluate them against your domain.
• HMMModel is a trainable model API, not a pretrained language model service.
• NaiveBayesClassifier is a small in-memory classifier without persistence or probability calibration.
• The CLI is a deterministic demonstration, not a general-purpose argument parser.