moongettext

Pure MoonBit GNU gettext catalog parser, compiler, and runtime lookup library.

gettext
i18n
po
pot
mo
localization
moon add YanFangNie/moongettext@0.1.1
Download zip
Version
0.1.1
License
Apache-2.0
Last updated
8 days ago
Downloads
7

Dependencies

README

#moongettext

Pure MoonBit tooling for GNU gettext catalogs.

moongettext parses and writes PO/POT text, compiles and reads MO revision 0 binaries, evaluates Plural-Forms, validates catalogs, merges templates, and provides context-aware runtime lookup. The library code is backend-neutral and is checked on MoonBit's wasm, wasm-gc, js, and native stable targets.

The file-oriented CLI uses moonbitlang/x/fs, so that package is intentionally native-only.

#Why this package?

Applications often need a translation format that translators and existing localization platforms already understand. PO is reviewable text, MO is a compact runtime format, and the plural-expression language covers grammatical rules that cannot be represented by a simple singular/plural Boolean.

moongettext keeps those concerns in one typed model:

  • five PO comment classes: translator, extracted, reference, flag, previous;
  • contexts, singular and plural entries, multiline strings, standard C escapes;
  • obsolete #~ entries and fuzzy filtering;
  • deterministic semantic PO/POT serialization;
  • GNU MO revision 0 read/write in both byte orders;
  • a C-like Plural-Forms parser with short-circuit and ternary evaluation;
  • runtime gettext, pgettext, ngettext, and npgettext operations;
  • explicit catalog fallback;
  • validation, statistics, source-reference queries, and POT/PO merging.

#Install

Install the package with Moon:

moon add YanFangNie/moongettext

For this source checkout, no global installation is required:

moon check --target all --deny-warn --warn-list +73 moon test --target all --deny-warn --warn-list +73 moon run examples/basic

The module pins moonbitlang/x for the native file CLI. The root library itself uses only MoonBit core APIs.

#Quick start: parse and normalize PO

parse_po raises a structured GettextError for malformed input. write_po normalizes layout but preserves the represented messages and comments.

///|
test "parse and normalize a PO entry" {
let source =
#|#. Shown on the launch screen
#|#: src/main.mbt:12
#|msgctxt "button"
#|msgid "Open"
#|msgstr "Ouvrir"
#|
let document = @moongettext.parse_po(source)
assert_eq(document.entries.length(), 1)
assert_true(
document.find_entry("Open", context="button") is Some(entry) &&
entry.translation(0) == Some("Ouvrir"),
)
let reparsed = @moongettext.parse_po(@moongettext.write_po(document))
assert_true(reparsed == document)
}

PO serialization is a semantic round trip, not a byte-for-byte formatter: indentation, blank lines, quoting, and multiline wrapping are canonicalized.

#Runtime lookup

Catalog construction reads Plural-Forms from the empty-msgid metadata header. Missing translations return source text. Fuzzy and obsolete entries are excluded from runtime catalogs by default.

///|
test "context and plural-aware lookup" {
let document = @moongettext.PoFile::new([
@moongettext.PoEntry::singular(
"",
translation=(
#|Content-Type: text/plain; charset=UTF-8
#|Language: fr
#|Plural-Forms: nplurals=2; plural=(n > 1);
#|
),
),
@moongettext.PoEntry::singular(
"Open",
translation="Ouvrir",
context="button",
),
@moongettext.PoEntry::plural("file", "files", ["fichier", "fichiers"]),
])
let catalog = @moongettext.Catalog::from_po(document)
assert_eq(catalog.pgettext("button", "Open"), "Ouvrir")
assert_eq(catalog.ngettext("file", "files", 1), "fichier")
assert_eq(catalog.ngettext("file", "files", 3), "fichiers")
assert_eq(catalog.gettext("missing"), "missing")
}

Fallback is explicit:

///|
test "fallback catalog" {
let primary = @moongettext.Catalog::from_po(
@moongettext.PoFile::new([
@moongettext.PoEntry::singular("Hello", translation="Salut"),
]),
)
let fallback = @moongettext.Catalog::from_po(
@moongettext.PoFile::new([
@moongettext.PoEntry::singular("Open", translation="Ouvrir"),
]),
)
assert_eq(primary.gettext("Hello", fallback~), "Salut")
assert_eq(primary.gettext("Open", fallback~), "Ouvrir")
assert_eq(primary.gettext("Unknown", fallback~), "Unknown")
}

#MO compile and load

compile_mo_checked first rejects validation errors and then emits deterministic GNU MO revision 0 bytes. Obsolete, fuzzy, and untranslated non-header entries are omitted. parse_mo validates all table ranges, NUL terminators, duplicate keys, and UTF-8 before exposing a document.

///|
test "compile and load big-endian MO" {
let source =
#|msgid ""
#|msgstr "Content-Type: text/plain; charset=UTF-8\n"
#|
#|msgid "Save"
#|msgstr "Enregistrer"
#|
let po = @moongettext.parse_po(source)
let bytes = @moongettext.compile_mo_checked(po, endian=Big)
assert_true(@moongettext.mo_endian(bytes) == Big)
let loaded = @moongettext.Catalog::from_mo(bytes)
assert_eq(loaded.gettext("Save"), "Enregistrer")
}

Use compile_mo instead of compile_mo_checked only when a caller intentionally handles validation separately.

#Plural-Forms

Supported operators, from high to low precedence:

  1. primary values: integer, n, parentheses;
  2. unary !, unary +, unary -;
  3. *, /, %;
  4. +, -;
  5. <, <=, >, >=;
  6. ==, !=;
  7. &&;
  8. ||;
  9. right-associative ?:.

Logical operations short-circuit and produce 0 or 1.

///|
test "Russian three-form expression" {
let rule = @moongettext.parse_plural_forms(
"nplurals=3; plural=n%10==1 && n%100!=11 ? 0 : n%10>=2 && n%10<=4 && (n%100<10 || n%100>=20) ? 1 : 2;",
)
assert_eq(rule.select(1), 0)
assert_eq(rule.select(2), 1)
assert_eq(rule.select(5), 2)
assert_eq(rule.select(21), 0)
}

Negative counts, division/modulo by zero, malformed syntax, and results outside [0, nplurals) are reported as errors.

#Validate a catalog

validate_po returns every finding in stable order. It checks:

  • metadata charset and Plural-Forms;
  • duplicate context-aware keys;
  • plural translation arity;
  • embedded MO separator characters;
  • fuzzy, obsolete, and untranslated entries;
  • representative evaluation of plural rules for counts 0..=200.

///|
test "structured validation report" {
let invalid = @moongettext.PoFile::new([
@moongettext.PoEntry::plural("item", "", [""]),
])
let issues = @moongettext.validate_po(invalid)
assert_true(@moongettext.validation_has_errors(issues))
let report = @moongettext.validation_report(issues)
assert_true(report.contains("empty-plural-id"))
}

validate_po_strict and compile_mo_checked raise if any IssueError exists. Warnings do not block compilation.

#Merge a POT template

merge_template(template, existing) retains translations for matching context/msgid keys, refreshes extracted comments and source references, keeps translator notes, marks changed plural sources fuzzy, and carries removed messages as obsolete.

///|
test "merge template with translated PO" {
let template = @moongettext.PoFile::new([
@moongettext.PoEntry::singular("Open"),
@moongettext.PoEntry::singular("New"),
])
let existing = @moongettext.PoFile::new([
@moongettext.PoEntry::singular("Open", translation="Ouvrir"),
@moongettext.PoEntry::singular("Old", translation="Ancien"),
])
let merged = @moongettext.merge_template(template, existing)
assert_true(merged.stats == { matched: 1, added: 1, obsoleted: 1 })
assert_eq(merged.document.entries[0].translations[0], "Ouvrir")
assert_true(merged.document.entries[2].obsolete)
}

Set keep_obsolete=false to omit removed entries.

#CLI

The native CLI is a reproducible example and a useful catalog probe:

moon run cmd/main -- demo moon run cmd/main -- validate examples/fr.po moon run cmd/main -- normalize examples/fr.po moon run cmd/main -- compile examples/fr.po _build/fr.mo moon run cmd/main -- compile examples/fr.po _build/fr-be.mo --big-endian moon run cmd/main -- inspect _build/fr.mo

validate returns exit status 1 for error findings. compile refuses error-level findings and writes no partial output. Run moon run cmd/main -- help for the same command summary.

The backend-neutral example is:

moon run examples/basic

Expected output:

Ouvrir 0: fichier 1: fichier 2: fichiers 5: fichiers

#Main public API

AreaEntry points
PO/POTparse_po, write_po, write_pot, PoFile::as_template
MOcompile_mo, compile_mo_checked, parse_mo, mo_endian
Pluralsparse_plural_forms, evaluate_plural_expression, PluralRule::select
RuntimeCatalog::from_po, Catalog::from_mo, gettext, pgettext, ngettext, npgettext
Mergemerge_template, MergeResult, MergeStats
Validationvalidate_po, validate_po_strict, validation_report
Queryfind_entry, flags, source_references, statistics, metadata

Run moon info to regenerate pkg.generated.mbti, which is the exact public surface for the checked-out version.

#Compatibility and limits

See docs/compatibility.md for the detailed matrix. Important boundaries:

  • PO output preserves semantics, not original whitespace or line wrapping.
  • MO revision 0 is supported; system-dependent revision extensions are not.
  • MO strings must be UTF-8. Other declared charsets are reported.
  • MO hash tables are accepted as header fields but ignored; emitted files use binary-search-compatible sorted original tables and no hash table.
  • Advanced quoted source-reference filenames are retained as comment text but the convenience source_references tokenizer does not reconstruct them.
  • This package does not extract translatable strings from MoonBit source and does not replace a translation-management platform.

#Project layout

model / escape shared PO model and string escaping po_parser / po_writer semantic PO/POT codec plural_lexer / plural_parser Plural-Forms grammar and evaluator mo_reader / mo_writer strict MO revision 0 codec catalog indexed runtime lookup and fallback validation / query / merge tooling APIs cmd/main native file CLI examples/basic backend-neutral runnable example

#Development and release checks

moon version --all moon fmt --check moon info --target all moon check --target all --deny-warn --warn-list +73 moon test --target all --deny-warn --warn-list +73 moon build --target native moon run examples/basic moon run cmd/main -- validate examples/fr.po moon package --frozen git diff --exit-code

CI runs the same format/interface/check/build/test/example/CLI/package gates.

#Specification and source notice

This is an original MoonBit implementation informed by the public GNU gettext format specifications and observable behavior. It is not a line-by-line port of an existing codebase:

The ecosystem survey also found Zhouz-z/moon_l10n, which focuses on an ICU MessageFormat subset and catalog linting. moongettext instead focuses on GNU PO/POT/MO interoperability and gettext-style runtime lookups; the projects have no implementation-source relationship.

No GNU gettext implementation source, generated code, or third-party catalog fixture is copied into this repository. Test inputs are small original fixtures. The project is licensed under Apache-2.0; see LICENSE.

#
GettextError

pub(all) suberror GettextError {
PoSyntax(line~ : Int, column~ : Int, message~ : String)
PluralSyntax(position~ : Int, message~ : String)
MoFormat(offset~ : Int, message~ : String)
Validation(message~ : String)
} derive(Eq,
Debug
)

A structured error raised while reading, compiling, or validating a gettext catalog.

#
Catalog

pub struct Catalog {
messages : Map[String, PoEntry]
rule : PluralRule
}

An indexed runtime translation catalog.

Construct catalogs with Catalog::from_po or Catalog::from_mo; fields are intentionally private so duplicate-key and plural-rule checks cannot be bypassed.

#
Catalog::from_mo

fn Catalog::from_mo(bytes : Bytes) -> Catalog raise GettextError

Build a runtime catalog directly from GNU MO bytes.

#
Catalog::from_po

fn Catalog::from_po(document : PoFile, default_rule? : PluralRule, include_fuzzy? : Bool) -> Catalog raise GettextError

Build a runtime catalog from a parsed PO document.

Obsolete entries and the metadata header are excluded. Fuzzy entries are excluded by default and can be explicitly included for preview tooling. When no Plural-Forms header exists, the English rule (or default_rule) is used.

#
Catalog::gettext

fn Catalog::gettext(self : Catalog, msgid : String, fallback? : Catalog) -> String

Look up a singular message, returning the source msgid when neither this catalog nor the optional fallback has a non-empty translation.

#
Catalog::length

fn Catalog::length(self : Catalog) -> Int

Number of indexed message entries, excluding the metadata header.

#
Catalog::ngettext

fn Catalog::ngettext(self : Catalog, singular : String, plural : String, n : Int, fallback? : Catalog) -> String raise GettextError

Look up a plural message using this catalog's Plural-Forms rule.

If no translation is available, the fallback catalog is tried. Source text finally falls back to singular for n == 1 and plural otherwise.

#
Catalog::npgettext

fn Catalog::npgettext(self : Catalog, context : String, singular : String, plural : String, n : Int, fallback? : Catalog) -> String raise GettextError

Look up a context-qualified plural message.

#
Catalog::pgettext

fn Catalog::pgettext(self : Catalog, context : String, msgid : String, fallback? : Catalog) -> String

Look up a context-qualified singular message.

#
Catalog::plural_rule

fn Catalog::plural_rule(self : Catalog) -> PluralRule

Return the catalog's validated plural rule.

#
CommentKind

pub(all) enum CommentKind {
Translator
Extracted
Reference
Flag
Previous
} derive(Eq,
Debug
)

The semantic kind of a comment attached to a PO entry.

GNU gettext distinguishes comments by the marker immediately following #: no marker for translator comments, . for extracted comments, : for source references, , for flags, and | for previous values.

#
Endian

pub(all) enum Endian {
Little
Big
} derive(Eq,
Debug
)

Byte order used when writing or reading a GNU MO file.

#
IssueSeverity

pub(all) enum IssueSeverity {
IssueInfo
IssueWarning
IssueError
} derive(Eq,
Debug
)

Severity assigned to a catalog validation issue.

#
MergeResult

pub(all) struct MergeResult {
document : PoFile
stats : MergeStats
} derive(Eq,
Debug
)

Result of merging a POT-style template with an existing translation.

#
MergeStats

pub(all) struct MergeStats {
matched : Int
added : Int
obsoleted : Int
} derive(Eq,
Debug
)

Summary of a template/catalog merge.

#
PluralRule

pub struct PluralRule {
nplurals : Int
expression : String
} derive(Eq,
Debug
)

A validated Plural-Forms rule.

#
PluralRule::english

fn PluralRule::english() -> PluralRule

The conventional English rule: singular only when n == 1.

#
PluralRule::select

fn PluralRule::select(self : PluralRule, n : Int) -> Int raise GettextError

Select a translation index and ensure the expression stays inside the declared [0, nplurals) range.

#
PoComment

pub(all) struct PoComment {
kind : CommentKind
text : String
} derive(Eq,
Debug
)

A comment attached to a PO entry.

#
PoComment::new

fn PoComment::new(kind : CommentKind, text : String) -> PoComment

Construct a PO comment.

#
PoEntry

pub(all) struct PoEntry {
comments : Array[PoComment]
context : String?
msgid : String
msgid_plural : String?
translations : Array[String]
obsolete : Bool
} derive(Eq,
Debug
)

One message entry in a PO or POT catalog.

translations stores msgstr as element zero for singular entries and stores msgstr[N] values by numeric index for plural entries. Missing indexes are represented by empty strings so indexes remain stable.

#
PoEntry::flags

fn PoEntry::flags(self : PoEntry) -> Array[String]

Return unique comma-separated flags in first-seen order.

#
PoEntry::has_flag

fn PoEntry::has_flag(self : PoEntry, flag : String) -> Bool

Return true when a flag comment contains the exact comma-separated flag.

#
PoEntry::is_fuzzy

fn PoEntry::is_fuzzy(self : PoEntry) -> Bool

Return true when this entry has GNU gettext's fuzzy flag.

#
PoEntry::is_header

fn PoEntry::is_header(self : PoEntry) -> Bool

Return true when this entry is the conventional metadata header.

#
PoEntry::plural

fn PoEntry::plural(msgid : String, msgid_plural : String, translations : Array[String], context? : String) -> PoEntry

Construct a plural entry without comments.

#
PoEntry::singular

fn PoEntry::singular(msgid : String, translation? : String, context? : String) -> PoEntry

Construct a singular entry without comments.

#
PoEntry::source_references

fn PoEntry::source_references(self : PoEntry) -> Array[SourceReference]

Decode whitespace-separated #: source reference tokens.

path, path:line, and path:line:column are recognized. GNU PO permits tool-specific quoting for paths containing whitespace; this helper keeps those advanced spellings as separate raw tokens rather than guessing.

#
PoEntry::translation

fn PoEntry::translation(self : PoEntry, index : Int) -> String?

Return a translation by plural index, or None if it is absent or empty.

#
PoFile

pub(all) struct PoFile {
entries : Array[PoEntry]
} derive(Eq,
Debug
)

A parsed PO or POT file.

#
PoFile::as_template

fn PoFile::as_template(self : PoFile) -> PoFile

Return a POT-style copy with non-header translations cleared.

The metadata header is retained because template generators commonly store MIME and project fields there. Singular entries receive one empty msgstr; plural entries retain at least two indexed slots.

#
PoFile::find_entry

fn PoFile::find_entry(self : PoFile, msgid : String, context? : String, include_obsolete? : Bool) -> PoEntry?

Find a context-aware entry. Obsolete entries are ignored by default.

#
PoFile::header

fn PoFile::header(self : PoFile) -> PoEntry?

Return the first conventional metadata header entry, if present.

#
PoFile::metadata

fn PoFile::metadata(self : PoFile) -> Map[String, String]

Return parsed fields from the conventional empty-msgid header entry.

#
PoFile::new

fn PoFile::new(entries : Array[PoEntry]) -> PoFile

Construct a catalog document from entries.

#
PoFile::statistics

fn PoFile::statistics(self : PoFile) -> PoStatistics

Compute catalog status counts in one pass.

#
PoStatistics

pub(all) struct PoStatistics {
headers : Int
messages : Int
active : Int
translated : Int
untranslated : Int
fuzzy : Int
obsolete : Int
singular : Int
plural : Int
contextual : Int
} derive(Eq,
Debug
)

Aggregate status counts for a parsed PO/POT document.

#
SourceReference

pub(all) struct SourceReference {
path : String
line : Int?
column : Int?
} derive(Eq,
Debug
)

A source location decoded from a #: PO reference comment.

#
ValidationIssue

pub(all) struct ValidationIssue {
severity : IssueSeverity
entry : Int?
code : String
message : String
} derive(Eq,
Debug
)

One deterministic catalog validation finding.

#
compile_mo

fn compile_mo(document : PoFile, endian? : Endian) -> Bytes raise GettextError

Compile a PO document into GNU MO revision 0 bytes.

Obsolete, fuzzy, and untranslated non-header entries are deliberately omitted, matching the behavior expected from a production message compiler. Original strings are sorted by their UTF-8 byte representation.

#
compile_mo_checked

fn compile_mo_checked(document : PoFile, endian? : Endian) -> Bytes raise GettextError

Validate a document strictly and then compile it to GNU MO bytes.

#
escape_po_string

fn escape_po_string(input : String) -> String

Escape a string for the body of a double-quoted PO string.

The returned value does not include the surrounding quote characters.

#
evaluate_plural_expression

fn evaluate_plural_expression(expression : String, n : Int) -> Int raise GettextError

Evaluate one GNU gettext plural expression for a non-negative count.

#
merge_template

fn merge_template(template : PoFile, existing : PoFile, keep_obsolete? : Bool) -> MergeResult raise GettextError

Merge a POT-style template with an existing translated PO document.

Source comments and references come from the template; translator and previous-value comments come from the existing catalog. Matching translations are retained. A changed plural source is marked fuzzy. Existing entries absent from the template are appended as obsolete unless keep_obsolete=false.

#
mo_endian

fn mo_endian(bytes : Bytes) -> Endian raise GettextError

Detect byte order from a GNU MO magic word.

#
parse_header_fields

fn parse_header_fields(header : String) -> Map[String, String]

Parse the translated string of a gettext header entry into metadata fields.

Lines without a colon are ignored. A line beginning with a space or tab is treated as a continuation of the preceding field.

#
parse_mo

fn parse_mo(bytes : Bytes) -> PoFile raise GettextError

Parse a GNU MO revision 0 file into the shared catalog model.

Both little- and big-endian files are accepted. Header/table bounds, NUL terminators, UTF-8 strings, duplicate keys, context separators, and plural separators are checked before a document is returned.

#
parse_plural_forms

fn parse_plural_forms(input : String) -> PluralRule raise GettextError

Parse a Plural-Forms value or a complete gettext metadata header.

Both nplurals and plural assignments are required. Assignment order and surrounding whitespace are ignored.

#
parse_po

fn parse_po(source : String) -> PoFile raise GettextError

Parse a GNU gettext PO or POT document.

The parser accepts LF and CRLF input, multiline quoted strings, contexts, plurals, the five standard comment classes, and obsolete #~ entries. Parsed output is semantic rather than lossless: whitespace and line wrapping are normalized when the document is written again.

#
unescape_po_string

fn unescape_po_string(input : String) -> String raise GettextError

Decode C-style escapes used inside a PO quoted string.

Common named escapes, octal escapes of up to three digits, and hexadecimal escapes are supported. Unknown escapes follow gettext's permissive reader behavior and evaluate to the escaped character itself.

#
validate_po

fn validate_po(document : PoFile) -> Array[ValidationIssue]

Validate PO/POT semantics used by the parser, MO compiler, and runtime.

The returned array is stable in document order. Validation checks metadata, plural rules, duplicate context-aware keys, translation arity, fuzzy and obsolete status, embedded NULs, and context separator collisions.

#
validate_po_strict

fn validate_po_strict(document : PoFile) -> Unit raise GettextError

Raise a single Validation error when a catalog has any error findings.

#
validation_has_errors

fn validation_has_errors(issues : ArrayView[ValidationIssue]) -> Bool

Return true when at least one validation issue is an error.

#
validation_report

fn validation_report(issues : ArrayView[ValidationIssue]) -> String

Format findings as a line-oriented CLI report.

#
write_po

fn write_po(document : PoFile) -> String

Serialize a PO/POT document into deterministic GNU gettext syntax.

Semantic information is preserved, while whitespace, entry separation, and multiline layout are normalized. Every entry is followed by one blank line.

#
write_pot

fn write_pot(document : PoFile) -> String

Serialize a document as a translation template.

Powered by MoonBit

Site sourceReport issuePackagesBuild queueSkillsStatistics

© 2026 mooncakes.io