moon_proto

A MoonBit protobuf ecosystem lab for dynamic schema validation, compatibility testing, and AI code verification.

protobuf
schema
verification
codegen
wasm
moon add 123123213weqw/moon_proto@0.1.1
Download zip
Version
0.1.1
License
MIT
Last updated
last month
Downloads
15
README

#Moon Proto Lab

CI Mooncakes MoonBit License

Moon Proto Lab 是面向 MoonBit protobuf 生态的 schema 验证、兼容性检查与 AI 代码验证工具链。

它把 .proto 从“看起来正确的文本”变成可诊断、可生成、可编译、可回归、可报告的工程资产:

.proto -> Schema Doctor -> compatibility check -> dynamic binary/JSON runtime -> MoonBit codegen -> generated-code compile -> Markdown/JUnit/CI evidence

本项目不替代现有的 moonbitlang/protobufmoonbitlang/protoc-gen-mbt。它提供的是围绕官方 protobuf 栈的验证与工具层,重点解决 AI 生成 schema/代码难以确认、schema 演进容易破坏兼容性、生成代码缺少持续验证等问题。

#30 秒快速开始

#作为 MoonBit 库安装

moon add 123123213weqw/moon_proto

在使用该库的 moon.pkg 中导入:

import {
"123123213weqw/moon_proto" @proto,
}

最小示例:

fn main {
let encoded = @proto.encode_varint_u64(300UL)
println(encoded.length()) // 2
}

#使用完整验证工具

git clone https://github.com/123123213weqw/moon_proto.git cd moon_proto # 检查 AI/人工生成的 schema python3 scripts/moon_proto_lab.py doctor examples/ai/good_order.proto # 完成 doctor、inspect、codegen、生成代码编译并输出报告 python3 scripts/moon_proto_lab.py verify \ examples/ai/good_order.proto \ --report generated/verify_report.md \ --junit-out generated/verify_report.xml # 检查 old/new schema 是否兼容 python3 scripts/moon_proto_lab.py compat \ examples/ai/good_order.proto \ examples/ai/good_order_v2.proto \ --report generated/compat_report.md

完整本地验收:

bash scripts/release_gate.sh

运行要求:MoonBit;文件版报告工具需要 Python 3;完整跨语言 oracle 还需要 Python protobuf 与 Go。

#Agent 反馈闭环

下图展示 AI Agent 生成 schema 后,Moon Proto Lab 如何返回稳定诊断并驱动修复。它是工作流说明图;后面的终端图来自仓库真实命令记录。

#实际运行证据

每张图都有同名 .txt 命令记录,可核对命令、工作目录、时间和退出码。

#核心能力

能力当前实现
Wire/runtimevarint、zig-zag、fixed32/64、length-delimited、unknown-field skip
Schemaproto3 message、enum、optional/repeated、map、oneof、reserved、nested type、常见声明容错
动态消息descriptor-driven scalar、repeated、packed、enum、nested、map、oneof 二进制编解码
Protobuf JSONenum name、64-bit integer、base64、Unicode、lowerCamel、map key normalization、严格数字语法
Schema Doctor字段号/名称冲突、enum 规则、map 约束、reserved 复用等稳定诊断
兼容性检查old/new .proto 与 FileDescriptorSet 的破坏性变更检查
CodegenMoonBit struct、enum、descriptor、动态 runtime helper,并执行真实 moon check
工程证据Python/Go oracle、建模的 conformance-lite 用例、官方接口契约检查、Markdown/JSON/JUnit 报告
Registry adapterdescriptor registry、release policy、文件/HTTP/profile/GitHub Contents 适配验证

#一个完整场景

故意错误的 schema:

syntax = "proto3"; message Order { reserved 7; reserved "legacy_note"; uint64 id = 1; string duplicate = 1; bytes legacy_note = 7; map<bytes, string> invalid_labels = 8; }

运行:

python3 scripts/moon_proto_lab.py doctor examples/ai/bad_order.proto

输出包含稳定路径:

schema invalid issues: 6 message.Order.field.1: duplicate field number message.Order.field.invalid_labels.key: invalid map key type message.Order.field.legacy_note.number: field uses reserved number message.Order.field.legacy_note.name: field uses reserved name

修复后再执行 verifycompat,生成代码必须真实编译,schema 演进也必须通过兼容性门禁。

#MoonBit API 示例

let desc = @proto.MessageDescriptor::{
name: "User",
fields: [
@proto.FieldDescriptor::{
name: "id",
typ: @proto.UInt64Type,
number: 1,
label: @proto.Singular,
},
@proto.FieldDescriptor::{
name: "name",
typ: @proto.StringType,
number: 2,
label: @proto.Singular,
},
],
}

let msg = @proto.message_value([
@proto.message_field("id", @proto.UInt64Value(150UL)),
@proto.message_field("name", @proto.StringValue("Alice")),
])

let encoded = @proto.encode_message(desc, msg)

#验证与质量

当前提交线的可复现结果:

  • moon check --deny-warn:通过;
  • moon test --deny-warn60/60 passed
  • moon test --target all:wasm、wasm-gc、JS、native 全通过;
  • MoonBit 核心包行覆盖率:1935/2388,约 81.0%
  • generated-code compile check:通过;
  • Python/Go protobuf oracle:通过;
  • AI schema 正例、兼容演进与故意错误负例:通过;
  • GitHub Actions:通过。

常用命令:

moon fmt --check moon info moon package --list moon check --deny-warn moon build moon test --deny-warn moon test --target all moon coverage analyze -p 123123213weqw/moon_proto -- -f summary tests/codegen/compile_generated.sh

#与现有 MoonBit protobuf 项目的关系

MoonBit 生态已有:

Moon Proto Lab 的独立贡献是:

  1. 在生成前检查 schema,并为 AI 输出提供稳定诊断;
  2. 在 schema 演进时检查字段号、类型、reserved 等兼容性合同;
  3. 在生成后真实编译 MoonBit 代码,而不是只做文本快照;
  4. 使用 Python/Go oracle 与 MoonBit golden tests 验证 wire/JSON 行为;
  5. 输出适合 CI、代码审查和 Agent 消费的 Markdown/JSON/JUnit 证据。

#已知边界

当前版本是面向验证场景的 proto3 子集,不宣称完整 protobuf conformance:

  • servicerpc、custom option 等主要做解析容错,不生成完整 RPC 实现;
  • typed struct 的生产级 encode/decode 能力不替代官方生成器,核心验证路径使用动态 MessageValue
  • conformance-lite 是基于公开 protobuf 语义建模的小型 fixture 集,不是上游官方 conformance suite 的镜像;
  • official source/output-shape contract 检查验证公开接口契约;只有显式启用 live-generator 路径时才会实际运行官方生成器;
  • FileDescriptorSet、报告和 registry adapter 目前由 Python 集成层承载,MoonBit 核心实现集中在 parser/runtime/JSON/codegen/compat CLI。

#文档

#仓库与发布

#License

项目原创代码采用 MIT License。第三方依赖、公开规范、测试 oracle 和契约 fixture 的来源与许可证见 THIRD_PARTY_NOTICES.md

#
CodegenOptions

pub(all) struct CodegenOptions {
emit_descriptor_functions : Bool
emit_runtime_helpers : Bool
} derive(Eq,
Debug
)

Options for emitting MoonBit source from a parsed proto file.

#
CodegenOptions::default

Default generator settings used by examples and tests.

#
DecodeBytesResult

pub(all) enum DecodeBytesResult {
BytesOk(Bytes, Int)
BytesErr(DecodeError)
} derive(Eq,
Debug
)

Result of reading a length-delimited payload.

#
DecodeError

pub(all) enum DecodeError {
UnexpectedEof
VarintOverflow
InvalidWireType(Int)
InvalidFieldNumber(Int)
NegativeLength
Unsupported(String)
} derive(Eq,
Debug
)

Error values returned by the small, allocation-light protobuf readers.

#
DecodeMessageResult

pub(all) enum DecodeMessageResult {
DecodeMessageOk(MessageValue, Int)
DecodeMessageErr(DecodeError)
} derive(Eq,
Debug
)

offset after the decoded message.

#
DecodeStringResult

pub(all) enum DecodeStringResult {
StringOk(String, Int)
StringErr(DecodeError)
} derive(Eq,
Debug
)

Result of reading a string.

#
DecodeU32Result

pub(all) enum DecodeU32Result {
U32Ok(UInt, Int)
U32Err(DecodeError)
} derive(Eq,
Debug
)

Result of reading a 32-bit little-endian fixed value.

#
DecodeU64FixedResult

pub(all) enum DecodeU64FixedResult {
U64FixedOk(UInt64, Int)
U64FixedErr(DecodeError)
} derive(Eq,
Debug
)

Result of reading a 64-bit little-endian fixed value.

#
DecodeU64Result

pub(all) enum DecodeU64Result {
U64Ok(UInt64, Int)
U64Err(DecodeError)
} derive(Eq,
Debug
)

Result of reading an unsigned protobuf varint.

#
EncodeMessageResult

pub(all) enum EncodeMessageResult {
EncodeMessageOk(Bytes)
EncodeMessageErr(DecodeError)
} derive(Eq,
Debug
)

Result of encoding a dynamic message.

#
EnumDescriptor

pub(all) struct EnumDescriptor {
name : String
values : Array[EnumValueDescriptor]
allow_alias : Bool
} derive(Eq,
Debug
)

A parsed enum descriptor.

#
EnumValueDescriptor

pub(all) struct EnumValueDescriptor {
name : String
number : Int
} derive(Eq,
Debug
)

A parsed enum value descriptor.

#
FieldDescriptor

pub(all) struct FieldDescriptor {
name : String
typ : ScalarType
number : Int
label : FieldLabel
} derive(Eq,
Debug
)

A parsed field descriptor.

#
FieldLabel

pub(all) enum FieldLabel {
Singular
Optional
Repeated
Oneof(String)
} derive(Eq,
Debug
)

Field cardinality as written in a proto3 schema.

#
JsonDecodeResult

pub(all) enum JsonDecodeResult {
JsonDecodeOk(MessageValue)
JsonDecodeErr(DecodeError)
} derive(Eq,
Debug
)

Result of parsing protobuf-style JSON into a dynamic message.

#
JsonEncodeResult

pub(all) enum JsonEncodeResult {
JsonOk(String)
JsonErr(DecodeError)
} derive(Eq,
Debug
)

Result of converting a dynamic message to protobuf-style JSON.

#
MessageDescriptor

pub(all) struct MessageDescriptor {
name : String
fields : Array[FieldDescriptor]
} derive(Eq,
Debug
)

A parsed message descriptor.

#
MessageField

pub(all) struct MessageField {
name : String
values : Array[ProtoValue]
} derive(Eq,
Debug
)

one value.

#
MessageValue

pub(all) struct MessageValue {
fields : Array[MessageField]
} derive(Eq,
Debug
)

Dynamic message value consumed and produced by the schema-driven runtime.

#
ParseKeyResult

pub(all) enum ParseKeyResult {
KeyOk(Int, WireType)
KeyErr(DecodeError)
} derive(Eq,
Debug
)

Result of splitting a protobuf key into field number and wire type.

#
ParseProtoResult

pub(all) enum ParseProtoResult {
ProtoOk(ProtoFile)
ProtoErr(SchemaError)
} derive(Eq,
Debug
)

Result of parsing a proto file.

#
ProtoFile

pub(all) struct ProtoFile {
syntax : String
package_name : String
messages : Array[MessageDescriptor]
enums : Array[EnumDescriptor]
message_reservations : Array[ReservedDescriptor]
enum_reservations : Array[ReservedDescriptor]
} derive(Eq,
Debug
)

Top-level proto file model used by parser and codegen.

#
ProtoFile::empty

fn ProtoFile::empty() -> ProtoFile

Empty proto file constructor used by tests and future codegen.

#
ProtoToken

pub(all) enum ProtoToken {
Ident(String)
IntLit(Int)
StrLit(String)
Sym(String)
} derive(Eq,
Debug
)

Tokens for a deliberately small proto3 parser.

#
ProtoValue

pub(all) enum ProtoValue {
UInt64Value(UInt64)
Int64Value(Int64)
FloatValue(Float)
DoubleValue(Double)
BoolValue(Bool)
StringValue(String)
BytesValue(Bytes)
NestedMessageValue(MessageValue)
MapEntryValue(MessageValue)
} derive(Eq,
Debug
)

fixed32/fixed64 without a large amount of boxing.

#
ReservedDescriptor

pub(all) struct ReservedDescriptor {
owner : String
numbers : Array[ReservedNumberRange]
names : Array[String]
} derive(Eq,
Debug
)

Reserved numbers/names attached to a top-level message or enum.

#
ReservedNumberRange

pub(all) struct ReservedNumberRange {
start : Int
end : Int
} derive(Eq,
Debug
)

A reserved numeric interval written by a proto reserved declaration.

#
ScalarType

pub(all) enum ScalarType {
DoubleType
FloatType
Int32Type
Int64Type
UInt32Type
UInt64Type
SInt32Type
SInt64Type
Fixed32Type
Fixed64Type
SFixed32Type
SFixed64Type
BoolType
StringType
BytesType
EnumType(String)
NamedType(String)
MapType(ScalarType, ScalarType)
} derive(Eq,
Debug
)

Proto3 scalar types supported by the stage-1 runtime/codegen plan.

#
ScalarType::wire_type

fn ScalarType::wire_type(self : ScalarType) -> WireType

Map a scalar type to its protobuf wire type.

#
SchemaError

pub(all) enum SchemaError {
Expected(String)
UnexpectedEofInSchema
} derive(Eq,
Debug
)

Parser error type for schema parsing.

#
SchemaIssue

pub(all) struct SchemaIssue {
path : String
message : String
} derive(Eq,
Debug
)

One schema validation issue with a stable path for diagnostics.

#
SchemaValidationResult

pub(all) enum SchemaValidationResult {
SchemaValid
SchemaInvalid(Array[SchemaIssue])
} derive(Eq,
Debug
)

Result of validating a proto schema model.

#
WireType

pub(all) enum WireType {
Varint
Fixed64
LengthDelimited
StartGroup
EndGroup
Fixed32
} derive(Eq,
Debug
)

Protobuf wire type numbers as defined by the encoding specification.

#
WireType::number

fn WireType::number(self : WireType) -> Int

Numeric tag used in the low three bits of a protobuf key.

#
base64_decode

fn base64_decode(input : String) -> DecodeBytesResult

Decode base64 used by protobuf JSON bytes fields. In addition to the standard alphabet, protobuf JSON parsers accept URL-safe -/_ variants and missing final padding.

#
base64_encode

fn base64_encode(input : Bytes) -> String

Encode bytes with the standard base64 alphabet used by protobuf JSON.

#
concat_bytes

fn concat_bytes(parts : Array[Bytes]) -> Bytes

Concatenate byte buffers without exposing mutable implementation details.

#
decode_fixed32

fn decode_fixed32(input : Bytes, offset? : Int) -> DecodeU32Result

Decode a fixed32 value from protobuf little-endian order.

#
decode_fixed64

fn decode_fixed64(input : Bytes, offset? : Int) -> DecodeU64FixedResult

Decode a fixed64 value from protobuf little-endian order.

#
decode_length_delimited

fn decode_length_delimited(input : Bytes, offset? : Int) -> DecodeBytesResult

Decode a length-delimited payload from input.

#
decode_message

fn decode_message(desc : MessageDescriptor, input : Bytes, offset? : Int) -> DecodeMessageResult

singular fields keep the last value.

#
decode_message_with_descriptors

fn decode_message_with_descriptors(desc : MessageDescriptor, descriptors : Array[MessageDescriptor], input : Bytes, offset? : Int) -> DecodeMessageResult

Decode a dynamic message, resolving NamedType fields through descriptors for message-valued nested fields.

#
decode_string_lossy

fn decode_string_lossy(input : Bytes, offset? : Int) -> DecodeStringResult

Decode a UTF-8 string lossily from length-delimited protobuf data.

#
decode_varint_u64

fn decode_varint_u64(input : Bytes, offset? : Int) -> DecodeU64Result

Decode a UInt64 varint from input starting at offset.

#
decode_zigzag_i64

fn decode_zigzag_i64(value : UInt64) -> Int64

Decode a zig-zag UInt64 into Int64.

#
decode_zigzag_int

fn decode_zigzag_int(value : UInt) -> Int

Decode a zig-zag UInt into an Int.

#
encode_bool_field

fn encode_bool_field(field_number : Int, value : Bool) -> Bytes

Encode a bool field payload with its key.

#
encode_bytes_field

fn encode_bytes_field(field_number : Int, value : Bytes) -> Bytes

Encode a bytes field payload with its key.

#
encode_fixed32

fn encode_fixed32(value : UInt) -> Bytes

Encode a fixed32 value in protobuf little-endian order.

#
encode_fixed64

fn encode_fixed64(value : UInt64) -> Bytes

Encode a fixed64 value in protobuf little-endian order.

#
encode_key

fn encode_key(field_number : Int, wire : WireType) -> Bytes

Encode a key as a protobuf varint.

#
encode_length_delimited

fn encode_length_delimited(payload : Bytes) -> Bytes

Encode a length-delimited payload: varint length followed by bytes.

#
encode_message

fn encode_message(desc : MessageDescriptor, value : MessageValue) -> EncodeMessageResult

golden-vector tests easy to read.

#
encode_message_with_descriptors

fn encode_message_with_descriptors(desc : MessageDescriptor, descriptors : Array[MessageDescriptor], value : MessageValue) -> EncodeMessageResult

Encode a dynamic message, resolving NamedType fields through descriptors for message-valued nested fields.

#
encode_sint64_field

fn encode_sint64_field(field_number : Int, value : Int64) -> Bytes

Encode a sint64 field payload with protobuf zig-zag mapping and key.

#
encode_string

fn encode_string(value : String) -> Bytes

Encode a UTF-8 string as length-delimited protobuf data.

#
encode_string_field

fn encode_string_field(field_number : Int, value : String) -> Bytes

Encode a string field payload with its key.

#
encode_uint64_field

fn encode_uint64_field(field_number : Int, value : UInt64) -> Bytes

Encode uint32/uint64 field payload with its key.

#
encode_varint_u64

fn encode_varint_u64(value : UInt64) -> Bytes

Encode a UInt64 using protobuf's unsigned base-128 varint format.

#
encode_varint_uint

fn encode_varint_uint(value : UInt) -> Bytes

Encode a UInt32/UInt as a protobuf varint.

#
encode_zigzag_i64

fn encode_zigzag_i64(value : Int64) -> UInt64

Encode an Int64 with protobuf zig-zag mapping.

#
encode_zigzag_int

fn encode_zigzag_int(value : Int) -> UInt

Encode a signed 32-bit-ish Int with protobuf zig-zag mapping.

#
generate_descriptor_function

fn generate_descriptor_function(desc : MessageDescriptor) -> String

code and passed back to encode_message / decode_message.

#
generate_enum

fn generate_enum(desc : EnumDescriptor) -> String

Emit a MoonBit enum declaration for one enum descriptor.

#
generate_enum_descriptor_function

fn generate_enum_descriptor_function(desc : EnumDescriptor) -> String

Emit a descriptor function for one enum descriptor.

#
generate_enum_descriptor_registry

fn generate_enum_descriptor_registry(file : ProtoFile) -> String

Emit a registry function used by generated JSON helpers to resolve enum descriptors and render protobuf JSON enum names.

#
generate_message_descriptor_registry

fn generate_message_descriptor_registry(file : ProtoFile) -> String

Emit a registry function used by generated runtime helpers to resolve nested message descriptors.

#
generate_message_runtime_helpers

fn generate_message_runtime_helpers(desc : MessageDescriptor) -> String

Emit dynamic encode/decode/JSON helper functions for one generated message.

#
generate_message_struct

fn generate_message_struct(desc : MessageDescriptor) -> String

Emit a MoonBit struct declaration for one message descriptor.

#
generate_moonbit_source

fn generate_moonbit_source(file : ProtoFile, options? : CodegenOptions) -> String

Generate MoonBit declarations for all messages in a parsed proto file.

#
is_valid_field_number

fn is_valid_field_number(number : Int) -> Bool

Protobuf field numbers must be positive, at most 2^29 - 1, and must not fall inside the implementation-reserved 19000..19999 range.

#
is_valid_map_key_type

fn is_valid_map_key_type(typ : ScalarType) -> Bool

Protobuf map keys may be integral, bool or string scalar types.

#
json_escape_string

fn json_escape_string(value : String) -> String

Escape a MoonBit string as a JSON string literal.

#
json_to_message

fn json_to_message(desc : MessageDescriptor, input : String) -> JsonDecodeResult

Parse a protobuf-style JSON object into a dynamic message. Known fields are accepted in any JSON order and returned in descriptor order.

#
json_to_message_with_descriptors

fn json_to_message_with_descriptors(desc : MessageDescriptor, descriptors : Array[MessageDescriptor], input : String) -> JsonDecodeResult

Parse protobuf-style JSON, resolving NamedType fields through descriptors for nested message values.

#
json_to_message_with_schema

fn json_to_message_with_schema(desc : MessageDescriptor, descriptors : Array[MessageDescriptor], enum_descriptors : Array[EnumDescriptor], input : String) -> JsonDecodeResult

Parse protobuf-style JSON while resolving both nested message descriptors and enum value names.

#
lex_proto

fn lex_proto(src : String) -> Array[ProtoToken]

Lex a useful proto3 subset: identifiers, integers, strings and punctuation.

#
make_key

fn make_key(field_number : Int, wire : WireType) -> UInt64

Build a protobuf key: (field_number << 3) | wire_type.

#
map_entry_value

fn map_entry_value(key : ProtoValue, value : ProtoValue) -> ProtoValue

Convenience constructor for a protobuf map entry value. Map fields are represented as repeated MapEntryValue items in the dynamic API.

#
message_field

fn message_field(name : String, value : ProtoValue) -> MessageField

Convenience constructor for a singular field.

#
message_to_json

fn message_to_json(desc : MessageDescriptor, value : MessageValue) -> JsonEncodeResult

Convert a dynamic message to deterministic protobuf-style JSON. Known fields are emitted in descriptor order; absent fields are omitted.

#
message_to_json_lower_camel

fn message_to_json_lower_camel(desc : MessageDescriptor, value : MessageValue) -> JsonEncodeResult

Convert a dynamic message to deterministic protobuf-style JSON using lowerCamelCase JSON field names, matching the protobuf JSON default.

#
message_to_json_lower_camel_with_descriptors

fn message_to_json_lower_camel_with_descriptors(desc : MessageDescriptor, descriptors : Array[MessageDescriptor], value : MessageValue) -> JsonEncodeResult

Convert protobuf-style JSON with lowerCamelCase field names while resolving NamedType fields through descriptors.

#
message_to_json_lower_camel_with_schema

fn message_to_json_lower_camel_with_schema(desc : MessageDescriptor, descriptors : Array[MessageDescriptor], enum_descriptors : Array[EnumDescriptor], value : MessageValue) -> JsonEncodeResult

Convert lowerCamelCase protobuf-style JSON using enum value names when matching EnumDescriptors are available.

#
message_to_json_with_descriptors

fn message_to_json_with_descriptors(desc : MessageDescriptor, descriptors : Array[MessageDescriptor], value : MessageValue) -> JsonEncodeResult

Convert a dynamic message to deterministic protobuf-style JSON, resolving NamedType fields through descriptors for nested message values.

#
message_to_json_with_schema

fn message_to_json_with_schema(desc : MessageDescriptor, descriptors : Array[MessageDescriptor], enum_descriptors : Array[EnumDescriptor], value : MessageValue) -> JsonEncodeResult

Convert protobuf-style JSON using enum value names when matching EnumDescriptors are available.

#
message_value

fn message_value(fields : Array[MessageField]) -> MessageValue

Convenience constructor for a dynamic message.

#
parse_key

fn parse_key(raw : UInt64) -> ParseKeyResult

Parse a raw protobuf key varint value.

#
parse_proto

fn parse_proto(src : String) -> ParseProtoResult

Parse a proto3 subset: syntax/package declarations, messages, fields and top-level enums.

#
repeated_message_field

fn repeated_message_field(name : String, values : Array[ProtoValue]) -> MessageField

Convenience constructor for a repeated field.

#
scalar_type_from_string

fn scalar_type_from_string(name : String) -> ScalarType

Convert a schema type token into a scalar or named message type.

#
schema_is_valid

fn schema_is_valid(file : ProtoFile) -> Bool

#
validate_enum_descriptor

fn validate_enum_descriptor(enum_desc : EnumDescriptor) -> SchemaValidationResult

#
validate_message_descriptor

fn validate_message_descriptor(message : MessageDescriptor) -> SchemaValidationResult

#
validate_proto_file

fn validate_proto_file(file : ProtoFile) -> SchemaValidationResult

#
wire_type_from_number

fn wire_type_from_number(n : Int) -> WireType?

Convert a raw wire type number to a typed value.