🥮 mooncakes.io

moon-robots

RFC 9309 robots.txt parser, matcher, serializer and audit toolkit for MoonBit.

robots
robots-txt
rfc9309
parser
matcher
audit
moonbit
moon add liuhang0001/moon-robots@0.1.1
Download zip
Version
0.1.1
License
Apache-2.0
Repository
Last updated
4 days ago
Downloads
4
README

#moon-robots

  • Module: liuhang0001/moon-robots
  • Version: 0.1.1
  • Status: first public release
  • Repository: https://github.com/liuhang0001/moon-robots
  • Maintainer: liuhang0001

#Overview

moon-robots is a strict, local-only RFC 9309 robots.txt parser, matcher, serializer and audit toolkit for MoonBit.

It turns existing robots.txt text into a semantic model, selects the applicable User-agent group for a crawler product token, evaluates URI paths with RFC-style pattern matching, emits deterministic robots.txt text, and reports lightweight advisory audit findings.

#Why

Robots exclusion logic is easy to under-specify. This project keeps the MVP focused on the core interpretation path:

robots.txt -> parse -> groups/rules -> product-token selection -> path matching -> access decision

#RFC 9309 Scope

Implemented:

  • User-agent, Allow, Disallow
  • comments, inline comments and blank lines
  • CR, LF and CRLF inputs
  • UTF-8 text
  • multiple matching groups merged by equal specificity
  • wildcard *
  • end marker $
  • percent-encoding normalization for unreserved octets
  • longest matching rule, with Allow winning equal-length ties
  • implicit allow for /robots.txt
  • deterministic serializer
  • lightweight audit
  • defensive resource limits

Recognized but non-normative extension:

  • Sitemap

Unknown records are preserved as OtherRecord; they do not alter group parsing or matching.

#Quick Start

let text = "User-agent: *\nDisallow: /private/\nAllow: /private/public/\n"
match parse_robots(text) {
  Ok(file) => {
    let decision = evaluate(file, "ExampleBot", "/private/public/page").unwrap()
    println(decision.summary())
  }
  Err(e) => println(e.to_display())
}

#Parsing

Use parse_robots(input) for default limits or parse_robots_with_limits(input, limits) for explicit limits.

Every public parser returns Result[RobotsFile, RobotsError]; external input must not panic.

#Checking Access

Use evaluate(file, product_token, uri_path) for full decision metadata, or can_fetch(file, product_token, uri_path) for a boolean convenience API.

#Pattern Matching

Matching is case-sensitive for paths. Product tokens use case-insensitive exact matching, with * used only as a fallback. Within path rules, * matches zero or more characters and $ anchors the pattern to the end of the input path. Raw non-ASCII text is converted to percent-encoded UTF-8 octets; percent-encoded unreserved ASCII is decoded for comparison, while reserved octets remain encoded.

#Serialization

serialize_robots(file) emits deterministic LF-terminated robots.txt text using canonical field names:

  • User-agent
  • Allow
  • Disallow
  • Sitemap

Whitespace round-tripping is intentionally out of scope.

#Audit

audit_robots(file) reports advisory issues such as duplicate rules, duplicate agents, empty rules, broad Disallow: /, suspicious patterns and unknown records. Audit findings do not make otherwise valid robots.txt input invalid.

#CLI

The portable CLI is robots-tool.

Commands:

  • parse
  • validate
  • check
  • format
  • audit
  • stats

Target-portable input uses --text:

robots-tool check --agent ExampleBot --path /private/page --text "User-agent: *\nDisallow: /private/\n"

CLI output is deterministic JSON for command results and errors.

#Examples

Four executable examples are included:

  • examples/parse
  • examples/check_access
  • examples/multi_bot
  • examples/audit

#Testing

The test suite covers parser behavior, comments, newlines, group selection, wildcard fallback, merged groups, allow/disallow decisions, longest match, equal-length ties, *, $, percent encoding, UTF-8, unknown records, serializer, audit, limits, property tests and truncation safety.

#Security

robots.txt is not an access-control system. Disallow does not protect private resources. This library only interprets a policy document supplied by the caller. It does not fetch URLs, crawl the web, parse HTML, resolve DNS, perform TLS, or enforce authorization.

#Limitations

This MVP does not implement a web crawler, HTTP client, sitemap XML parser, full RFC 3986 URL library, database, cache, scheduler, search engine, indexing layer, DNS or TLS.

#License

Apache-2.0.

#Release Status

Version 0.1.1 fixes RFC 9309 product-token selection and Unicode path normalization.

#
RobotsError

source
pub(all) suberror RobotsError {
  RobotsError(RobotsErrorStage, RobotsErrorKind, Int, Int, String)
}

#
RobotsError::byte_offset

source
fn RobotsError::byte_offset(self : RobotsError) -> Int

#
RobotsError::context

source
fn RobotsError::context(self : RobotsError) -> String

#
RobotsError::kind

source

#
RobotsError::line

source
fn RobotsError::line(self : RobotsError) -> Int

#
RobotsError::stage

source

#
RobotsError::to_display

source
fn RobotsError::to_display(self : RobotsError) -> String

#
AccessDecision

source
pub struct AccessDecision {
  allowed : Bool
  product_token : String
  uri_path : String
  normalized_path : String
  matched_rule : RobotsRule?
  matched_length : Int
  matched_groups : Int
  reason : String
} derive(Eq, 
Debug)

#
AccessDecision::allowed

source
fn AccessDecision::allowed(self : AccessDecision) -> Bool

#
AccessDecision::matched_groups

source
fn AccessDecision::matched_groups(self : AccessDecision) -> Int

#
AccessDecision::matched_length

source
fn AccessDecision::matched_length(self : AccessDecision) -> Int

#
AccessDecision::matched_rule

source
fn AccessDecision::matched_rule(self : AccessDecision) -> RobotsRule?

#
AccessDecision::normalized_path

source
fn AccessDecision::normalized_path(self : AccessDecision) -> String

#
AccessDecision::product_token

source
fn AccessDecision::product_token(self : AccessDecision) -> String

#
AccessDecision::reason

source
fn AccessDecision::reason(self : AccessDecision) -> String

#
AccessDecision::summary

source
fn AccessDecision::summary(self : AccessDecision) -> String

#
AccessDecision::uri_path

source
fn AccessDecision::uri_path(self : AccessDecision) -> String

#
AuditIssue

source
pub struct AuditIssue {
  severity : AuditSeverity
  kind : AuditKind
  line : Int
  message : String
} derive(Eq, 
Debug)

#
AuditIssue::kind

source
fn AuditIssue::kind(self : AuditIssue) -> AuditKind

#
AuditIssue::line

source
fn AuditIssue::line(self : AuditIssue) -> Int

#
AuditIssue::message

source
fn AuditIssue::message(self : AuditIssue) -> String

#
AuditIssue::severity

source
fn AuditIssue::severity(self : AuditIssue) -> AuditSeverity

#
AuditKind

source
pub(all) enum AuditKind {
  EmptyUserAgent
  RuleBeforeFirstGroup
  EmptyRule
  DuplicateRule
  DuplicateUserAgent
  UnknownRecord
  VeryBroadDisallow
  UnusedGroup
  SuspiciousPattern
} derive(Eq, 
Debug)

#
AuditKind::label

source
fn AuditKind::label(self : AuditKind) -> String

#
AuditSeverity

source
pub(all) enum AuditSeverity {
  Info
  Warning
} derive(Eq, 
Debug)

#
AuditSeverity::label

source
fn AuditSeverity::label(self : AuditSeverity) -> String

#
Limits

source
pub struct Limits {
  max_input_bytes : Int
  max_lines : Int
  max_groups : Int
  max_user_agents_per_group : Int
  max_rules_per_group : Int
  max_pattern_bytes : Int
  max_record_bytes : Int
} derive(Eq, 
Debug)

#
Limits::custom

source
fn Limits::custom(max_input_bytes : Int, max_lines : Int, max_groups : Int, max_user_agents_per_group : Int, max_rules_per_group : Int, max_pattern_bytes : Int, max_record_bytes : Int) -> Limits

#
Limits::default

source
fn Limits::default() -> Limits

#
Limits::max_groups

source
fn Limits::max_groups(self : Limits) -> Int

#
Limits::max_input_bytes

source
fn Limits::max_input_bytes(self : Limits) -> Int

#
Limits::max_lines

source
fn Limits::max_lines(self : Limits) -> Int

#
Limits::max_pattern_bytes

source
fn Limits::max_pattern_bytes(self : Limits) -> Int

#
Limits::max_record_bytes

source
fn Limits::max_record_bytes(self : Limits) -> Int

#
Limits::max_rules_per_group

source
fn Limits::max_rules_per_group(self : Limits) -> Int

#
Limits::max_user_agents_per_group

source
fn Limits::max_user_agents_per_group(self : Limits) -> Int

#
Limits::permissive

source
fn Limits::permissive() -> Limits

#
Limits::strict

source
fn Limits::strict() -> Limits

#
OtherRecord

source
pub struct OtherRecord {
  name : String
  value : String
  line : Int
  byte_offset : Int
} derive(Eq, 
Debug)

#
OtherRecord::byte_offset

source
fn OtherRecord::byte_offset(self : OtherRecord) -> Int

#
OtherRecord::canonical_name

source
fn OtherRecord::canonical_name(self : OtherRecord) -> String

#
OtherRecord::line

source
fn OtherRecord::line(self : OtherRecord) -> Int

#
OtherRecord::name

source
fn OtherRecord::name(self : OtherRecord) -> String

#
OtherRecord::value

source
fn OtherRecord::value(self : OtherRecord) -> String

#
RecordKind

source
pub(all) enum RecordKind {
  UserAgentRecord
  AllowRecord
  DisallowRecord
  SitemapRecord
  OtherRecordKind
} derive(Eq, 
Debug)

#
RobotsErrorKind

source
pub(all) enum RobotsErrorKind {
  EmptyInput
  MissingColon
  EmptyUserAgent
  InvalidUserAgent
  InvalidRecordName
  InvalidPattern
  InvalidPercentEncoding
  InvalidUtf8
  LimitExceeded
  RuleBeforeFirstGroup
  ControlCharacter
} derive(Eq)

#
RobotsErrorKind::to_string

source
fn RobotsErrorKind::to_string(self : RobotsErrorKind) -> String

#
RobotsErrorStage

source
pub(all) enum RobotsErrorStage {
  Input
  Line
  UserAgent
  Rule
  Pattern
  Utf8
  Limit
} derive(Eq)

#
RobotsErrorStage::to_string

source
fn RobotsErrorStage::to_string(self : RobotsErrorStage) -> String

#
RobotsFile

source
pub struct RobotsFile {
  groups : Array[RobotsGroup]
  other_records : Array[OtherRecord]
  comments : Int
  blank_lines : Int
  source_lines : Int
} derive(Eq, 
Debug)

#
RobotsFile::blank_lines

source
fn RobotsFile::blank_lines(self : RobotsFile) -> Int

#
RobotsFile::comments

source
fn RobotsFile::comments(self : RobotsFile) -> Int

#
RobotsFile::group_count

source
fn RobotsFile::group_count(self : RobotsFile) -> Int

#
RobotsFile::groups

source
fn RobotsFile::groups(self : RobotsFile) -> Array[RobotsGroup]

#
RobotsFile::other_record_count

source
fn RobotsFile::other_record_count(self : RobotsFile) -> Int

#
RobotsFile::other_records

source
fn RobotsFile::other_records(self : RobotsFile) -> Array[OtherRecord]

#
RobotsFile::rule_count

source
fn RobotsFile::rule_count(self : RobotsFile) -> Int

#
RobotsFile::source_lines

source
fn RobotsFile::source_lines(self : RobotsFile) -> Int

#
RobotsFile::stats

source
fn RobotsFile::stats(self : RobotsFile) -> RobotsStats

#
RobotsFile::user_agent_count

source
fn RobotsFile::user_agent_count(self : RobotsFile) -> Int

#
RobotsGroup

source
pub struct RobotsGroup {
  user_agents : Array[String]
  rules : Array[RobotsRule]
  start_line : Int
  end_line : Int
} derive(Eq, 
Debug)

#
RobotsGroup::agent_count

source
fn RobotsGroup::agent_count(self : RobotsGroup) -> Int

#
RobotsGroup::end_line

source
fn RobotsGroup::end_line(self : RobotsGroup) -> Int

#
RobotsGroup::rule_count

source
fn RobotsGroup::rule_count(self : RobotsGroup) -> Int

#
RobotsGroup::rules

source
fn RobotsGroup::rules(self : RobotsGroup) -> Array[RobotsRule]

#
RobotsGroup::start_line

source
fn RobotsGroup::start_line(self : RobotsGroup) -> Int

#
RobotsGroup::user_agents

source
fn RobotsGroup::user_agents(self : RobotsGroup) -> Array[String]

#
RobotsRule

source
pub struct RobotsRule {
  kind : RuleKind
  pattern : String
  line : Int
  byte_offset : Int
} derive(Eq, 
Debug)

#
RobotsRule::byte_offset

source
fn RobotsRule::byte_offset(self : RobotsRule) -> Int

#
RobotsRule::is_empty

source
fn RobotsRule::is_empty(self : RobotsRule) -> Bool

#
RobotsRule::kind

source
fn RobotsRule::kind(self : RobotsRule) -> RuleKind

#
RobotsRule::line

source
fn RobotsRule::line(self : RobotsRule) -> Int

#
RobotsRule::pattern

source
fn RobotsRule::pattern(self : RobotsRule) -> String

#
RobotsStats

source
pub struct RobotsStats {
  groups : Int
  user_agents : Int
  rules : Int
  allows : Int
  disallows : Int
  other_records : Int
  comments : Int
  blank_lines : Int
  source_lines : Int
} derive(Eq, 
Debug)

#
RobotsStats::allows

source
fn RobotsStats::allows(self : RobotsStats) -> Int

#
RobotsStats::blank_lines

source
fn RobotsStats::blank_lines(self : RobotsStats) -> Int

#
RobotsStats::comments

source
fn RobotsStats::comments(self : RobotsStats) -> Int

#
RobotsStats::disallows

source
fn RobotsStats::disallows(self : RobotsStats) -> Int

#
RobotsStats::groups

source
fn RobotsStats::groups(self : RobotsStats) -> Int

#
RobotsStats::other_records

source
fn RobotsStats::other_records(self : RobotsStats) -> Int

#
RobotsStats::rules

source
fn RobotsStats::rules(self : RobotsStats) -> Int

#
RobotsStats::source_lines

source
fn RobotsStats::source_lines(self : RobotsStats) -> Int

#
RobotsStats::user_agents

source
fn RobotsStats::user_agents(self : RobotsStats) -> Int

#
RuleKind

source
pub(all) enum RuleKind {
  Allow
  Disallow
} derive(Eq, 
Debug)

#
RuleKind::directive_name

source
fn RuleKind::directive_name(self : RuleKind) -> String

#
RuleKind::json_name

source
fn RuleKind::json_name(self : RuleKind) -> String

#
RuleKind::label

source
fn RuleKind::label(self : RuleKind) -> String

#
access_decision

source
fn access_decision(allowed : Bool, product_token : String, uri_path : String, normalized_path : String, matched_rule : RobotsRule?, matched_length : Int, matched_groups : Int, reason : String) -> AccessDecision

#
audit_robots

source
fn audit_robots(file : RobotsFile) -> Array[AuditIssue]

#
audit_to_text

source
fn audit_to_text(issues : Array[AuditIssue]) -> String

#
can_fetch

source
fn can_fetch(file : RobotsFile, product_token : String, uri_path : String) -> Bool

#
empty_robots_file

source
fn empty_robots_file() -> RobotsFile

#
evaluate

source
fn evaluate(file : RobotsFile, product_token : String, uri_path : String) -> Result[AccessDecision, RobotsError]

#
is_allowed

source
fn is_allowed(file : RobotsFile, product_token : String, uri_path : String) -> Result[AccessDecision, RobotsError]

#
is_sitemap

source
fn is_sitemap(record : OtherRecord) -> Bool

#
json_string

source
fn json_string(input : String) -> String

#
library_version

source
fn library_version() -> String

#
max_context_bytes

source
fn max_context_bytes() -> Int

#
normalize_path_for_match

source
fn normalize_path_for_match(path : String) -> Result[String, RobotsError]

#
other_record

source
fn other_record(name : String, value : String, line : Int, byte_offset : Int) -> OtherRecord

#
parse_robots

source
fn parse_robots(input : String) -> Result[RobotsFile, RobotsError]

#
parse_robots_with_limits

source
fn parse_robots_with_limits(input : String, limits : Limits) -> Result[RobotsFile, RobotsError]

#
pattern_matches

source
fn pattern_matches(pattern : String, path : String) -> Bool

#
pattern_specificity

source
fn pattern_specificity(pattern : String) -> Int

#
robots_error

source
fn robots_error(stage : RobotsErrorStage, kind : RobotsErrorKind, line : Int, byte_offset : Int, context : String) -> RobotsError

#
robots_file

source
fn robots_file(groups : Array[RobotsGroup], other_records : Array[OtherRecord], comments : Int, blank_lines : Int, source_lines : Int) -> RobotsFile

#
robots_group

source
fn robots_group(user_agents : Array[String], rules : Array[RobotsRule], start_line : Int, end_line : Int) -> RobotsGroup

#
robots_rule

source
fn robots_rule(kind : RuleKind, pattern : String, line : Int, byte_offset : Int) -> RobotsRule

#
rules_for

source
fn rules_for(file : RobotsFile, product_token : String) -> Array[RobotsRule]

#
serialize_robots

source
fn serialize_robots(file : RobotsFile) -> String

#
stats_to_text

source
fn stats_to_text(file : RobotsFile) -> String

Source Files

Powered by MoonBit

Site sourceReport issuePackagesBuild queueSkillsStatistics

© 2026 mooncakes.io