🥮 mooncakes.io

robots-gate

审计 robots.txt 策略、解释访问决策、校验站点地图并规划合规抓取任务

robots
robots-txt
crawler
sitemap
audit
moonbit
moon add 2111950632/robots-gate@0.2.1
Download zip
Version
0.2.1
License
Apache-2.0
Repository
Last updated
18 days ago
Downloads
4
README

#MoonRobots Gate

MoonRobots Gate 是一套使用 MoonBit 实现的 Robots Exclusion Protocol 策略工程工具。它可以解析并规范化 robots.txt、解释访问判断、审计策略质量、比较策略版本、校验站点地图交互，并在明确的预算约束下生成确定性抓取计划。

本项目面向爬虫开发者、文档索引器、链接检查器、静态站点工具、CI 流程和 AI 智能体，使它们无须引入完整爬虫框架，也能获得可复现的策略判断。

#当前状态

  • MoonBit 核心库代码 4,284 行。
  • MoonBit 测试代码 1,213 行。
  • 共 102 项测试，覆盖解析、匹配、URL 处理、语义检查、渲染、策略差异、访问矩阵、站点地图、抓取规划和审计报告。
  • 除 moonbitlang/core 外没有其他运行时依赖。
  • moon check、moon build、moon test 和 moon package --list 均已通过。
  • GitHub 与 Mooncakes 命名空间均已独立核对为 2111950632。

#主要能力

#协议解析

  • 解析 User-agent、Allow 和 Disallow 指令。
  • 保存 Sitemap、Crawl-delay、Host、Clean-param 和未知扩展，同时避免扩展指令破坏标准分组。
  • 处理注释、CRLF、UTF-8 BOM、空记录、重复值、格式错误和防御式输入限制。
  • 生成带稳定代码、严重程度、行号、问题说明和修复建议的源位置诊断。

#访问判断

  • 选择匹配度最高的产品令牌。
  • 合并所有同等匹配度的用户代理组。
  • 使用最长路径匹配规则，并在同等长度下由 Allow 胜出。
  • 支持 *、末尾 $、区分大小写的路径，以及对未保留字符百分号编码的 RFC 风格归一化。
  • 对 /robots.txt 提供隐式允许。
  • 返回完整决策轨迹，而不只是布尔值。

#策略工程

  • 语义检查器可发现全站阻断、缺少兜底组、重复和冲突规则、无效百分号转义、错误的 $ 位置、冗余通配符、异常抓取延迟、查询字符串敏感规则和非标准指令。
  • 提供规范化、紧凑化和带源位置注释的渲染结果。
  • 支持用户代理组、规则、站点地图、主机和抓取延迟的结构差异比较。
  • 支持基于调用方探针的行为差异比较。
  • 支持 Markdown 和 CSV 访问矩阵。

#站点地图与抓取规划

  • 解析 XML urlset、XML sitemapindex 和纯文本站点地图。
  • 校验绝对 URL、重复地址、日期前缀、优先级、更新频率和跨域条目。
  • 报告被指定爬虫策略阻止的站点地图 URL。
  • 将站点地图条目转换为带优先级的抓取任务。
  • 按请求数、字节数、深度、单主机配额、Robots 规则、重复 URL 和抓取延迟约束规划任务。
  • 输出顺序稳定的已接受任务和被拒绝任务报告。

#审计报告

  • 在一次操作中完成解析、语义检查和可配置访问矩阵。
  • 生成有上限的风险分数和等级。
  • 为人工审阅输出 Markdown，为 CI 或智能体输出稳定 JSON。

#安装

moon add 2111950632/robots-gate

包命名空间已通过 moon whoami 核对为 2111950632。

#库使用示例

let policy = @robots_gate.parse(
  #|User-agent: *
  #|Disallow: /private/
  #|Allow: /private/public/
  #|Crawl-delay: 2
)

let decision = @robots_gate.decide(
  policy,
  "ExampleBot/1.0",
  "/private/public/index.html",
)

println(decision.summary())
println(@robots_gate.render_decision_trace(decision))

运行完整审计：

let audit = @robots_gate.audit(
  robots_text,
  ["Googlebot", "Bingbot", "GPTBot"],
  ["/", "/admin/", "/api/", "/private/", "/public/"],
)

println(@robots_gate.render_audit_markdown(audit))

校验站点地图并生成抓取计划：

let sitemap = @robots_gate.parse_sitemap(sitemap_text)
let tasks = @robots_gate.tasks_from_sitemap(sitemap, 32768)
let budget = @robots_gate.crawl_budget(100, 10000000, 8, 25, 1000)
let plan = @robots_gate.plan_crawl(policy, "ExampleBot", tasks, budget)

println(@robots_gate.render_crawl_plan(plan))

#命令行工具

CLI 默认使用内置策略和站点地图。可以通过 MOONROBOTS_POLICY 或 MOONROBOTS_SITEMAP 提供自定义内容，无须增加文件系统或网络依赖。

moon run cmd/main -- decide ExampleBot /private/report
moon run cmd/main -- lint
moon run cmd/main -- normalize
moon run cmd/main -- matrix
moon run cmd/main -- sitemap
moon run cmd/main -- plan
moon run cmd/main -- diff
moon run cmd/main -- audit
moon run cmd/main -- audit json
moon run cmd/main -- stats

使用自定义输入的 PowerShell 示例：

$env:MOONROBOTS_POLICY = @"
User-agent: *
Disallow: /internal/
Allow: /internal/public/
Sitemap: https://example.com/sitemap.xml
"@

moon run cmd/main -- audit

#可运行示例

moon run examples/basic
moon run examples/advanced
moon run examples/sitemap
moon run examples/planner

#架构

模块职责
model.mbt公共协议类型、诊断类型、决策类型和统计类型
parser.mbt防御式逐行解析和源位置诊断
matcher.mbt用户代理选择、通配符匹配和决策轨迹
url.mbtHTTP URL 解析及来源、路径辅助函数
lint.mbt语义和可维护性诊断
render.mbt策略、诊断、轨迹和摘要输出
matrix.mbt批量探针和行为比较
diff.mbt结构变化和探针驱动的行为变化
sitemap.mbt站点地图解析、校验和策略过滤
frontier.mbt带预算约束的确定性抓取规划
audit.mbt聚合评分以及 Markdown、JSON 报告

模块边界和数据流详见架构文档。

#标准定位

匹配模型遵循 RFC 9309 的核心规则，包括产品令牌选择、最长路径匹配、同等匹配下的 Allow 优先、区分大小写的路径、百分号编码处理，以及对 /robots.txt 的隐式访问。

Sitemap、Crawl-delay、Host 和 Clean-param 属于扩展指令，而不是 RFC 9309 的规范记录。MoonRobots Gate 会保存并报告这些指令，同时让扩展解析与标准分组边界保持独立。

#范围与非目标

0.2.0 版本是协议与规划核心，明确不负责以下工作：

  • 获取远程资源或跟随重定向。
  • 执行 DNS、TLS、HTTP 缓存或内容解码。
  • 实际执行抓取或持久化分布式任务队列。
  • 实现通用 XML 解析器。
  • 宣称与所有搜索引擎扩展逐字节兼容。

调用方负责网络传输和持久化。这样的边界使核心库可以移植到不同 MoonBit 目标，并保证策略行为在测试、CI 和 WebAssembly 应用中保持确定。

#生态查重

选题检查范围包括：

  • 较早一次 MoonBit 公开活动展示的 30 个项目，该页面仅作为历史生态查重来源。
  • Mooncakes 中与 robots、robots.txt、爬虫、站点地图和 SEO 有关的搜索。
  • GitHub 中将 MoonBit 与 robots.txt、User-agent、Crawl-delay、爬虫和站点地图组合的搜索。

公开结果中未发现专门面向 Robots 协议策略工程和抓取治理的 MoonBit 包或展示项目。相邻项目 MoonSEO 用于生成和审计 SEO 落地页，并不解析 robots.txt、判断爬虫访问权限、校验站点地图与策略冲突，也不规划受策略约束的抓取任务。

详细且带日期的搜索记录见生态查重报告。

#开发

moon info
moon fmt
moon check
moon build
moon test
moon package --list

CI 会在推送和拉取请求中运行同一套验证流程。

#文档

#许可证

Apache-2.0。

#
AccessProbe

source
pub struct AccessProbe {
  name : String
  agent : String
  path : String
} derive(Eq, 
Debug)

A named user-agent and path probe.

#
AccessProbe::agent

source
fn AccessProbe::agent(self : AccessProbe) -> String

#
AccessProbe::name

source
fn AccessProbe::name(self : AccessProbe) -> String

#
AccessProbe::path

source
fn AccessProbe::path(self : AccessProbe) -> String

#
BehaviorChange

source
pub(all) enum BehaviorChange {
  UnchangedAllow
  UnchangedDeny
  BecameAllowed
  BecameDenied
  MissingBaseline
} derive(Eq, 
Debug)

Behavior classification used when comparing two matrices.

#
BehaviorChange::label

source
fn BehaviorChange::label(self : BehaviorChange) -> String

#
ChangeKind

source
pub(all) enum ChangeKind {
  GroupAdded
  GroupRemoved
  RuleAdded
  RuleRemoved
  SitemapAdded
  SitemapRemoved
  HostAdded
  HostRemoved
  DelayChanged
  AccessBecameAllowed
  AccessBecameDenied
} derive(Eq, 
Debug)

Structural or behavioral policy change.

#
ChangeKind::is_behavioral

source
fn ChangeKind::is_behavioral(self : ChangeKind) -> Bool

#
ChangeKind::label

source
fn ChangeKind::label(self : ChangeKind) -> String

#
CrawlBudget

source
pub struct CrawlBudget {
  max_requests : Int
  max_bytes : Int
  max_depth : Int
  max_requests_per_host : Int
  fallback_delay_millis : Int
} derive(Eq, 
Debug)

Resource and safety limits for a crawl planning run.

#
CrawlBudget::fallback_delay_millis

source
fn CrawlBudget::fallback_delay_millis(self : CrawlBudget) -> Int

#
CrawlBudget::max_bytes

source
fn CrawlBudget::max_bytes(self : CrawlBudget) -> Int

#
CrawlBudget::max_depth

source
fn CrawlBudget::max_depth(self : CrawlBudget) -> Int

#
CrawlBudget::max_requests

source
fn CrawlBudget::max_requests(self : CrawlBudget) -> Int

#
CrawlBudget::max_requests_per_host

source
fn CrawlBudget::max_requests_per_host(self : CrawlBudget) -> Int

#
CrawlPlan

source
pub struct CrawlPlan {
  accepted : Array[PlannedTask]
  rejected : Array[RejectedTask]
  total_estimated_bytes : Int
  estimated_duration_millis : Int
} derive(Eq, 
Debug)

Output of a deterministic, policy-aware crawl planning pass.

#
CrawlPlan::accepted

source
fn CrawlPlan::accepted(self : CrawlPlan) -> Array[PlannedTask]

#
CrawlPlan::accepted_count

source
fn CrawlPlan::accepted_count(self : CrawlPlan) -> Int

#
CrawlPlan::estimated_duration_millis

source
fn CrawlPlan::estimated_duration_millis(self : CrawlPlan) -> Int

#
CrawlPlan::rejected

source
fn CrawlPlan::rejected(self : CrawlPlan) -> Array[RejectedTask]

#
CrawlPlan::rejected_count

source
fn CrawlPlan::rejected_count(self : CrawlPlan) -> Int

#
CrawlPlan::total_estimated_bytes

source
fn CrawlPlan::total_estimated_bytes(self : CrawlPlan) -> Int

#
CrawlTask

source
pub struct CrawlTask {
  id : String
  url : String
  priority : Int
  depth : Int
  estimated_bytes : Int
  discovered_from : String
} derive(Eq, 
Debug)

Candidate URL submitted to the deterministic crawl planner.

#
CrawlTask::depth

source
fn CrawlTask::depth(self : CrawlTask) -> Int

#
CrawlTask::discovered_from

source
fn CrawlTask::discovered_from(self : CrawlTask) -> String

#
CrawlTask::estimated_bytes

source
fn CrawlTask::estimated_bytes(self : CrawlTask) -> Int

#
CrawlTask::id

source
fn CrawlTask::id(self : CrawlTask) -> String

#
CrawlTask::priority

source
fn CrawlTask::priority(self : CrawlTask) -> Int

#
CrawlTask::url

source
fn CrawlTask::url(self : CrawlTask) -> String

#
Decision

source
pub struct Decision {
  allowed : Bool
  agent : String
  path : String
  normalized_path : String
  rule_kind : RuleKind
  rule_pattern : String
  line : Int
  reason : String
  agent_specificity : Int
  rule_specificity : Int
  matched_groups : Int
  trace : Array[DecisionStep]
} derive(Eq, 
Debug)

Decision returned for a user-agent and path.

#
Decision::agent

source
fn Decision::agent(self : Decision) -> String

#
Decision::agent_specificity

source
fn Decision::agent_specificity(self : Decision) -> Int

#
Decision::allowed

source
fn Decision::allowed(self : Decision) -> Bool

#
Decision::line

source
fn Decision::line(self : Decision) -> Int

#
Decision::matched_groups

source
fn Decision::matched_groups(self : Decision) -> Int

#
Decision::normalized_path

source
fn Decision::normalized_path(self : Decision) -> String

#
Decision::path

source
fn Decision::path(self : Decision) -> String

#
Decision::reason

source
fn Decision::reason(self : Decision) -> String

#
Decision::rule_kind

source
fn Decision::rule_kind(self : Decision) -> RuleKind

#
Decision::rule_pattern

source
fn Decision::rule_pattern(self : Decision) -> String

#
Decision::rule_specificity

source
fn Decision::rule_specificity(self : Decision) -> Int

#
Decision::summary

source
fn Decision::summary(self : Decision) -> String

#
Decision::trace

source
fn Decision::trace(self : Decision) -> Array[DecisionStep]

#
Decision::verdict

source
fn Decision::verdict(self : Decision) -> String

#
DecisionMatrix

source
pub struct DecisionMatrix {
  outcomes : Array[ProbeOutcome]
  allowed_count : Int
  denied_count : Int
  default_count : Int
} derive(Eq, 
Debug)

Deterministic batch of access decisions.

#
DecisionMatrix::all_allowed

source
fn DecisionMatrix::all_allowed(self : DecisionMatrix) -> Bool

#
DecisionMatrix::allowed_count

source
fn DecisionMatrix::allowed_count(self : DecisionMatrix) -> Int

#
DecisionMatrix::default_count

source
fn DecisionMatrix::default_count(self : DecisionMatrix) -> Int

#
DecisionMatrix::denied_count

source
fn DecisionMatrix::denied_count(self : DecisionMatrix) -> Int

#
DecisionMatrix::outcomes

source

#
DecisionMatrix::total

source
fn DecisionMatrix::total(self : DecisionMatrix) -> Int

#
DecisionStep

source
pub struct DecisionStep {
  stage : String
  detail : String
} derive(Eq, 
Debug)

One step in an explainable access decision.

#
DecisionStep::detail

source
fn DecisionStep::detail(self : DecisionStep) -> String

#
DecisionStep::stage

source
fn DecisionStep::stage(self : DecisionStep) -> String

#
DecisionStep::summary

source
fn DecisionStep::summary(self : DecisionStep) -> String

#
Diagnostic

source
pub struct Diagnostic {
  severity : Severity
  code : String
  message : String
  span : SourceSpan
  hint : String
} derive(Eq, 
Debug)

A source-aware parser or policy diagnostic.

#
Diagnostic::code

source
fn Diagnostic::code(self : Diagnostic) -> String

#
Diagnostic::hint

source
fn Diagnostic::hint(self : Diagnostic) -> String

#
Diagnostic::line

source
fn Diagnostic::line(self : Diagnostic) -> Int

#
Diagnostic::message

source
fn Diagnostic::message(self : Diagnostic) -> String

#
Diagnostic::severity

source
fn Diagnostic::severity(self : Diagnostic) -> Severity

#
Diagnostic::span

source
fn Diagnostic::span(self : Diagnostic) -> SourceSpan

#
Diagnostic::summary

source
fn Diagnostic::summary(self : Diagnostic) -> String

#
Directive

source
pub struct Directive {
  kind : DirectiveKind
  key : String
  value : String
  raw : String
  span : SourceSpan
} derive(Eq, 
Debug)

A preserved source directive, including extensions and unknown fields.

#
Directive::is_standard

source
fn Directive::is_standard(self : Directive) -> Bool

#
Directive::key

source
fn Directive::key(self : Directive) -> String

#
Directive::kind

source
fn Directive::kind(self : Directive) -> DirectiveKind

#
Directive::line

source
fn Directive::line(self : Directive) -> Int

#
Directive::raw

source
fn Directive::raw(self : Directive) -> String

#
Directive::span

source
fn Directive::span(self : Directive) -> SourceSpan

#
Directive::value

source
fn Directive::value(self : Directive) -> String

#
DirectiveKind

source
pub(all) enum DirectiveKind {
  UserAgentDirective
  AllowDirective
  DisallowDirective
  SitemapDirective
  CrawlDelayDirective
  HostDirective
  CleanParamDirective
  UnknownDirective
} derive(Eq, 
Debug)

Classification of a parsed robots.txt record.

#
DirectiveKind::label

source
fn DirectiveKind::label(self : DirectiveKind) -> String

#
Group

source
pub struct Group {
  agents : Array[String]
  rules : Array[Rule]
  crawl_delay_millis : Int
  start_line : Int
  end_line : Int
} derive(Eq, 
Debug)

A user-agent group in robots.txt.

#
Group::agent_count

source
fn Group::agent_count(self : Group) -> Int

#
Group::agents

source
fn Group::agents(self : Group) -> Array[String]

#
Group::crawl_delay_millis

source
fn Group::crawl_delay_millis(self : Group) -> Int

#
Group::end_line

source
fn Group::end_line(self : Group) -> Int

#
Group::has_crawl_delay

source
fn Group::has_crawl_delay(self : Group) -> Bool

#
Group::rule_count

source
fn Group::rule_count(self : Group) -> Int

#
Group::rules

source
fn Group::rules(self : Group) -> Array[Rule]

#
Group::start_line

source
fn Group::start_line(self : Group) -> Int

#
LintOptions

source
pub struct LintOptions {
  require_wildcard_group : Bool
  require_sitemap : Bool
  warn_unknown_directives : Bool
  max_crawl_delay_millis : Int
  max_rules_per_group : Int
} derive(Eq, 
Debug)

Configuration for policy linting.

#
LintOptions::max_crawl_delay_millis

source
fn LintOptions::max_crawl_delay_millis(self : LintOptions) -> Int

#
LintOptions::max_rules_per_group

source
fn LintOptions::max_rules_per_group(self : LintOptions) -> Int

#
LintOptions::require_sitemap

source
fn LintOptions::require_sitemap(self : LintOptions) -> Bool

#
LintOptions::require_wildcard_group

source
fn LintOptions::require_wildcard_group(self : LintOptions) -> Bool

#
LintOptions::warn_unknown_directives

source
fn LintOptions::warn_unknown_directives(self : LintOptions) -> Bool

#
MatrixComparison

source
pub struct MatrixComparison {
  name : String
  before : ProbeOutcome?
  after : ProbeOutcome
  change : BehaviorChange
} derive(Eq, 
Debug)

#
MatrixComparison::after

source

#
MatrixComparison::before

source

#
MatrixComparison::change

source

#
MatrixComparison::name

source
fn MatrixComparison::name(self : MatrixComparison) -> String

#
ParseOptions

source
pub struct ParseOptions {
  max_input_chars : Int
  max_lines : Int
  max_line_chars : Int
  max_groups : Int
  max_rules : Int
  preserve_unknown : Bool
} derive(Eq, 
Debug)

Limits used by the defensive parser.

#
ParseOptions::max_groups

source
fn ParseOptions::max_groups(self : ParseOptions) -> Int

#
ParseOptions::max_input_chars

source
fn ParseOptions::max_input_chars(self : ParseOptions) -> Int

#
ParseOptions::max_line_chars

source
fn ParseOptions::max_line_chars(self : ParseOptions) -> Int

#
ParseOptions::max_lines

source
fn ParseOptions::max_lines(self : ParseOptions) -> Int

#
ParseOptions::max_rules

source
fn ParseOptions::max_rules(self : ParseOptions) -> Int

#
ParseOptions::preserve_unknown

source
fn ParseOptions::preserve_unknown(self : ParseOptions) -> Bool

#
ParsedUrl

source
pub struct ParsedUrl {
  scheme : String
  authority : String
  host : String
  port : Int
  path_query : String
  fragment : String
  valid : Bool
  error : String
} derive(Eq, 
Debug)

Parsed HTTP or HTTPS URL used by sitemap and audit helpers.

#
ParsedUrl::authority

source
fn ParsedUrl::authority(self : ParsedUrl) -> String

#
ParsedUrl::error

source
fn ParsedUrl::error(self : ParsedUrl) -> String

#
ParsedUrl::fragment

source
fn ParsedUrl::fragment(self : ParsedUrl) -> String

#
ParsedUrl::host

source
fn ParsedUrl::host(self : ParsedUrl) -> String

#
ParsedUrl::origin

source
fn ParsedUrl::origin(self : ParsedUrl) -> String

#
ParsedUrl::path_query

source
fn ParsedUrl::path_query(self : ParsedUrl) -> String

#
ParsedUrl::port

source
fn ParsedUrl::port(self : ParsedUrl) -> Int

#
ParsedUrl::robots_url

source
fn ParsedUrl::robots_url(self : ParsedUrl) -> String

#
ParsedUrl::scheme

source
fn ParsedUrl::scheme(self : ParsedUrl) -> String

#
ParsedUrl::valid

source
fn ParsedUrl::valid(self : ParsedUrl) -> Bool

#
ParsedUrl::without_fragment

source
fn ParsedUrl::without_fragment(self : ParsedUrl) -> String

#
PlannedTask

source
pub struct PlannedTask {
  task : CrawlTask
  canonical_url : String
  host : String
  path : String
  sequence : Int
  not_before_millis : Int
  decision : Decision
} derive(Eq, 
Debug)

A task accepted into the crawl plan.

#
PlannedTask::canonical_url

source
fn PlannedTask::canonical_url(self : PlannedTask) -> String

#
PlannedTask::decision

source
fn PlannedTask::decision(self : PlannedTask) -> Decision

#
PlannedTask::host

source
fn PlannedTask::host(self : PlannedTask) -> String

#
PlannedTask::not_before_millis

source
fn PlannedTask::not_before_millis(self : PlannedTask) -> Int

#
PlannedTask::path

source
fn PlannedTask::path(self : PlannedTask) -> String

#
PlannedTask::sequence

source
fn PlannedTask::sequence(self : PlannedTask) -> Int

#
PlannedTask::task

source
fn PlannedTask::task(self : PlannedTask) -> CrawlTask

#
Policy

source
pub struct Policy {
  groups : Array[Group]
  sitemaps : Array[String]
  hosts : Array[String]
  directives : Array[Directive]
  diagnostics : Array[Diagnostic]
  source_lines : Int
  truncated : Bool
} derive(Eq, 
Debug)

Parsed robots.txt policy with source metadata and diagnostics.

#
Policy::diagnostic_count

source
fn Policy::diagnostic_count(self : Policy) -> Int

#
Policy::diagnostics

source
fn Policy::diagnostics(self : Policy) -> Array[Diagnostic]

#
Policy::directives

source
fn Policy::directives(self : Policy) -> Array[Directive]

#
Policy::error_count

source
fn Policy::error_count(self : Policy) -> Int

#
Policy::group_count

source
fn Policy::group_count(self : Policy) -> Int

#
Policy::groups

source
fn Policy::groups(self : Policy) -> Array[Group]

#
Policy::hosts

source
fn Policy::hosts(self : Policy) -> Array[String]

#
Policy::is_valid

source
fn Policy::is_valid(self : Policy) -> Bool

#
Policy::rule_count

source
fn Policy::rule_count(self : Policy) -> Int

#
Policy::sitemaps

source
fn Policy::sitemaps(self : Policy) -> Array[String]

#
Policy::source_lines

source
fn Policy::source_lines(self : Policy) -> Int

#
Policy::stats

source
fn Policy::stats(self : Policy) -> PolicyStats

#
Policy::truncated

source
fn Policy::truncated(self : Policy) -> Bool

#
Policy::warning_count

source
fn Policy::warning_count(self : Policy) -> Int

#
PolicyAudit

source
pub struct PolicyAudit {
  policy : Policy
  diagnostics : Array[Diagnostic]
  matrix : DecisionMatrix
  risk_score : Int
  grade : String
  explicit_decisions : Int
  default_decisions : Int
} derive(Eq, 
Debug)

Aggregated, deterministic policy audit suitable for CI output.

#
PolicyAudit::default_decisions

source
fn PolicyAudit::default_decisions(self : PolicyAudit) -> Int

#
PolicyAudit::diagnostics

source
fn PolicyAudit::diagnostics(self : PolicyAudit) -> Array[Diagnostic]

#
PolicyAudit::explicit_decisions

source
fn PolicyAudit::explicit_decisions(self : PolicyAudit) -> Int

#
PolicyAudit::grade

source
fn PolicyAudit::grade(self : PolicyAudit) -> String

#
PolicyAudit::matrix

source
fn PolicyAudit::matrix(self : PolicyAudit) -> DecisionMatrix

#
PolicyAudit::passed

source
fn PolicyAudit::passed(self : PolicyAudit) -> Bool

#
PolicyAudit::policy

source
fn PolicyAudit::policy(self : PolicyAudit) -> Policy

#
PolicyAudit::risk_score

source
fn PolicyAudit::risk_score(self : PolicyAudit) -> Int

#
PolicyChange

source
pub struct PolicyChange {
  kind : ChangeKind
  subject : String
  before : String
  after : String
  line_before : Int
  line_after : Int
} derive(Eq, 
Debug)

One source-aware difference between two policies.

#
PolicyChange::after

source
fn PolicyChange::after(self : PolicyChange) -> String

#
PolicyChange::before

source
fn PolicyChange::before(self : PolicyChange) -> String

#
PolicyChange::kind

source

#
PolicyChange::line_after

source
fn PolicyChange::line_after(self : PolicyChange) -> Int

#
PolicyChange::line_before

source
fn PolicyChange::line_before(self : PolicyChange) -> Int

#
PolicyChange::subject

source
fn PolicyChange::subject(self : PolicyChange) -> String

#
PolicyChange::summary

source
fn PolicyChange::summary(self : PolicyChange) -> String

#
PolicyDiff

source
pub struct PolicyDiff {
  changes : Array[PolicyChange]
  structural_count : Int
  behavioral_count : Int
} derive(Eq, 
Debug)

Complete structural and optional behavioral difference.

#
PolicyDiff::behavioral_count

source
fn PolicyDiff::behavioral_count(self : PolicyDiff) -> Int

#
PolicyDiff::changes

source
fn PolicyDiff::changes(self : PolicyDiff) -> Array[PolicyChange]

#
PolicyDiff::is_empty

source
fn PolicyDiff::is_empty(self : PolicyDiff) -> Bool

#
PolicyDiff::structural_count

source
fn PolicyDiff::structural_count(self : PolicyDiff) -> Int

#
PolicyStats

source
pub struct PolicyStats {
  groups : Int
  agents : Int
  rules : Int
  allows : Int
  disallows : Int
  sitemaps : Int
  diagnostics : Int
} derive(Eq, 
Debug)

Compact aggregate statistics for a policy.

#
PolicyStats::agents

source
fn PolicyStats::agents(self : PolicyStats) -> Int

#
PolicyStats::allows

source
fn PolicyStats::allows(self : PolicyStats) -> Int

#
PolicyStats::diagnostics

source
fn PolicyStats::diagnostics(self : PolicyStats) -> Int

#
PolicyStats::disallows

source
fn PolicyStats::disallows(self : PolicyStats) -> Int

#
PolicyStats::groups

source
fn PolicyStats::groups(self : PolicyStats) -> Int

#
PolicyStats::rules

source
fn PolicyStats::rules(self : PolicyStats) -> Int

#
PolicyStats::sitemaps

source
fn PolicyStats::sitemaps(self : PolicyStats) -> Int

#
ProbeOutcome

source
pub struct ProbeOutcome {
  probe : AccessProbe
  decision : Decision
} derive(Eq, 
Debug)

Result of running one access probe.

#
ProbeOutcome::allowed

source
fn ProbeOutcome::allowed(self : ProbeOutcome) -> Bool

#
ProbeOutcome::decision

source
fn ProbeOutcome::decision(self : ProbeOutcome) -> Decision

#
ProbeOutcome::probe

source

#
ProbeOutcome::summary

source
fn ProbeOutcome::summary(self : ProbeOutcome) -> String

#
RejectedTask

source
pub struct RejectedTask {
  task : CrawlTask
  reason : RejectionReason
  detail : String
} derive(Eq, 
Debug)

A task rejected by one deterministic planning constraint.

#
RejectedTask::detail

source
fn RejectedTask::detail(self : RejectedTask) -> String

#
RejectedTask::reason

source

#
RejectedTask::task

source
fn RejectedTask::task(self : RejectedTask) -> CrawlTask

#
RejectionReason

source
pub(all) enum RejectionReason {
  InvalidUrl
  RobotsDenied
  DepthExceeded
  RequestBudgetExceeded
  ByteBudgetExceeded
  HostBudgetExceeded
  DuplicateUrl
} derive(Eq, 
Debug)

Why a task was rejected by the planner.

#
RejectionReason::label

source
fn RejectionReason::label(self : RejectionReason) -> String

#
RenderStyle

source
pub(all) enum RenderStyle {
  Canonical
  Compact
  Annotated
} derive(Eq, Show, 
Debug)

Formatting style for normalized robots.txt output.

#
RenderStyle::label

source
fn RenderStyle::label(self : RenderStyle) -> String

#
Rule

source
pub struct Rule {
  kind : RuleKind
  pattern : String
  normalized_pattern : String
  line : Int
} derive(Eq, 
Debug)

One parsed Allow or Disallow rule.

#
Rule::is_empty

source
fn Rule::is_empty(self : Rule) -> Bool

#
Rule::kind

source
fn Rule::kind(self : Rule) -> RuleKind

#
Rule::line

source
fn Rule::line(self : Rule) -> Int

#
Rule::normalized_pattern

source
fn Rule::normalized_pattern(self : Rule) -> String

#
Rule::pattern

source
fn Rule::pattern(self : Rule) -> String

#
Rule::specificity

source
fn Rule::specificity(self : Rule) -> Int

#
RuleKind

source
pub(all) enum RuleKind {
  Allow
  Disallow
} derive(Eq, 
Debug)

Whether a matching path is allowed or disallowed.

#
RuleKind::directive_name

source
fn RuleKind::directive_name(self : RuleKind) -> String

#
RuleKind::label

source
fn RuleKind::label(self : RuleKind) -> String

#
RuleKind::tie_priority

source
fn RuleKind::tie_priority(self : RuleKind) -> Int

#
Severity

source
pub(all) enum Severity {
  Info
  Warning
  Error
} derive(Eq, 
Debug)

Severity of a parser or policy diagnostic.

#
Severity::label

source
fn Severity::label(self : Severity) -> String

#
Severity::rank

source
fn Severity::rank(self : Severity) -> Int

#
SitemapDocument

source
pub struct SitemapDocument {
  kind : SitemapKind
  entries : Array[SitemapEntry]
  child_sitemaps : Array[String]
  diagnostics : Array[Diagnostic]
  source_items : Int
} derive(Eq, 
Debug)

Parsed sitemap entries, child indexes, and diagnostics.

#
SitemapDocument::child_count

source
fn SitemapDocument::child_count(self : SitemapDocument) -> Int

#
SitemapDocument::child_sitemaps

source
fn SitemapDocument::child_sitemaps(self : SitemapDocument) -> Array[String]

#
SitemapDocument::diagnostics

source
fn SitemapDocument::diagnostics(self : SitemapDocument) -> Array[Diagnostic]

#
SitemapDocument::entries

source

#
SitemapDocument::entry_count

source
fn SitemapDocument::entry_count(self : SitemapDocument) -> Int

#
SitemapDocument::is_valid

source
fn SitemapDocument::is_valid(self : SitemapDocument) -> Bool

#
SitemapDocument::kind

source

#
SitemapDocument::source_items

source
fn SitemapDocument::source_items(self : SitemapDocument) -> Int

#
SitemapEntry

source
pub struct SitemapEntry {
  location : String
  last_modified : String
  change_frequency : String
  priority_millis : Int
  ordinal : Int
} derive(Eq, 
Debug)

One URL from a sitemap document.

#
SitemapEntry::change_frequency

source
fn SitemapEntry::change_frequency(self : SitemapEntry) -> String

#
SitemapEntry::has_last_modified

source
fn SitemapEntry::has_last_modified(self : SitemapEntry) -> Bool

#
SitemapEntry::has_priority

source
fn SitemapEntry::has_priority(self : SitemapEntry) -> Bool

#
SitemapEntry::last_modified

source
fn SitemapEntry::last_modified(self : SitemapEntry) -> String

#
SitemapEntry::location

source
fn SitemapEntry::location(self : SitemapEntry) -> String

#
SitemapEntry::ordinal

source
fn SitemapEntry::ordinal(self : SitemapEntry) -> Int

#
SitemapEntry::path

source
fn SitemapEntry::path(self : SitemapEntry) -> String

#
SitemapEntry::priority_millis

source
fn SitemapEntry::priority_millis(self : SitemapEntry) -> Int

#
SitemapKind

source
pub(all) enum SitemapKind {
  UrlSet
  SitemapIndex
  TextSitemap
  UnknownSitemap
} derive(Eq, 
Debug)

Detected sitemap representation.

#
SitemapKind::label

source
fn SitemapKind::label(self : SitemapKind) -> String

#
SourceSpan

source
pub struct SourceSpan {
  line : Int
  column : Int
  length : Int
} derive(Eq, 
Debug)

Source location for a line-oriented robots.txt record.

#
SourceSpan::column

source
fn SourceSpan::column(self : SourceSpan) -> Int

#
SourceSpan::length

source
fn SourceSpan::length(self : SourceSpan) -> Int

#
SourceSpan::line

source
fn SourceSpan::line(self : SourceSpan) -> Int

#
SourceSpan::summary

source
fn SourceSpan::summary(self : SourceSpan) -> String

#
access_probe

source
fn access_probe(name : String, agent : String, path : String) -> AccessProbe

#
all_diagnostics

source
fn all_diagnostics(policy : Policy) -> Array[Diagnostic]

#
all_diagnostics_with_options

source
fn all_diagnostics_with_options(policy : Policy, options : LintOptions) -> Array[Diagnostic]

#
allowed_sitemap_entries

source
fn allowed_sitemap_entries(policy : Policy, agent : String, document : SitemapDocument) -> Array[SitemapEntry]

#
annotated_policy

source
fn annotated_policy(policy : Policy) -> String

#
audit

source
fn audit(input : String, agents : Array[String], paths : Array[String]) -> PolicyAudit

Parses, lints, and probes a policy in one deterministic operation.

#
audit_default

source
fn audit_default(input : String) -> PolicyAudit

#
audit_exit_code

source
fn audit_exit_code(audit : PolicyAudit) -> Int

#
behavior_changes

source
fn behavior_changes(diff : PolicyDiff) -> Array[PolicyChange]

#
behavior_flip_count

source
fn behavior_flip_count(comparisons : Array[MatrixComparison]) -> Int

#
blocked_sitemap_entries

source
fn blocked_sitemap_entries(policy : Policy, agent : String, document : SitemapDocument) -> Array[SitemapEntry]

#
build_probe_grid

source
fn build_probe_grid(agents : Array[String], paths : Array[String]) -> Array[AccessProbe]

#
can_fetch

source
fn can_fetch(policy : Policy, agent : String, path : String) -> Bool

Convenience helper that returns only the allow/deny boolean.

#
canonical_url

source
fn canonical_url(input : String) -> String

#
canonicalize

source
fn canonicalize(input : String) -> String

#
compact_policy

source
fn compact_policy(policy : Policy) -> String

#
compare_matrices

source
fn compare_matrices(before : DecisionMatrix, after : DecisionMatrix) -> Array[MatrixComparison]

#
crawl_budget

source
fn crawl_budget(max_requests : Int, max_bytes : Int, max_depth : Int, max_requests_per_host : Int, fallback_delay_millis : Int) -> CrawlBudget

#
crawl_delay

source
fn crawl_delay(policy : Policy, agent : String) -> Int

Returns the applicable crawl delay in milliseconds, or -1 when absent.

#
crawl_task

source
fn crawl_task(id : String, url : String, priority : Int, depth : Int, estimated_bytes : Int) -> CrawlTask

#
cross_origin_entries

source
fn cross_origin_entries(document : SitemapDocument, expected_origin : String) -> Array[SitemapEntry]

#
decide

source
fn decide(policy : Policy, agent : String, path : String) -> Decision

Returns the full matching decision for a user-agent and URL path.

#
default_crawl_budget

source
fn default_crawl_budget() -> CrawlBudget

#
default_lint_options

source
fn default_lint_options() -> LintOptions

#
default_outcomes

source
fn default_outcomes(matrix : DecisionMatrix) -> Array[ProbeOutcome]

#
default_parse_options

source
fn default_parse_options() -> ParseOptions

#
denied_outcomes

source
fn denied_outcomes(matrix : DecisionMatrix) -> Array[ProbeOutcome]

#
diagnostics_at_least

source
fn diagnostics_at_least(diagnostics : Array[Diagnostic], severity : Severity) -> Array[Diagnostic]

#
diff_policies

source
fn diff_policies(before : Policy, after : Policy, probes : Array[AccessProbe]) -> PolicyDiff

Compares policy structure and access behavior on caller-provided probes.

#
discovered_crawl_task

source
fn discovered_crawl_task(id : String, url : String, priority : Int, depth : Int, estimated_bytes : Int, discovered_from : String) -> CrawlTask

#
effective_rules

source
fn effective_rules(policy : Policy, agent : String) -> Array[Rule]

Returns every effective rule from equally specific matching groups.

#
find_outcome

source
fn find_outcome(matrix : DecisionMatrix, name : String) -> ProbeOutcome?

#
has_diagnostic_code

source
fn has_diagnostic_code(diagnostics : Array[Diagnostic], code : String) -> Bool

#
lint

source
fn lint(policy : Policy) -> Array[Diagnostic]

#
lint_with_options

source
fn lint_with_options(policy : Policy, options : LintOptions) -> Array[Diagnostic]

Performs semantic and maintainability checks beyond parsing.

#
normalize_path

source
fn normalize_path(path : String) -> String

Normalizes a URL path for RFC 9309 comparison.

#
normalize_percent_encoding

source
fn normalize_percent_encoding(input : String) -> String

Decodes percent-encoded unreserved ASCII while preserving reserved octets.

#
parse

source
fn parse(input : String) -> Policy

Parses robots.txt with safe defaults.

#
parse_options

source
fn parse_options(max_input_chars : Int, max_lines : Int, max_line_chars : Int, max_groups : Int, max_rules : Int, preserve_unknown : Bool) -> ParseOptions

#
parse_render_style

source
fn parse_render_style(input : String) -> Result[RenderStyle, String]

#
parse_sitemap

source
fn parse_sitemap(input : String) -> SitemapDocument

Parses XML urlsets, XML sitemap indexes, or line-oriented text sitemaps.

#
parse_url

source
fn parse_url(input : String) -> ParsedUrl

Parses an absolute HTTP or HTTPS URL without network access.

#
parse_with_options

source
fn parse_with_options(input : String, options : ParseOptions) -> Policy

Parses robots.txt using defensive limits and preserves diagnostics.

#
plan_crawl

source
fn plan_crawl(policy : Policy, agent : String, tasks : Array[CrawlTask], budget : CrawlBudget) -> CrawlPlan

Plans crawl order by priority while enforcing protocol and budget limits.

#
rejected_for_reason

source
fn rejected_for_reason(plan : CrawlPlan, reason : RejectionReason) -> Array[RejectedTask]

#
remove_url_fragment

source
fn remove_url_fragment(input : String) -> String

#
render_audit_json

source
fn render_audit_json(audit : PolicyAudit) -> String

#
render_audit_markdown

source
fn render_audit_markdown(audit : PolicyAudit) -> String

#
render_blocked_sitemap_report

source
fn render_blocked_sitemap_report(policy : Policy, agent : String, document : SitemapDocument) -> String

#
render_crawl_plan

source
fn render_crawl_plan(plan : CrawlPlan) -> String

#
render_decision_trace

source
fn render_decision_trace(decision : Decision) -> String

#
render_diagnostics

source
fn render_diagnostics(diagnostics : Array[Diagnostic]) -> String

#
render_diff

source
fn render_diff(diff : PolicyDiff) -> String

#
render_matrix_comparison

source
fn render_matrix_comparison(comparisons : Array[MatrixComparison]) -> String

#
render_matrix_csv

source
fn render_matrix_csv(matrix : DecisionMatrix) -> String

#
render_matrix_markdown

source
fn render_matrix_markdown(matrix : DecisionMatrix) -> String

#
render_policy

source
fn render_policy(policy : Policy, style : RenderStyle) -> String

Serializes the semantic policy into deterministic robots.txt text.

#
render_policy_summary

source
fn render_policy_summary(policy : Policy) -> String

#
render_sitemap_summary

source
fn render_sitemap_summary(document : SitemapDocument) -> String

#
robots_url_for

source
fn robots_url_for(input : String) -> String

#
run_matrix

source
fn run_matrix(policy : Policy, probes : Array[AccessProbe]) -> DecisionMatrix

Evaluates a list of named probes against one policy.

#
same_origin

source
fn same_origin(left : String, right : String) -> Bool

#
strict_lint_options

source
fn strict_lint_options() -> LintOptions

#
strict_parse_options

source
fn strict_parse_options() -> ParseOptions

#
structural_diff

source
fn structural_diff(before : Policy, after : Policy) -> PolicyDiff

#
tasks_from_sitemap

source
fn tasks_from_sitemap(document : SitemapDocument, default_bytes : Int) -> Array[CrawlTask]

#
url_depth

source
fn url_depth(input : String) -> Int

#
url_has_query

source
fn url_has_query(input : String) -> Bool

#
url_path

source
fn url_path(input : String) -> String

Source Files

Powered by MoonBit

Site sourceReport issuePackagesBuild queueSkillsStatistics

© 2026 mooncakes.io