READMEihb2032/MoonFrame/lazy does not have a README file
#LazyFrame
pub struct LazyFrame {
// private fields
}The public face of the lazy layer: LazyFrame wraps a LogicalPlan
and grows it through builder methods that mirror the eager DataFrame
verbs name-for-name. Building is total — every builder just wraps
the plan in one more node, so a LazyFrame can always be constructed,
chained, and explained, even when collecting it would fail. All
computation (and all failure) happens in collect, which optimizes the
plan and then interprets it through the public eager operators:
LazyFrame::LazyFrame(df).f(…).g(…).collect() equals df.f(…).g(…) — the
faithful-deferred-executor contract every test in lazy_test.mbt pins, and
the reason the optimizer's rewrites have to preserve results rather than
merely preserve rows.
Every builder that takes an array copies it, so a plan can never
observe later mutation of a caller's argument — the immutability the
paragraph above promises holds for the arguments too, not just the
captured frames.
The entry point is the type's own constructor, LazyFrame::LazyFrame(df)
— not a DataFrame method, which would have to live in frame and close a
frame ↔ lazy import cycle, and not lazy(df), since lazy is a MoonBit
reserved word.
#LazyFrame::LazyFrame
Wrap an in-memory frame as the leaf of a new plan (a Scan node):
LazyFrame::LazyFrame(df).filter(…).collect(). Total — the frame is
captured as-is, and frames are immutable, so the plan can never observe
later state.
#LazyFrame::collect
Run the plan and materialize the result — the only point in the lazy
layer that computes (or fails). The plan first passes through the
total optimizer rewrites (optimize.mbt): filters sink below the
stages they provably commute with, so rows drop as early as possible,
and a required-columns pass then drops scan columns nothing downstream
reads (and, for a scan_csv source, never parses them). The rewrites'
contract is exactly this method's, so they change
what work happens, and for a successful result never what comes back: the
(optimized) plan is walked bottom-up, delegating each node to the public
eager operator it defers, so the result equals running the same
verbs eagerly in the same order. Errors match the eager operators'
(ColumnNotFound, TypeMismatch, IndexOutOfBounds, …) with one
carve-out: a scan_csv / scan_ndjson source absorbs projection and
predicate pushdown, so it never parses a column no consumer reads, nor the
non-predicate cells of a row the predicate drops — a ParseError confined
there, which an eager read-then-filter would raise, does not surface. Every
other operator still produces its eager error. To see the rewritten plan
this method actually runs, render it with explain(optimized=true).
#LazyFrame::count
Defer DataFrame::count: a 1-row Int frame of each column's non-null cell
count (every dtype).
#LazyFrame::drop
Defer DataFrame::drop: remove the columns named by exprs (bare col
references), keeping every other column. A name absent from the frame, or a
non-col expression, surfaces as the eager drop's error at collect time.
#LazyFrame::drop_nulls
Defer DataFrame::drop_nulls: drop every row with a null in subset (a
list of col references), or — with no subset — in any column. A name
absent from the frame surfaces at collect time.
#LazyFrame::explain
Render the logical plan as an indented tree — the root operation on
the first line, inputs two spaces deeper, expressions in their
documented Show form, and SCAN [rows×cols] leaves:
SELECT [col(region), col(adj)]
WITH_COLUMNS [(col(revenue) * 1.1) as adj]
FILTER (col(region) == "west")
SCAN [4×3]By default this is the plan as built — a faithful mirror of the
chained verbs, which is the package's contract. Pass optimized=true
to render the plan collect actually runs instead: a sunk FILTER
appears below the stages it crossed — or disappears into the leaf as a
WHERE suffix, when the stage it reaches is a file source that can apply
it while reading — and the projection pushdown shows
up as a narrowing SELECT over an in-memory SCAN or as the column list
on a SCAN_CSV source, so printing both forms is the before/after view
of what the optimizer moved and pruned. (Polars' LazyFrame::explain(optimized) is the
namesake; plans are immutable, so the flag rewrites a copy and never
perturbs this frame.)
Total either way — the rewrite is a pure tree walk, so a plan that
would fail to collect still explains, which is the point: inspect
first, compute later.
#LazyFrame::fill_null
Defer DataFrame::fill_null: replace every null cell, in every column,
with value. A column whose dtype does not match value surfaces the
eager verb's error at collect time.
#LazyFrame::filter
Defer DataFrame::filter: keep the rows where predicate
evaluates to true (false / null cells drop the row). Total at build
time — a predicate over missing columns or of a non-Bool type only
fails when the plan is collected.
#LazyFrame::group_by
Defer DataFrame::group_by: attach grouping key expressions to the plan
and return a LazyGroupBy waiting for its aggregations. Each key is an
Expr evaluated over the whole frame at collect time (a bare col, a
derived key, or a length-1 key broadcasting to one group), exactly like
the eager verb. Nothing is partitioned yet — like every builder this is
total, so a missing column, a dtype clash, or a repeated output name
only surfaces at collect time, as the eager group_by's
ColumnNotFound / TypeMismatch / DuplicateColumn. Only
LazyGroupBy::agg grows the plan; a LazyGroupBy is never itself
collectable, mirroring how the eager GroupedDataFrame is not a frame.
#LazyFrame::head
Defer DataFrame::head: the first n rows. Inherits eager head's
total clamp — n beyond the frame keeps every row, negative n keeps
none — so collecting can't fail on this node. Also exposed under its
Polars / SQL name limit (via #alias): the same deferred node, so
lf.limit(n) collects to the same rows and explains as HEAD n.
#LazyFrame::join
Defer DataFrame::join: combine this plan's output (the left side)
with another plan's output (the right side) under options — each
side carries its own deferred pipeline. Key resolution, type checks,
and the cross-join key rules all surface at collect time.
#LazyFrame::max
Defer DataFrame::max: the maximum counterpart of min.
#LazyFrame::mean
Defer DataFrame::mean: a 1-row frame of each numeric column's mean as
Float (empty / all-null → Null), non-numeric columns a Null cell.
#LazyFrame::min
Defer DataFrame::min: a 1-row frame of each numeric column's minimum
(NaN skipped, source dtype kept), non-numeric columns a Null cell.
#LazyFrame::null_count
Defer DataFrame::null_count: a 1-row Int frame of each column's null
count — the complement of count.
#LazyFrame::rename
Defer DataFrame::rename: apply each (from, to) rename in order. A
missing source name or a colliding target surfaces at collect time.
#LazyFrame::rename_with
Defer DataFrame::rename_with: rename every column through f
(new = f(old)). A collision — two columns f maps to the same name —
surfaces as the eager rename_with's DuplicateColumn at collect time.
#LazyFrame::reverse
Defer DataFrame::reverse: flip the row order at collect time. Total.
#LazyFrame::select
Defer DataFrame::select: project the frame down to exactly the
evaluated expressions (an all-scalar selection collapses to one row,
like its eager counterpart).
#LazyFrame::slice
Defer DataFrame::slice: the half-open row window [start, end).
Unlike head / tail this mirrors eager slice's bounds checks —
out-of-range bounds (IndexOutOfBounds) or start > end
(InvalidOperation) surface when the plan is collected.
#LazyFrame::sort
Defer DataFrame::sort: reorder rows by one or more
(key, order, null placement) keys, later keys breaking ties. Each key
is an Expr evaluated over the whole frame at collect time, so a
missing column or a dtype clash only surfaces then.
#LazyFrame::sum
Defer DataFrame::sum: collapse the plan to a 1-row frame of each column's
sum (numeric columns their sum in the source dtype, non-numeric columns a
Null cell). collect equals the eager df.sum().
#LazyFrame::tail
Defer DataFrame::tail: the last n rows, with the same total clamp
as head.
#LazyFrame::unique
Defer DataFrame::unique: drop duplicate rows, keeping survivors in their
original order. subset and keep mirror the eager verb — subset picks
the columns forming the duplicate key (every column when omitted, and the
output always carries all columns), First (the default) keeps each key's
first occurrence, Last its last, and None keeps only rows that occur
exactly once. Total like every builder: an unknown subset name surfaces
as ColumnNotFound at collect.
#LazyFrame::with_columns
Defer DataFrame::with_columns: derive new columns (or replace
same-named ones) from expressions, keeping every existing column.
#LazyFrame::with_row_index
Defer DataFrame::with_row_index: prepend an Int counter column named
name (default "index") running from offset. Total like every builder
— a name that collides with an existing column surfaces as
DuplicateColumn at collect, and an offset the counter would overflow
as InvalidOperation there.
#LazyGroupBy
pub struct LazyGroupBy {
// private fields
}A deferred group_by awaiting its aggregations — the lazy mirror of
the eager GroupedDataFrame stage in group_by(keys).agg(…),
except it holds no groups: just the input plan and the key
expressions, which agg completes into a single Aggregate node. Both
fields stay private — a LazyGroupBy is only ever the step between
LazyFrame::group_by and LazyGroupBy::agg, and sharing one across
several agg calls just forks the plan, like every other builder.
#LazyGroupBy::agg
Complete the deferred group-by: defer group_by(keys).agg(exprs)
as one Aggregate node and return to the LazyFrame chain. Each
expression must be reduction-shaped (aggregations / literals and their
combinators — a bare column reference is not); building stays total,
so a non-reduction expression, a missing column, a dtype clash, or an
output-name collision waits for collect and surfaces as the eager
agg error (InvalidOperation / ColumnNotFound /
TypeMismatch / DuplicateColumn).
#scan_csv
The lazy CSV source: scan_csv(path) builds a LazyFrame whose leaf is a
deferred read of path — the push-down-aware counterpart of eager
read_csv. Nothing is read or parsed until collect: the builder is
total, like every other LazyFrame constructor, so a missing file or a
malformed CSV only surfaces then (as the eager reader's IoError /
ParseError / DuplicateColumn). It is not a streaming read — the reader
still tokenises the whole file, and streaming it is tracked as future work in
docs/api.md.
The payoff over LazyFrame::LazyFrame(read_csv(path)) is that this leaf
absorbs both push-downs, so the work it skips is work the eager pipeline
does:
- projection — the optimizer narrows the leaf to the columns the pipeline provably consumes, and only those are parsed, so scan_csv("sales.csv").select([col("region"), col("revenue")]).collect() never builds the columns it drops;
- predicate — a filter sitting on the leaf moves into the read: the reader builds the predicate's columns, asks which rows survive, then parses the remaining columns for the survivors alone. Only the first such filter is absorbed; a second stays a node above the scan.
Both prune cells an eager read-then-filter would have parsed, which is the
one way a narrowed scan diverges from a full eager read: a ParseError
confined to a dropped column, or to a row the predicate drops in a column the
predicate does not read, does not surface. Dtype inference still walks the
whole file, so dtypes and the surviving cells match the eager read exactly.
options (delimiter, header, null strings, inference window, parse-error
policy, strict quotes, …) defaults to CsvReadOptions::CsvReadOptions(), mirroring eager
read_csv. It is captured into the plan and applied by the reader at
collect time; projection push-down narrows which columns those options
parse, never which options apply. The projection starts None (read every
column) — only the optimizer fills it.
#scan_ndjson
The lazy NDJSON source: scan_ndjson(path) builds a LazyFrame whose leaf
is a deferred read of path — the push-down-aware counterpart of eager
read_ndjson, and the line-oriented sibling of scan_csv. Like every
LazyFrame constructor it is total: nothing is read or parsed until
collect, so a missing file or a malformed line only surfaces then (as the
eager reader's IoError / ParseError). It is not a streaming read — the
reader still parses every line — and it absorbs the same two push-downs
scan_csv documents: projection (only the consumed columns are built) and
the first predicate sitting on the leaf (the surviving rows' cells are the
only ones built past the predicate's own columns). So a parse error confined
to a dropped column, or to a dropped row, does not surface; the surviving
cells and the inferred dtypes match an eager read exactly. (There is no
scan_json for the single-array shape [{...}], which must be parsed whole
to find its records — so nothing can be pruned at read time, the same reason
Polars has scan_ndjson but no scan_json.)
options (inference window, parse-error policy) defaults to
JsonReadOptions::JsonReadOptions(), mirroring eager read_ndjson. It is captured into the
plan and applied by the reader at collect time; projection push-down narrows
which columns those options parse, never which options apply. The projection
starts None (read every column) — only the optimizer fills it.
A lightweight DataFrame and tabular-data library for MoonBit
dataframedata-analysiscsvtabular
0.6.0
Apache-2.0
Updated 17 days ago
46 downloadsDependencies
- moonbitlang/x0.4.47
- moonbit-community/NyaCSV0.3.3
- moonbitlang/quickcheck0.14.0