READMEihb2032/MoonFrame/frame does not have a README file
#DataFrame
A column-oriented, schema-aware table. DataFrame owns an ordered list
of equally-tall Series, a derived Schema, and a private
name_to_index cache so column lookup by name is O(1).
The fields are priv (private to this package), so a frame can only ever
be built through the constructors below — which rebuild the cache and the
schema in lock-step with the column vector — and read through accessors
that hand out nothing the caller can write back through: columns() and
column_series() build a fresh array each call, and schema() returns a
Schema, whose own fields are private behind a copying reader.
External code cannot reach the live columns / name_to_index containers,
so it cannot mutate a validated frame into an inconsistent state.
What every constructor here establishes and every transform preserves is
that those four parts agree. That agreement is written once — as INV1–INV7 in
frame/invariants.mbt, checked by check_invariants — and deliberately not
restated here, since a second copy is how the two would drift.
#DataFrame::DataFrame
Build a DataFrame from a list of Series. Validates:
- all columns have the same length (raise LengthMismatch otherwise);
- no two columns share a name (raise DuplicateColumn(name)).
Zero columns is valid and produces a 0×0 frame: with no column to anchor
a height, this constructor has nothing to infer a row count from. A
column-less frame may carry rows — select([]), a drop of every
column and from_rows under an empty schema all keep their input's height
— but those entry points know the height independently and build it
through from_parts.
The input array is copied, so mutating columns after construction
cannot perturb the frame's invariants (the Series values themselves
are immutable).
The type's own constructor — the spelling every canonically-constructed
type in MoonFrame uses (Schema::Schema, Field::Field, the options
types). The entry points that build a frame a different way keep their
own names: empty from a schema, from_rows from a Scalar matrix.
#DataFrame::column_series
The columns as Series, in declaration order — the total, ordered way to
walk a frame column by column, parallel to columns() and the schema's
fields. Reach for it instead of looping over columns() and calling
get_column(name) per name: that pays a lookup per column and is fallible
on a name this frame does not have, while this is neither.
A fresh array is returned (the Series themselves are immutable, storage
shared), so mutating it cannot perturb the frame's invariants. It is a
user-facing accessor, not an engine seam: it hands back the public Series
type and never the storage behind it, and the frame's own operators use it
for exactly the same reason a caller would.
#DataFrame::columns
Column names in declaration order. A fresh array is returned so that
mutation by the caller cannot break the schema/columns invariants.
#DataFrame::count
Non-null cell count of every column as a 1-row Int DataFrame — the
row-oriented counterpart of the column-oriented null_count. Total over
every dtype (no numeric restriction).
#DataFrame::describe
Per-column statistical summary, returned as a DataFrame with one
row per source column. The result schema is fixed and dtype-pure —
every cell has the dtype declared below, regardless of the source
frame's shape or column dtypes:
- column (String): source column name, in declaration order
- dtype (String): source column dtype rendered via DataType::to_string
- count (Int): non-null cell count
- null_count (Int): null cell count
- n_unique (Int): distinct non-null value count
- mean (Float, nullable): arithmetic mean for numeric columns; Null for non-numeric or empty / all-null numeric columns
- min (String, nullable): minimum cell rendered via Scalar::to_string; Null for empty / all-null columns
- max (String, nullable): maximum cell rendered via Scalar::to_string; Null for empty / all-null columns
min / max are rendered as String so the summary can carry
extrema for every dtype in a single column without forcing a
uniform value type. A Float extremum renders via Double::to_string,
which drops the decimal point of a whole value (1.0 → "1"), so a
Float column's extrema can read like an Int's — the dtype column
disambiguates. The per-column reductions (min, max) keep the
original dtype if a caller needs it typed.
This is a deliberately reduced summary: it omits the standard deviation,
variance, and quantiles (25% / 50% / 75%) that Polars' describe
reports. The per-column kernels expose those directly (Series std /
variance / median) for callers who need them.
Because min / max skip NaN (Polars' regular extrema) while count
and n_unique treat NaN as a present value, an all-NaN Float column
reports a non-zero count and an n_unique of 1 (every NaN folds into
one distinct bucket) but Null min / max — these can legitimately
disagree, in addition to the empty / all-null case noted above.
Raises only because the summary frame is built through the fallible
DataFrame::DataFrame; the output columns are hardcoded-unique and equal
length, so the raise is forwarded, never actually taken. The 0-column
case collapses to a 0×8 frame: zero source columns ⇒ zero output
rows, but the eight schema columns are still present so downstream
code can rely on the result's column layout.
#DataFrame::drop
Return a copy of self with the columns named by columns removed.
Remaining columns keep their relative order — MoonFrame's single drop
verb (Polars' df.drop(...)).
Each entry is an Expr resolved to a column name through
Expr::output_name: a bare col("x") names "x", an alias names
the alias. The container is Array[Expr] so a future column selector
(all / exclude) can drop the matched set without a signature change;
today only col / aliased keys are meaningful and the expression is never
evaluated — only its output name is consulted. The name-pattern selectors
that do exist (cols_starts_with / cols_ends_with / cols_contains /
cols_matching) need no support here: each expands to a plain col list
against a frame before the call.
Duplicate keys are tolerated and act idempotently — dropping
[col("a"), col("a")] is the same as dropping [col("a")]. This matches
pandas / polars behavior and avoids forcing callers to deduplicate
upstream when assembling a drop list from multiple sources.
Dropping nothing (drop([])) returns the frame itself — a literal
identity, declared schema included. A drop that does remove a column
filters the field vector alongside the column vector, so every surviving
column keeps the field it arrived with, its declared nullable included.
Raises:
- ColumnNotFound(name) — a resolved name does not exist. Reported on the first offending key in columns order.
Dropping every column keeps the height, like any other projection to
zero columns: the result is self.nrows() × 0, not 0×0.
Structural invariants on the returned frame follow from
DataFrame::from_parts (called with the surviving column subset and the
frame's own row count, which removing columns never changes).
#DataFrame::drop_nulls
Drop every row in which a gating column has a null cell. The schema
(column names, dtypes, order) is preserved verbatim; only the row
count shrinks.
subset selects the gating columns — MoonFrame's single null-dropping
verb (Polars' df.drop_nulls(subset)):
- omitted → every column gates (drop a row null in any column). An N×0 frame is the degenerate case: no column's validity can fail, so all N rows pass and the output is structurally identical to the input. (DataFrame::DataFrame([]) is the N = 0 instance of that, not a rule of its own — a column-less frame built through select([]) keeps its height.)
- Some(keys) → only the columns named by keys gate; a row null in an unlisted column is kept. Each key is an Expr resolved to a column name through Expr::output_name — a bare col("x") names "x". The Array[Expr] container mirrors drop and leaves room for a future column selector; today only col / aliased keys are meaningful and the expression is never evaluated.
- Some([]) → no gating columns, so every row passes: a no-op identity, schema preserved.
Duplicate keys are tolerated and act idempotently; asking the same
column to gate twice is the same as asking once.
Raises:
- ColumnNotFound(name) — a resolved name does not exist in self. Reported on the first offending key in subset order. (Cannot arise when subset is omitted: the names come straight from the frame.)
#DataFrame::empty
Build a 0-row DataFrame matching schema. Each column is an empty
Series of the field's declared dtype.
The schema is re-validated through Schema::Schema (raiseDuplicateColumn(name) on a repeated name): every Schema constructor
already rejects duplicates, so this is defence-in-depth that keeps the
frame's "no duplicate names" invariant true regardless of how the schema
was built.
raise Unsupported(...) if any field carries DataType::Null — there is
no concrete Null backend to materialise into. Only an explicitly built
schema can carry it here: the readers never infer Null, and a probe
window that is entirely null falls back to String.
#DataFrame::fill_null
Fill every null cell of the dtype-compatible columns with value,
leaving the rest of the frame verbatim — Polars' frame-wide
DataFrame.fill_null(value). A column is compatible when its dtype
matches value's variant: an Int value fills Int columns, a Float
value fills Float columns, and so on. Columns of any other dtype are
left untouched (not an error), and so are all columns when value is
Scalar::Null — filling nulls with null is a no-op, so it skips
everything and returns the frame unchanged.
Per-column control, cross-dtype fills (an Int value into a Float
column, say), or filling with a computed value go through the expression
form instead: df.with_columns([col("c").fill_null(lit(value))]).
Filling rewrites cells in place — each filled column keeps its name and
dtype (a compatible fill is dtype-preserving) and every other column is
untouched — so names, dtypes, and the row count are unchanged. The
returned frame therefore reuses self's schema and O(1) name→index
cache verbatim (the same same-schema rebuild head / tail / reverse
use) rather than re-deriving them through DataFrame::DataFrame.
raise-typed because Series::fill_null is, but the compatibility gate
means it is only ever called on a column it cannot fail on (a matching
non-null value), so no error is actually produced here.
#DataFrame::filter
Keep only the rows where predicate evaluates to true — MoonFrame's
single row-selection verb. The predicate is an Expr, not a closure, and
that is what the built-in expression algebra buys: it evaluates vectorized
(one column pass per node, not one call per row), it can be printed, and a
lazy Filter node can introspect it for predicate pushdown. A row-wise host
predicate is still reachable through the map_many escape hatch —
filter(map_many(label~, inputs, f)) reifies the original closure predicate
as a Bool-returning Expr — but a closure buys none of the three: the map
node renders as its label with the closure opaque, the optimizer can only
treat it as a barrier and sinks no filter across it, and f is called once
per row (once per batch for map_batches).
The predicate must evaluate to a Bool column (TypeMismatch
otherwise). A row is kept where its cell is true; false and
null cells drop the row (the Polars rule — an unknown is not a
keep). The mask rides the length contract in expr_eval.mbt: a length-1
result — a literal, or an aggregation comparison like
col("qty").sum().gt(lit_int(0)) — broadcasts over the frame, keeping every
row or none, while a mask that is neither frame-tall nor length-1 raises
LengthMismatch instead of selecting rows by position.
The returned frame has the same schema as self (a
rejects-everything predicate leaves a 0-row frame with the original
schema). A filter that keeps every row returns self unchanged; once
any row drops, the surviving cells are re-gathered and each column
converges onto the backend its content implies (an all-valid numeric
column lands on Numeric) — the engine-wide "backend is a function of
content" rule, not a verbatim carry-over. Evaluation errors (unknown
columns, dtype mismatches, an off-frame mask length) surface here; building
the predicate was total.
#DataFrame::from_rows
Build a DataFrame from a row-major matrix of Scalars. Each row
must have exactly schema.len() cells, and each cell must either
match the column's declared dtype or be Scalar::Null.
The height is always rows.length(), including under an empty schema:
from_rows(Schema::Schema([]), [[], []]) is the 2×0 frame, since
width-0 rows are still rows (see from_parts).
Raises:
- DuplicateColumn(name) — the schema carries a repeated field name. Every Schema constructor already rejects duplicates, so this is defence-in-depth: re-validating through Schema::Schema here (as empty does) keeps the "no duplicate names" invariant regardless of how the schema reached this call.
- LengthMismatch — a row's width differs from schema.len().
- TypeMismatch(Expected(expected, got, column)) — a non-null cell's dtype doesn't match the schema's dtype for that column (the pieces kept structured).
- Unsupported(...) — schema declares a Null-dtype column (same reason as empty).
- NullInNonNullable(name) — a Scalar::Null lands in a column whose field is declared nullable = false.
#DataFrame::gather
Gather rows by index. Returns a frame whose i-th row is
self[indices[i]]. Any out-of-bounds index surfaces as
IndexOutOfBounds(idx).
#DataFrame::get_column
O(1) lookup of the column named name. raise ColumnNotFound(name)
if the column is missing.
#DataFrame::get_column_at
Column at position i. Out-of-bounds raise IndexOutOfBounds(i).
#DataFrame::group_by
Partition self into groups by the keys expressions — MoonFrame's
single group_by verb (Polars' df.group_by(...)). Rows sharing the
same key tuple land in the same group; group order is first
appearance (equivalent to Polars' maintain_order=True), so the
result is deterministic and snapshot-stable without a sort.
Each key is an arbitrary expression, evaluated over the whole frame
under the rules in expr_eval.mbt, exactly like a sort key: a bare
col("region") groups by an existing column, a derived key such as
(col("a") + col("b")) groups by the computed value, and a key that
reduces to a single cell (a literal, or an aggregation like
col("x").sum()) broadcasts over the frame — every row shares it, so it
collapses the frame into one group. The materialised key columns head
the agg output, each named by its expression's output name (alias,
else leftmost column reference, else "literal") and keeping its
evaluated dtype.
Group identity is the composite KeyCell tuple of the row's key cells
(see frame/row_key.mbt), hashed on the native cell values, so:
- a Float NaN collapses into one group — matching Polars, where NaN compares equal for grouping (the same rule join keys on); -0.0 and +0.0 likewise share a group;
- a null key cell forms its own group rather than being dropped — the Polars default (pandas drops null keys), and the deliberate semantic difference from join, where a null key matches nothing (null != null).
keys may hold one expression (group_by([col("region")])) or several
(group_by([col("region"), col("product")])). An empty keys list
places every row in a single group (a grand-total partition); a 0-row
frame yields zero groups regardless of keys.
Raises:
- ColumnNotFound(name) — a key expression references an absent column. Reported on the first offending key, in keys order.
- TypeMismatch(...) — a key expression's dtypes don't unify. Reported on the first offending key, in keys order.
- LengthMismatch — a key is neither frame-tall nor length-1, which only a lit_series or a map_batches closure can produce (the shared length contract in expr_eval.mbt). Reported on the first such key.
- DuplicateColumn(name) — two keys produce the same output name (which would otherwise have agg emit two identically-named key columns and fail late with a confusing collision). Reported at the second such key, in keys order — mirroring select's per-expression seen-check, and after that key's own evaluation, so an evaluation error there wins over the collision.
#DataFrame::head
First n rows (or all rows if n >= nrows). Negative n clamps to
0. Total — never fails. Also exposed under its Polars / SQL name
limit (via #alias) — the eager twin of LazyFrame::limit, keeping
the two surfaces verb-for-verb aligned.
#DataFrame::is_empty
true when the frame has zero rows. Note that a frame with zero
columns and zero rows is also empty (nrows == 0). A 0×N frame
(declared schema, no rows yet) is also empty.
#DataFrame::item
Read the cell at (row, name) as a Scalar, mirroring Polars'
DataFrame.item(row, column). Surfaces ColumnNotFound for unknown
names and IndexOutOfBounds for row indices outside [0, nrows).
#DataFrame::join
Equi-join self (the left frame) with other (the right frame) on the
key expressions the options carry — JoinOptions::on(keys), or the
paired JoinOptions::left_on(keys, right_on=keys) — producing a new
DataFrame.
Each key is an arbitrary expression, evaluated over the whole frame
under the rules in expr_eval.mbt, exactly like a sort / group_by
key: an on key is applied to both frames (a bare col("id") joins on
an existing column, a derived key such as col("ts") / lit_int(86400)
joins on the computed value), while a sided pair evaluates
position-paired keys on the left and right frame respectively (for
differently-named or differently-derived keys). A key that reduces to a
single cell (a literal, or an aggregation) broadcasts over the frame.
Reading back what an options value carries is on_keys() /
left_keys() / right_keys(), each returning a copy.
Two rows match when every key holds an equal value, using the same
composite-KeyCell-tuple encoding group_by uses (see
frame/row_key.mbt; the tuple is structurally injective across key
columns). The one deliberate difference from group_by: a null key
cell matches nothing (null != null, the SQL / Polars default) —
such an unmatched row is dropped by Inner and kept (with the other
side's columns null) by Left / Right / Outer. A Float NaN key
is not null, so — as in group_by, and matching Polars' "NaN compares
equal" rule — all NaN keys match each other.
how selects which unmatched rows survive:
- Inner — only matched pairs.
- Left — matched pairs plus every unmatched left row (right columns null).
- Right — matched pairs plus every unmatched right row (left columns null); the mirror of Left.
- Outer — matched pairs plus every unmatched row from both sides.
- Cross — the keyless Cartesian product (see below).
Output shape:
- columns = the left columns (original order and names) followed by the right frame's columns (original order). Coalescing applies only to an on join whose every key is a bare col(...) (the only shape where a key names the same column on both sides); left_on / right_on and any derived key never coalesce. Whether an eligible right key column is kept is then governed by options.coalesce (None = auto by how, matching Polars: an inner / left / right join coalesces, an outer join does not). When coalesced, the key appears once at the left key's position, taking each row's value from whichever side is present (the left on Inner / Left, the right on Right, the present side per row on Outer — the two are equal on a matched pair); the right key column is dropped. When not coalesced, the right key column is kept too — it clashes with the left key name, so it gains options.suffix (e.g. id_right) and is null wherever its row had no match. Any other right column whose name occurs in the left frame is likewise suffixed; the left column keeps its name. (A derived key contributes no column of its own — it only decides which rows pair — so the output is just the two frames' columns, the right suffixed on a clash.)
- rows = left rows in their original order (each with its right matches in ascending right-row order, then — for Left / Outer — unmatched left rows in place with null right columns), followed for Outer by the unmatched right rows in right-row order. A Right join instead emits every right row in right-row order (each with its left matches in ascending left-row order, else the right row alone with null left columns). The order is fully determined by the input order, so results are snapshot-stable.
how = Cross is the Cartesian product (every left row paired with
every right row); it takes no keys, ignores coalesce, and keeps all
columns of both frames (suffixing a right column that clashes with a
left name). It is the explicit form of what group_by([])'s
grand-total group is for aggregation.
The output height is the row plan's own length, never inferred from the
assembled columns, so a join between two column-less frames keeps its
rows: 2×0 cross 3×0 is the 6×0 frame, not 0×0.
The result routes through DataFrame::from_parts, so it satisfies
check_invariants().
Raises:
- ColumnNotFound(name) — a key expression references an absent column. Reported on the first offending key, in key order, evaluating the left frame's key before the right's.
- TypeMismatch(detail) — a key's left and right dtypes differ (so its values could never compare equal), or a derived key's own dtypes don't unify. Reported on the first such key, in key order.
- InvalidOperation(detail) — no keys for a non-Cross join (use how = Cross for a Cartesian product), any keys on a Cross join (which takes none), or a sided pair of unequal length. (Mixing shared and sided keys is a fourth case the engine still checks, though the three constructors make it unspellable from outside frame.)
- DuplicateColumn(name) — either a repeated shared key that names a column twice (only an on join of bare col keys is deduplicated, so only that form can reach this, like group_by([col("id"), col("id")])), or two output columns still colliding after suffixing (e.g. the left frame already has both value and value_right, and the right contributes a non-key value; surfaced by DataFrame::from_parts).
- LengthMismatch — a lit_series(s) key whose embedded series is neither length 1 nor the frame's height (the evaluator's broadcast rule, surfacing through key evaluation).
#DataFrame::max
Maximum of every column as a 1-row DataFrame. Mirrors min with the
order reversed; the NaN-skipping and non-numeric-Null rules are
identical.
#DataFrame::mean
Arithmetic mean of every column as a 1-row DataFrame: numeric columns
hold their mean as Float (an empty / all-null numeric column → Null),
non-numeric columns a Null cell in their own dtype.
#DataFrame::min
Minimum of every column as a 1-row DataFrame: numeric columns hold their
minimum non-null cell (Float NaN skipped, source dtype preserved),
non-numeric columns a Null cell. Note this nulls Bool / String
columns rather than ordering them — use Series::min for a typed
extremum over any dtype.
#DataFrame::nrows
Number of rows. With at least one column it equals every column's len()
(INV2); a column-less N×0 frame has none to agree with, and carries its
height in the explicit nrows field alone.
#DataFrame::null_count
Per-column null-count summary, returned as a 1 × ncols DataFrame.
Column names match self.columns() verbatim, in the same order, and
every output column has dtype Int with a single non-null cell —
the null count of the source column.
The summary is one row wide of whatever the source has, so a 0-column
frame summarises to 1×0 — the row is there, it just has nothing in it.
The height is passed explicitly for that reason: with no column to anchor
it, an inferred one would round the row away.
Raises only because the summary is built through the fallible
DataFrame::from_parts; its failure paths (duplicate-name, length-mismatch)
cannot fire here — the names come from self.column_series() (already
unique) and every output column has length 1 — so the raise is forwarded,
never actually taken.
#DataFrame::rename
Return a copy of self with one or more columns renamed. Renames
are applied in input order: each step's new_name becomes
visible to subsequent renames, which is what makes the three-step
swap [(a, _t), (b, a), (_t, b)] work.
Raises:
- ColumnNotFound(old_name) — old_name doesn't exist (or was already renamed away by an earlier step).
- DuplicateColumn(new_name) — new_name collides with another column that still bears that name at this step.
An empty mapping returns the frame itself — a literal identity. A
(name, name) mapping is a no-op that still validates the source name's
existence.
Only names change: each field is edited through Field::rename, so its
dtype and its declared nullable flag ride along instead of being
re-derived (DataFrame::DataFrame would reset the flag to the Field
constructor default). All structural invariants follow from re-validating
the edited field vector through Schema::Schema, which catches any residual
duplicate.
#DataFrame::rename_with
Return a copy of self with every column renamed through f: each
column's new name is f(old_name). The callable form of rename — Polars'
df.rename(function) — for a uniform transform (a prefix, a case fold) over
the whole schema rather than an explicit old -> new list.
f is total, so the only failure is a collision: if f maps two distinct
columns to the same name, the re-validating Schema::Schema raises
DuplicateColumn. (There is no ColumnNotFound — every existing column is
renamed, none is looked up by name.)
The identity f = name => name is a no-op. Column order, dtypes, the row
count, and each field's declared nullable flag are unchanged; only the
names are refreshed, through the same Field::rename edit rename applies.
#DataFrame::reverse
Reverse the row order, keeping every column and its dtype — Polars'
df.reverse(). The schema is untouched, so a reversed frame is a frame
with the same columns in the same order, read bottom-up.
Total: the permutation is [n-1, …, 0], in bounds by construction. A
0-row or 1-row frame is returned unchanged rather than gathered into an
identical copy.
#DataFrame::row
Row i's cells as a Scalar tuple in column order (a null cell as
Scalar::Null), mirroring Polars' DataFrame.row. Row indices outside
[0, nrows) raise IndexOutOfBounds(i); for many rows, rows() reads
the whole frame in one pass instead.
#DataFrame::rows
Every row as a Scalar tuple in column order (result[r][c]), the
row-major transpose of to_scalar_matrix, mirroring Polars'
DataFrame.rows. Total — materialises the frame once.
#DataFrame::schema
The schema (column names + dtypes + nullability) of this frame.
#DataFrame::select
Project the frame to exactly the evaluated expressions — MoonFrame's
single select verb (Polars' df.select(...)). Each expression is
evaluated over the whole frame under the dtype / null / NaN rules in
expr_eval.mbt, so a projection can pick an existing column
(col("a")), compute a new one (col("a") + col("b")), aggregate
(col("a").sum()), or inject a literal. The plain names-only projection
of earlier versions is select([col("a"), col("b")]) — or, equivalently,
select(cols(["a", "b"])) — and behaves identically: a list of bare
col references projects those columns in that order.
Output height (the Polars select rule), under the length contract in
expr_eval.mbt:
- any frame-tall result fixes the height at self.nrows(), and every length-1 result broadcasts up to it — down to zero rows on an empty frame;
- if every expression reduces to length 1, the output is a single row: df.select([col("a").sum()]) is the one-row summary frame, not nrows copies of it;
- a result of any other length — only a lit_series or a map_batches closure can produce one — raises LengthMismatch rather than dictating an off-frame height.
Column metadata follows the cells: an entry that only renames a column (a
bare col("x"), or an aliased one) carries that column's Field, declared
nullable included, while a computed entry gets a field derived from the
result.
Naming follows with_columns: each column takes its expression's
output name (alias, else leftmost column reference, else "literal"),
and a name produced twice raises DuplicateColumn at the second
expression — errors surface in expression order, before later
expressions are evaluated. select([]) is the projection to zero
columns, and like any projection it keeps the frame's height: the result
is self.nrows() × 0.
Evaluation errors (unknown columns, dtype mismatches, the
unrepresentable Null literal, an off-frame result length) surface here;
building the expressions was total.
#DataFrame::slice
Half-open [start, end) row slice. Bounds checks mirror
Series::slice (and BuiltinColumn::slice): an index outside the frame
surfaces as IndexOutOfBounds — start < 0, or an end outside
[0, nrows] — and only two individually valid indices in the wrong order
are InvalidOperation.
#DataFrame::sort
Sort self by one or more Expr keys, applied in order — MoonFrame's
single sort verb (Polars' df.sort(...)). keys is an array of
(key, order, null_order) tuples: a single-key sort passes a
one-element array (df.sort([(col("q"), Desc, NullsLast)])), a
multi-key sort lists several
(df.sort([(col("dept"), Asc, NullsLast), (col("salary"), Desc, NullsLast)])).
Earlier keys dominate; later keys only break ties between rows that
compare equal under all earlier keys.
Each key is an arbitrary expression, evaluated over the whole frame
under the rules in expr_eval.mbt: a bare col("q") sorts by an
existing column, a derived key like col("a") + col("b") sorts by the
computed value without materialising it into the output. A key that
reduces to a single cell (a literal, or an aggregation such as
col("q").sum()) broadcasts over the frame — every row shares it, so
it is a stable no-op that leaves earlier keys and the input order
untouched. A key of any other length — only a lit_series or a
map_batches closure can produce one — raises LengthMismatch under the
shared length contract, rather than ordering rows by a key it has no cell
for.
The sort is stable (bottom-up merge sort on row indices): two
rows that compare equal under every key keep their original relative
order. This is what makes [(col("dept"), Asc, _), (col("salary"), Desc, _)]
behave the same as "sort by dept, then within each dept sort by
salary descending".
Null and NaN placement is governed by each key's null_order.
NaN in a Float key is treated identically to Null for
ordering — IEEE 754 makes < / > against NaN return false, so a
naive comparator would scatter NaNs unpredictably. (This is the one
deliberate departure from Polars, which orders NaN as the largest
value and treats only Null as missing.)
The returned frame routes through DataFrame::gather, so the schema
(column names, dtypes, order) is preserved verbatim and only the row
order changes — the key expressions decide the permutation but never
appear in the output. Every output passes check_invariants().
Evaluation errors surface here (building the keys was total): a key
referencing an unknown column raises ColumnNotFound on the first
offending key, a dtype clash raises TypeMismatch, an off-frame key length
raises LengthMismatch, and the
unrepresentable Null literal raises Unsupported. Every evaluated
key is one of Int / Float / Bool / String (there is no
Null-dtype backend), so it is always sortable.
An empty key set is a no-op identity: zero keys ⇒ every comparison
returns 0 ⇒ stability preserves the input order.
#DataFrame::sum
Sum every column as a 1-row DataFrame: numeric columns hold their sum
(source dtype preserved), non-numeric columns a Null cell. See the file
header for the full semantics. For one column's scalar use
df.get_column(name).sum().
#DataFrame::tail
Last n rows (or all rows if n >= nrows). Negative n clamps to
0. Total — never fails.
#DataFrame::to_html
Render a DataFrame as an HTML <table>. options defaults to
HtmlOptions::HtmlOptions() — all rows, no class / <caption>, HTML-escaped — and
otherwise supplies an optional class / <caption>, a row cap with a
<tfoot> ... (K more rows) banner, and optional escaping.
The output is a <thead> with one <th> per column followed by a
<tbody> with one <tr> per record (a <td> per cell, in
declaration order); a null cell renders as <td></td>. & / < /
> / " / ' are escaped to their HTML entities.
A frame with nothing to tabulate degrades gracefully:
- 0 columns → the empty string, at any height (MoonFrame's own choice for a table with no cells to draw). This covers the N×0 frame too, whose rows are real (is_empty() is false) but carry no fields; shape() is what reports the height of a column-less result;
- N columns / 0 rows → header with an empty <tbody>.
Total — pure string assembly over already-validated frame data, and
options.max_rows is clamped into [0, nrows], so no input can fail.
#DataFrame::to_markdown
Render a DataFrame as a GitHub-flavored Markdown table.
The output is three blocks of pipe-bounded rows:
- header — column names, one cell per column;
- separator — dashes sized to each column;
- data — one row per record, in declaration order.
Column widths are aligned to max(header, every rendered cell),
with a 3-character minimum so the dash separator never collapses
below the GFM-required floor. Null cells render as the empty string
(matching Scalar::to_string), so a null and a genuine empty-string
cell render identically — the renderer carries no null marker.
A frame with nothing to tabulate degrades gracefully:
- 0 columns → the empty string, at any height. A GFM table is built out of cells, so a column-less frame has none to draw — including the N×0 frame, whose rows are real (is_empty() is false) but carry no fields. The rendering cannot show them; shape() is what reports the height of a column-less result;
- N columns / 0 rows → header + separator with no data rows.
max_rows caps the output at its first rows: when self has more rows
than that, the truncated count is appended below the table as
... (N more rows), separated by a blank line — per the GFM tables
extension a table only ends at an empty line, so a banner glued to the
last row would parse as one more (single-cell) table ROW rather than
the paragraph below it the wording promises. A negative max_rows is
clamped to 0, so a frame that has rows still renders as header +
separator + the ... (N more rows) banner; only a genuinely 0-row
frame is header + separator alone. Omitting max_rows renders every row.
Total — pure string assembly over already-validated frame data.
#DataFrame::unique
Drop duplicate rows — Polars' df.unique(maintain_order=True). Two rows are
duplicates when every cell is equal under the same composite KeyCell
encoding group_by and join use (see frame/row_key.mbt): a Float
NaN equals NaN, -0.0 folds into +0.0, and — like a group_by key — a
null cell is an ordinary value, so two rows that are null in the same
places (and equal elsewhere) are duplicates.
keep selects which occurrence survives (default First): First / Last
keep one representative per distinct row, None keeps only rows with no
duplicate at all. Result rows keep their first-appearance order (First),
their last-occurrence order (Last), or their original order among the
survivors (None) — always ascending by original row index, so the output
is deterministic without a sort. When duplicate rows are dropped, a column
that becomes all-valid may converge onto the Numeric fast-path backend —
exactly as gather does, since the backend is a function of the
gathered content, not the source; the schema, values, and frame invariants
are unchanged.
subset picks the columns whose values form the duplicate key (Polars'
subset); omitted, every column takes part. The output always carries all
columns either way — a subset narrows what counts as a duplicate, not what
is returned. Like drop / drop_nulls, each subset entry is consulted by
its output name only and never evaluated: pass bare col(name)
references — a computed key such as col(a) + col(b) resolves to its
leftmost column name, not the computed value. An explicitly empty subset
gives every row the same (empty)
key, so First / Last keep one row and None keeps none unless the frame
has a single row.
Raises ColumnNotFound(name) on the first subset name absent from the
frame, reported before any per-row work. With no subset the resolution
cannot fail — the names come straight from the frame — so the all-columns
form still never fails on a valid frame. A frame whose kept set is all of
[0, n) — every row distinct, so no strategy drops anything — is returned
unchanged (the gather would be a no-op), as is a 0-row frame.
#DataFrame::with_columns
Evaluate each expression against self and add the results as
columns. Derived columns are declared (col("a") + col("b")) rather
than pre-materialised (a lit_series(s) carries a ready-made Series).
Every Expr the evaluator supports is accepted here — aggregations
included, evaluating to length-1 and broadcasting back over the frame —
under the dtype / null / NaN rules documented in expr_eval.mbt, which is
the one place the node set is enumerated.
Naming and placement:
- each result takes its expression's output name — an alias if the tree carries one, else the leftmost column reference, else "literal" for a pure-literal tree (expr_output_name). The alias clause is what decides replace-vs-append below: col("a").with_alias("b") appends b, it does not replace a;
- an output name already present in self replaces that column in place (original position kept);
- a new output name appends rightmost, in expression order;
- two expressions producing the same output name raise DuplicateColumn at the second — nothing silently wins.
Every result rides the length contract in expr_eval.mbt: a frame-tall one
becomes the column, a length-1 one (a literal) broadcasts to nrows — to
zero rows on an empty frame — and any other length raises LengthMismatch,
which only a lit_series or a map_batches closure can produce. Errors
(unknown columns, dtype mismatches, the unrepresentable Null literal, an
off-frame length) surface here at evaluation time;
building the expressions was total. with_columns([]) adds nothing and
returns the frame itself — a literal identity, declared schema included.
A call that adds or replaces a column leaves the schema of every other
column alone: each keeps the field it arrived with, declared nullable
included. The column an expression writes takes a carried field when the
expression only renames one (a bare col("x"), or an aliased one — so a
replacement inherits from whichever column now supplies the cells) and a
freshly derived one, nullable = true, when it computes.
#DataFrame::with_row_index
Prepend a row-number column — Polars' df.with_row_index(name, offset).
The counter is an Int column named name (default "index") holding
offset, offset + 1, … in row order, and it lands first, ahead of
the frame's own columns, as in Polars.
Raises DuplicateColumn(name) when the frame already has a column of that
name — the frame's own invariant, surfaced by the Schema the rebuild
derives (DataFrame::from_parts_with_fields). The
counter is dense and always non-null, so it never changes any other
column's dtype or nullability.
Raises InvalidOperation when the last number would not fit: the counter
runs offset ..= offset + nrows - 1, and past Int64::MAX MoonBit's
wrapping addition would carry it round to Int64::MIN, leaving a column
that is neither increasing nor dense. Polars refuses the same overflow
against its own index type.
#GroupedDataFrame
pub struct GroupedDataFrame {
// private fields
}A DataFrame partitioned into groups by one or more key expressions,
produced by DataFrame::group_by and consumed by agg. The fields are
priv (private to this package); the only way to build one is
group_by, which guarantees every stored row index is in
[0, source.nrows()) and every group is non-empty. External code cannot
reach the live key_columns / groups arrays, so a handle cannot be
mutated out of those invariants; agg additionally rejects an
out-of-range handle with InvalidOperation.
- source — the frame being grouped (its columns back every aggregation).
- key_columns — the materialised key columns, one per key expression in the order passed to group_by, each already named by the expression's output name (alias, else leftmost column reference, else "literal") and carrying its evaluated dtype and backend. They head the agg output in this order; agg gathers one representative cell per group from them.
- groups — one entry per distinct key tuple, in first-appearance order, each holding that group's row indices in ascending row order.
#GroupedDataFrame::agg
Reduce each group of a grouped frame to one row, evaluating one
expression per output column. Each expression is a reduction over
its group: col("revenue").sum(), col("revenue").max() -col("revenue").min() (a per-group range), (col("revenue") -col("cost")).sum() (a reduction over a derived column) — the group
collapses to a single cell per expression.
Output shape: the key columns head the frame (in key order, each named
by its key expression's output name and keeping its evaluated dtype —
the backend, like every row gather's, follows the gathered content —
one representative row per group), followed by one column per
expression (in expression order), named by the expression's
output name (alias, else leftmost column reference, else "literal");
one row per group, in the first-appearance group order fixed by
group_by. The result routes through DataFrame::from_parts, so name
collisions — two expressions sharing an output name, or an expression
shadowing a key column — surface as DuplicateColumn, and every output
satisfies check_invariants(). agg([]) degenerates to a distinct over
the key tuples (zero expressions ⇒ zero aggregated columns).
The height is the group count, never inferred from the output columns, so
the one shape with neither a key column nor an aggregated one keeps its
rows: group_by([]).agg([]) over a non-empty frame is the 1×0 frame
(the grand-total group, reduced to nothing), not 0×0.
Each expression must be reduction-shaped (reduces_per_group):
aggregations and literals are per-group scalars, and every combinator —
arithmetic / comparison / Kleene operators, not and the null probes,
cast, with_alias, when/then/otherwise — preserves scalarness; a
bare column reference pins the result to the group height, so it is
not a reduction. The check is structural, hence deterministic:
agg([col("a")]) raises InvalidOperation even when every group
happens to hold a single row, where a dynamic length test would
data-dependently pass. (Polars would implicitly collect the group into
a list value; MoonFrame has no list dtype, so implicit list-aggregation
stays out of scope.)
Evaluation reuses the expr_eval.mbt engine with scope = each
group's row indices, inheriting the documented dtype / null / NaN
rules verbatim: sum / mean propagate NaN and reject non-numeric
dtypes, min / max skip NaN and are total over every dtype, an
all-null group sums to the additive identity and means to a null
cell, count counts non-null cells. Dtype errors (ColumnNotFound,
TypeMismatch) surface from the per-group evaluation itself (the
bare-column fast path still reports them up front, resolving its
reducer before any group work); the output dtype is taken from the
reduced cells themselves, since a map(...) closure's result dtype is
scope-dependent. A probe evaluation under the empty scope is consulted
only as a fallback that types — and gates — a zero-group or all-null
result: consulted eagerly it would spuriously reject a dtype-changing
map(...) under a numeric reduction, because the closure never runs
under the empty scope and the probe would type the Map by its
leftmost input column instead. Computed columns follow the
expression-engine backend convention: an all-valid numeric column
converges onto Numeric, anything nullable (or Bool / String) stays
Builtin.
An aggregation directly over a bare column — col(name).<agg>() for any
aggregation op, the common case once aliases are peeled — takes a
single-pass fast path (bare_col_agg): the shared reduction kernel
resolves its reducer and validity mask once and folds each group's row
indices straight off the source column, instead of gathering a fresh
sub-column per group through the general evaluator. It is purely an
optimization — the result is cell-for-cell and backend identical to the
general path, with the same dtype / null / NaN rules and the same up-front
error surfacing — so an aggregation over a derived operand
((col("a") - col("b")).sum()) transparently falls back to it.
Raises:
- InvalidOperation(...) — an expression is not reduction-shaped, or the handle's groups no longer satisfy the group_by invariants (an empty group, or a row index outside the source). The fields are priv, so no external caller can reach the arrays; only in-package code could plant such a handle, and it is re-checked here rather than allowed to drive the folds into an abort.
- ColumnNotFound(name) — an expression references an absent column.
- TypeMismatch(...) — an expression's dtypes don't unify (e.g. sum over a String column), from the first group's evaluation — or from the fallback probe when zero groups / an all-null result leave no dtype witness.
- DuplicateColumn(name) — two output column names collide.
- LengthMismatch — a group's reduction produced something other than one cell, which is the expr_eval.mbt length contract narrowed to a group. The built-in aggregations, the literals, and the combinators over them cannot; a map_batches(returns_scalar=true) closure declared to reduce can, by returning a series of another length. Reported on the first group that does, in group order.
#HtmlOptions
Options for DataFrame::to_html. Fields are read-only outside the
package — build them through HtmlOptions::HtmlOptions(...), naming only what differs
from the defaults:
- max_rows — cap the number of data rows. None (the default) renders every row; Some(n) shows the first n and appends a <tfoot> banner (... (K more rows)) for the remainder. A negative n clamps to 0 (header + banner, no data rows), matching to_markdown's max_rows.
- table_class — value of the <table class="..."> attribute. None (the default) omits the attribute.
- caption — text of a leading <caption> element. None (the default) omits it.
- escape — when true (the default) every emitted string — caption, header names, cell values, and the class attribute — has & / < / > / " / ' rewritten to their HTML entities, so untrusted data can't inject markup. Pass escape=false for trusted input that intentionally carries HTML.
#HtmlOptions::HtmlOptions
fn HtmlOptions::HtmlOptions(max_rows? : Int, table_class? : String, caption? : String, escape? : Bool) -> HtmlOptionsBuild render options. Omitting a parameter keeps its default: every row
rendered, no class, no <caption>, and HTML escaping on. HtmlOptions::HtmlOptions()
is the default rendering; name only what differs, as in
HtmlOptions::HtmlOptions(max_rows=20, caption="Summary").
#JoinOptions
The knobs for DataFrame::join: which columns to match on, the join
how, the suffix disambiguating a right column whose name collides
with a left one, and whether to coalesce the key columns. Fields are
read-only outside the package — build options through JoinOptions::on /
JoinOptions::left_on / JoinOptions::cross, naming only the knobs that
differ from their defaults. The three entry points are the legal shapes: an
on join, a paired left_on / right_on join, and a keyless cross join —
no combination of them can be spelled.
- on — the key expressions, evaluated over both frames (Polars' IntoExpr). A bare col("id") matches on an existing column; a derived key such as col("ts") / lit_int(86400) matches on the computed value. ["region", "product"]-style multi-key joins become [col("region"), col("product")], matched on the tuple. An empty key list is invalid for on — a non-Cross join requires at least one key (DataFrame::join raises InvalidOperation); use JoinOptions::cross() for a Cartesian product.
- left_on / right_on — paired key expressions evaluated on the left and right frame respectively, for joining on differently-named or differently-derived keys (left_on([col("a")]) against right_on([col("b")])). Mutually exclusive with on, and the two lists must name the same number of keys; key i on the left is matched against key i on the right.
- how — Inner (the on / left_on default), Left, Right, Outer, or Cross.
- suffix — appended to a right column whose name also occurs in the left frame (default "_right"); the left column keeps its name.
- coalesce — whether each key column is merged into a single output column. None (the default) auto-selects by how, matching Polars: an inner / left / right join coalesces (the key appears once), while Outer does not (the right key is kept as <key><suffix>, null wherever its row had no match, so matched and unmatched rows are distinguishable). Some(true) always coalesces; Some(false) never does. Coalescing only applies when keying by on whose every key is a bare col(...) (the precondition for the key to name a single column on both sides); left_on / right_on and any derived key turn it off, keeping both key columns (Polars' "join on a non-column expression turns off coalescing"). When coalesced the single key takes its value from whichever side is present on that row (the two are equal on a matched pair) — the left on Inner / Left, the right on Right, and the present side per row on Outer. The three key lists are priv, read through the copying on_keys() / left_keys() / right_keys(). A public Array field is readable, and reading an array hands over the array itself — enough to push a key into an options value a LazyFrame::join already captured, changing what a built plan collects. The constructors copy on the way in for the same reason; these accessors close the way out. The remaining fields are immutable values, so they stay public.
#JoinOptions::cross
Start a cross-join specification — the Cartesian product of the two
frames, with no key columns (how = Cross, all key lists empty).
suffix still applies to a right column whose name clashes with a left
one; coalesce is irrelevant (there are no keys to merge).
#JoinOptions::left_keys
The left-side key expressions (JoinOptions::left_on), or an empty array
when the join keys by on or is a cross join. A copy, like on_keys.
#JoinOptions::left_on
fn JoinOptions::left_on(keys : Array[Expr], right_on~ : Array[Expr], how? : JoinType, suffix? : String, coalesce? : Bool) -> JoinOptionsStart a join on differently-named (or differently-derived) keys: keys is
evaluated on the left frame, paired position-by-position with
right_on on the right
(JoinOptions::left_on([col("a")], right_on=[col("b")])). Taking both
sides at once makes an unpaired specification unspellable. Defaults match
on, except that left_on / right_on keys never coalesce — both key
columns are kept (Polars' rule).
Both arrays are copied, so mutating them after construction cannot alter
the join specification.
#JoinOptions::on
fn JoinOptions::on(keys : Array[Expr], how? : JoinType, suffix? : String, coalesce? : Bool) -> JoinOptionsStart a join on shared key columns — each key expression is evaluated on
both frames (JoinOptions::on([col("id")])). how defaults to an
inner join, suffix to "_right" for a right column whose name collides
with a left one, and an omitted coalesce leaves key coalescing on
automatic (by how, matching Polars); pass coalesce=true / false to
force it.
The keys array is copied (as in left_on), so mutating it after
construction cannot alter the join specification.
#JoinOptions::on_keys
The shared key expressions (JoinOptions::on), or an empty array for a
sided or cross join. A copy: mutating it cannot alter the options, or any
plan holding them.
#JoinOptions::right_keys
The right-side key expressions paired with left_keys, or an empty array
when the join keys by on or is a cross join. A copy, like on_keys.
#JoinType
How an equi-join combines rows that share (or fail to share) a key.
pub(all) so callers can name the variants directly (e.g.
JoinOptions::on(keys, how=Left)), matching @types.SortOrder / @types.NullOrder:
- Inner — keep only rows whose key matches on both sides.
- Left — keep every left row; where it has no right match, the right-hand columns are filled with nulls.
- Right — the mirror of Left: keep every right row; where it has no left match, the left-hand columns are filled with nulls. The output still leads with the left columns (only the row set changes to "all right rows").
- Outer — the full outer join: keep matched pairs plus every unmatched row from both sides (the missing side's columns null). Left rows (matched and unmatched) come first in left order, then the unmatched right rows in right order.
- Cross — the Cartesian product (every left row paired with every right row); takes no key columns.
#KeepStrategy
Which row of a set of duplicates DataFrame::unique keeps — Polars'
keep parameter. Two rows are duplicates under the same composite
KeyCell encoding group_by / join use (a Float NaN equals NaN,
-0.0 folds into +0.0, and a null cell is an ordinary value).
- First keeps the earliest occurrence of each distinct row;
- Last keeps the latest;
- None keeps only rows that have no duplicate at all — every row of a duplicated set is dropped (Polars' keep='none').
The kept rows always come out in ascending original-row order, so the result
is deterministic without a sort regardless of strategy.
#KeepStrategy::not_equal
#cols_contains
The col(...) references of every column of df whose name contains the
literal substr, in schema order — Polars' cs.contains(substr). A literal
substring test, so total. Empty when nothing matches; an empty substr
matches every column.
#cols_ends_with
The col(...) references of every column of df whose name ends with
the literal suffix, in schema order — Polars' cs.ends_with(suffix). A
literal test, so total. Empty when nothing matches; an empty suffix
matches every column.
#cols_matching
The col(...) references of every column of df whose name matches the
POSIX regular expression pattern (a partial match, like
str_contains(pattern, literal=false) over cells)
— Polars' cs.matches(pattern). The dialect is POSIX ([[:alpha:]], not the
PCRE \w), and an invalid pattern raises InvalidOperation. Schema order;
empty when nothing matches.
#cols_of_dtype
The col(...) references of every column of df whose dtype is exactly
dtype, in schema order — Polars' cs.by_dtype(dtype). An empty array when
no column has that dtype.
#cols_starts_with
The col(...) references of every column of df whose name starts with
the literal prefix, in schema order — Polars' cs.starts_with(prefix). A
literal (not regex) test, so — unlike cols_matching — it is total.
Empty when nothing matches; an empty prefix matches every column.
Source Files
- agg_expr.mbt
- dataframe.mbt
- describe.mbt
- drop.mbt
- drop_nulls.mbt
- expr_eval.mbt
- expr_eval_agg.mbt
- fill_null.mbt
- filter.mbt
- group_by.mbt
- invariants.mbt
- join.mbt
- join_options.mbt
- null_count.mbt
- rename.mbt
- reverse.mbt
- row_key.mbt
- select.mbt
- selectors.mbt
- series_use.mbt
- sort.mbt
- stats.mbt
- table_render.mbt
- to_html.mbt
- to_markdown.mbt
- unique.mbt
- with_columns.mbt
A lightweight DataFrame and tabular-data library for MoonBit
dataframedata-analysiscsvtabular
0.6.0
Apache-2.0
Updated 17 days ago
47 downloadsDependencies
- moonbitlang/x0.4.47
- moonbit-community/NyaCSV0.3.3
- moonbitlang/quickcheck0.14.0