🥮 mooncakes.io
    Sign in

    zarr

    Zarr v2 and v3 chunked array storage for MoonBit

    zarr
    array
    scientific-data
    storage
    Download zip
    Author
    Version
    0.2.0
    License
    Apache-2.0
    Repository
    Last updated
    26 days ago
    Downloads
    33

    #Zarr

    CI

    A MoonBit implementation of the Zarr v2 and v3 storage formats for chunked N-dimensional arrays. This is an independent implementation, not an official Zarr Developers project.

    #Status

    Early Mooncakes release (version 0.2.0). The library can read and write bool, signed/unsigned 8/16/32/64-bit integers, and float32/float64 arrays in memory and on native filesystems, including element access, rectangular slices, missing-chunk fill values and edge chunks. It supports raw chunks and gzip/zstd in both formats, plus zlib in v2, and a documented Blosc subset. Groups and attributes can be read and written for both formats. Native filesystem arrays can also read v3 sharding-indexed stores through the same typed API; sharded writes are not supported. A native-only HTTP adapter can download ordinary metadata and encoded chunks touched by a bounded rectangle into a MemoryStore; it does not hydrate shards. Rectangular slices batch file I/O and codec work by touched chunk, but still buffer the requested result and are not a streaming typed-array interface. Use it for the documented subset only; other dtypes/codecs and cloud object-store adapters are not implemented yet.

    Float32/float64 fill values also support the standard JSON strings "NaN", "Infinity", and "-Infinity". Creating an array with any NaN writes the canonical "NaN" fill value; v3 payload-specific hexadecimal NaN fills are not supported.

    CapabilityZarr v2Zarr v3
    Parse regular-grid array metadataYes, common string dtypesYes, common string data types
    Metadata and chunk keys. and / separatorsdefault and v2-compatible encodings
    Raw encoded chunk storageMemory and native filesystemMemory and native filesystem
    bool element and slice I/OMemory and native filesystemMemory and native filesystem; bytes codec
    uint8 element and slice I/OMemory and native filesystem; C/F orderMemory and native filesystem; bytes codec
    int8 element and slice I/O (Int API, range-checked)Memory and native filesystem; C/F orderMemory and native filesystem; bytes codec
    uint16 element and slice I/OMemory and native filesystem; little/big endian, C/F orderMemory and native filesystem; little/big endian bytes codec
    int16 element and slice I/OMemory and native filesystem; little/big endian, C/F orderMemory and native filesystem; little/big endian bytes codec
    uint32 element and slice I/OMemory and native filesystem; little/big endian, C/F orderMemory and native filesystem; little/big endian bytes codec
    float64 element I/OMemory and native filesystem; little/big endian, C/F orderMemory and native filesystem; little/big endian bytes codec
    float32 element I/OMemory and native filesystem; little/big endian, C/F orderMemory and native filesystem; little/big endian bytes codec
    int32 element I/OMemory and native filesystem; little/big endian, C/F orderMemory and native filesystem; little/big endian bytes codec
    int64 element and slice I/OMemory and native filesystem; little/big endian, C/F orderMemory and native filesystem; little/big endian bytes codec
    uint64 element and slice I/OMemory and native filesystem; little/big endian, C/F orderMemory and native filesystem; little/big endian bytes codec
    Rectangular slice I/O (all listed types)Memory and native filesystemMemory and native filesystem
    gzip compression (levels 0–9)Read/writeRead/write after bytes codec
    zstd compression (levels 0–22)Read/writeRead/write after bytes codec, with or without frame checksum
    zlib compression (levels 0–9)Read/writeNot a v3 core codec
    Blosc1 frames (LZ4/LZ4HC/Zlib/Zstd; no, byte or bit shuffle)LZ4 read/write; LZ4HC/Zlib/Zstd read-only; BloscLZ unsupportedLZ4 read/write after bytes codec; LZ4HC/Zlib/Zstd read-only; BloscLZ unsupported
    Other dtypes, filters and codecsPlannedPlanned
    Groups, ancestors and attributesMemory and native filesystem (.zgroup/.zattrs)Memory and native filesystem (zarr.json)
    Consolidated metadataRead-only native .zmetadata snapshotRead-only native inline root zarr.json snapshot (zarr-python convention)
    HTTP read-only regional hydrationNative, bounded metadata/chunk downloadsNative, bounded metadata/chunk downloads
    Sharding-indexed codecNot a v2 codecNative filesystem reads only: start/end index, little/big-endian index, optional CRC32C, supported inner bytes/compression codecs; no writes or HTTP/memory hydration
    Cloud object-store adaptersPlannedPlanned

    #Install and try

    Add the published module to a MoonBit project:

    moon add zlhahaha/zarr@0.2.0

    To run the repository's examples, clone the public repository:

    git clone https://github.com/zlhahaha/zarr.git
cd zarr
moon update
moon run --target wasm-gc cmd/main
moon run --target native cmd/native_demo
moon run --target native cmd/v2_demo

    The native examples create temporary v3 and v2 stores, reopen them, verify slice values, and clean up. They require a MoonBit installation with native support. See docs/USAGE.md for typed API examples and limits.

    #Build and run

    Requires the MoonBit toolchain. CI checks native builds and tests on Linux, macOS and Windows, plus wasm-gc on Linux; release packaging and Python interoperability run on Linux. From this directory:

    moon check --deny-warn
moon build
moon test --deny-warn
moon run cmd/main
moon test --target native --deny-warn
moon run --target native cmd/native_demo
moon run --target native cmd/v2_demo

    The example creates a v3 uint8 array and writes a slice across chunks. It prints Zarr v3 uint8 values: 0,7,9,11. The first native example creates and reopens a v3 zstd-compressed uint16 array in a temporary filesystem store. The v2 example does the same with a gzip-compressed, big-endian int16 array and verifies a rectangular read after reopening. All examples exit with an error if their checks fail. See docs/USAGE.md for the public API and format limits.

    #Interoperability tests

    Native tests read forty-three committed v2/v3 sample stores generated by zarr-python 3.4.0, covering numeric and boolean types, signed bytes, full-range 64-bit integers, non-finite floating fills, gzip/zlib/zstd including checked frames, Blosc LZ4/Zstd with byte and bit shuffle, nested groups, both consolidated-metadata conventions, and four v3 sharding-indexed layouts. The generators include scripts/generate_sharding_fixtures.py, scripts/generate_blosc_fixtures.py, scripts/generate_blosc_bitshuffle_fixtures.py, scripts/generate_zstd_checksum_fixture.py, and scripts/generate_v3_consolidated_fixture.py alongside the other scripts listed in the fixture inventory. In the reverse direction, the cmd/interop, cmd/nonfinite_interop, cmd/int64_interop, cmd/int8_interop, and cmd/blosc_interop generators write thirty-two stores—including v2/v3 Blosc LZ4 frames, a multi-chunk compressed slice, boolean masks, non-finite fills, full-range 64-bit integers, signed bytes and v2/v3 nested groups—to the ignored integration/.roundtrip/ directory. The main verifier and Blosc verifier open them with zarr-python. CI installs the pinned Python test oracle and runs both directions. The generators refuse to overwrite existing stores; remove only their generated directories before re-running locally.

    #Design

    Both formats share a storage-neutral array API, but retain their distinct metadata, chunk-key, dtype, and codec rules. Unknown or unsupported encodings must fail explicitly instead of returning incorrect values. The MoonBit packages follow one-way dependencies:

    metadata/        Parse and validate v2/v3 array and group documents
chunk/           Regular-grid indexing and metadata/chunk keys
shard/           V3 sharding layout and bounded index validation
dtype/           Numeric dtype and fill-value checks
codec/           Codec-chain validation and bounded gzip/zlib/zstd decode
store/           In-memory encoded-byte store
store/fs/        Native-only filesystem store
store/http/      Native-only read-only HTTP regional hydration
array/           Typed array creation, element and region I/O
hierarchy/       Group and attribute operations
integration/     Cross-package tests
cmd/main/        Portable in-memory v3 example
cmd/native_demo/ Native filesystem v3 example
cmd/v2_demo/     Native filesystem v2 example

    The public array functions live in zlhahaha/zarr/array; MemoryStore lives in zlhahaha/zarr/store, and format constants in zlhahaha/zarr/metadata. For example, @array.create_u8(store, @metadata.V3, "samples", [4], [2], b'\x00', compression=@codec.Zstd(3)) creates a compressed v3 array without hand-writing metadata JSON. On native, FileStore::create_u8 accepts the same compression option. gzip/zlib output is bounded while decoding; zstd frames are preflight-checked against the declared chunk size using the dependency's frame-size bound. Thus, valid zstd frames without a known content size may be rejected if the conservative bound exceeds the chunk size. See the roadmap for the staged interoperability targets.

    #Specification and attribution

    This project implements the publicly documented Zarr v2 storage specification, Zarr v3 core specification, and sharding-indexed codec specification. The Zarr specifications repository is CC BY 4.0. The Blosc codec follows the public C-Blosc chunk-format description and LZ4 block format (the reference projects have BSD-style licenses); no upstream implementation code was copied. The committed interoperability stores were generated by this repository's original scripts using zarr-python 3.4.0, which is MIT-licensed; see integration/fixtures/zarr-python/README.md. Compression uses the Apache-2.0 MoonBit dependencies moonbit-community/flate and Milky2018/zstd.

    This project is licensed under Apache-2.0; see LICENSE.

    Powered by MoonBit

    Site sourceReport issuePackagesBuild queueSkillsStatistics

    © 2026 mooncakes.io