🥮 mooncakes.io
    Sign in

    yugonlian/moon-avro/ocf does not have a README file

    OcfError

    source
    pub(all) suberror OcfError {
      InvalidMagic
      InvalidSyncMarkerLength(actual~ : Int)
      UnsupportedCodec(String)
      InvalidMetadataKey(String)
      Truncated(offset~ : Int)
      InvalidBlockCount(count~ : Int64)
      BlockTooLarge(length~ : Int, limit~ : Int)
      TooManyRecords(count~ : Int64, limit~ : Int)
      SyncMismatch(block_index~ : Int)
      HeaderSchema(
    SchemaError    )
      Datum(
    CodecError    )
      Compression(codec~ : String, message~ : String)
    } derive(Eq, 
    Debug    )

    Failures produced while reading or writing an Avro Object Container File.

    All offsets are byte offsets in the container. Block indices are zero based.

    CompressionCodec

    source
    pub(all) enum CompressionCodec {
      Null
      Deflate
    } derive(Eq, 
    Debug    )

    OCF block compression codecs supported by this release line.

    CompressionCodec::name

    source
    fn CompressionCodec::name(self : CompressionCodec) -> String

    CompressionCodec::parse

    source
    fn CompressionCodec::parse(name : String) -> CompressionCodec raise OcfError

    OcfBlockConfig

    source
    pub struct OcfBlockConfig {
      max_records : Int
      target_bytes : Int
    } derive(Eq, 
    Debug    )

    Controls how an in-memory OCF writer groups records into data blocks.

    max_records bounds the number of records in one block. target_bytes is a soft limit for the uncompressed datum payload: a single record larger than the target is still emitted in its own block as long as it satisfies the hard OcfLimits.max_block_bytes bound.

    OcfBlockConfig::max_records

    source
    fn OcfBlockConfig::max_records(self : OcfBlockConfig) -> Int

    OcfBlockConfig::new

    source
    fn OcfBlockConfig::new(max_records? : Int, target_bytes? : Int) -> OcfBlockConfig raise OcfError

    Construct a block policy for multi-block OCF output.

    OcfBlockConfig::target_bytes

    source
    fn OcfBlockConfig::target_bytes(self : OcfBlockConfig) -> Int

    OcfHeader

    source
    pub struct OcfHeader {
      schema : 
    Schema
      codec : CompressionCodec
      metadata : Map[String, Bytes]
      sync_marker : Bytes
    } derive(Eq, 
    Debug    )

    Parsed OCF header. The metadata map is kept intact so applications do not lose vendor-specific fields.

    OcfHeader::codec

    source
    fn OcfHeader::codec(self : OcfHeader) -> CompressionCodec

    OcfHeader::metadata

    source
    fn OcfHeader::metadata(self : OcfHeader) -> Map[String, Bytes]

    OcfHeader::new

    source
    fn OcfHeader::new(schema : 
    Schema    , codec? : CompressionCodec, metadata? : Map[String, Bytes], sync_marker : Bytes) -> OcfHeader raise OcfError

    OcfHeader::schema

    source

    OcfHeader::sync_marker

    source
    fn OcfHeader::sync_marker(self : OcfHeader) -> Bytes

    OcfLimits

    source
    pub(all) struct OcfLimits {
      max_metadata_entries : Int
      max_metadata_bytes : Int
      max_block_bytes : Int
      max_records_per_block : Int
    } derive(Eq, 
    Debug    )

    Bounds applied before OCF decoding allocates memory or visits records.

    OcfLimits::new

    source
    fn OcfLimits::new(max_metadata_entries~ : Int, max_metadata_bytes~ : Int, max_block_bytes~ : Int, max_records_per_block~ : Int) -> OcfLimits raise OcfError

    Build explicit OCF bounds. All bounds must be positive.

    OcfReader

    source
    pub struct OcfReader {
      header : OcfHeader
      records : Array[
    Datum    ]
      index : Int
    } derive(
    Debug    )

    Ordered in-memory OCF reader. next returns records in block/file order.

    OcfReader::header

    source
    fn OcfReader::header(self : OcfReader) -> OcfHeader

    OcfReader::next

    source

    OcfReader::open

    source
    fn OcfReader::open(input : Bytes, limits? : OcfLimits) -> OcfReader raise OcfError

    OcfReader::remaining

    source
    fn OcfReader::remaining(self : OcfReader) -> Int

    OcfWriter

    source
    pub struct OcfWriter {
      header : OcfHeader
      limits : OcfLimits
      block_config : OcfBlockConfig
      records : Array[
    Datum    ]
      finished : Bool
    } derive(
    Debug    )

    Ordered in-memory OCF writer. Records are appended in schema order and partitioned into bounded data blocks when finish is called.

    OcfWriter::append

    source
    fn OcfWriter::append(self : OcfWriter, record : 
    Datum    ) -> Unit raise OcfError

    OcfWriter::finish

    source
    fn OcfWriter::finish(self : OcfWriter) -> Bytes raise OcfError

    OcfWriter::new

    source
    fn OcfWriter::new(header : OcfHeader, limits? : OcfLimits, block_config? : OcfBlockConfig) -> OcfWriter

    OcfWriter::record_count

    source
    fn OcfWriter::record_count(self : OcfWriter) -> Int

    decode_container

    source
    fn decode_container(input : Bytes, limits? : OcfLimits) -> (OcfHeader, Array[
    Datum    ]) raise OcfError

    Decode an entire OCF file, checking block sizes, record counts, sync markers and all datum payloads against the schema carried by its header.

    decode_header

    source
    fn decode_header(input : Bytes, limits? : OcfLimits) -> (OcfHeader, Int) raise OcfError

    Decode only an OCF header. bytes_consumed is the first block offset.

    default_block_config

    source
    fn default_block_config() -> OcfBlockConfig

    A balanced default for ordinary in-memory files: at most 1,000 records or approximately 256 KiB of uncompressed payload per data block.

    default_limits

    source
    fn default_limits() -> OcfLimits

    Conservative defaults suitable for untrusted in-memory OCF input.

    encode_container

    source
    fn encode_container(header : OcfHeader, records : Array[
    Datum    ], limits? : OcfLimits) -> Bytes raise OcfError

    Encode one complete Avro Object Container File as a single data block. Use encode_container_blocks when output should be partitioned for bounded decompression and incremental interoperability.

    encode_container_blocks

    source
    fn encode_container_blocks(header : OcfHeader, records : Array[
    Datum    ], config? : OcfBlockConfig, limits? : OcfLimits) -> Bytes raise OcfError

    Encode one complete OCF file using independently compressed data blocks. Blocks are closed when either configured record count or target uncompressed byte size is reached. Record order is preserved.

    encode_header

    source
    fn encode_header(header : OcfHeader) -> Bytes raise OcfError

    Serialize an OCF header (magic, metadata and 16-byte sync marker).

    is_ocf

    source
    fn is_ocf(data : Bytes) -> Bool

    magic

    source
    fn magic() -> Bytes

    Apache Avro Object Container File magic bytes.

    Source Files

    Powered by MoonBit

    Site sourceReport issuePackagesBuild queueSkillsStatistics

    © 2026 mooncakes.io