Struct FileWriterCore 

pub struct FileWriterCore { /* private fields */ }

Writes the Hurray file container into bytes a driver sends onward.

Each tensor, and the footer, is submitted as one operation; the core emits the descriptor, the alignment padding and each payload in turn, and says when the item is written. The footer index is accumulated as it goes, so nothing has to be re-read and no offset is ever patched.

Examples

use std::io::Write;

use hurray_core::{
    BufferHandle, DeviceTag, ElementType, LayoutDescriptor, Shape, SyncMode,
    TensorDescriptor, MIN_BUFFER_ALIGNMENT,
};
use hurray_io::file::FileWriterCore;
use hurray_io::stream::WriteStep;

let handle =
    BufferHandle::new(64, MIN_BUFFER_ALIGNMENT, DeviceTag::Cpu, SyncMode::ProducerSynced)?;
let descriptor = TensorDescriptor::new(
    1, 0, ElementType::Float32, Shape::new(vec![4u64, 4])?, 0,
    LayoutDescriptor::RowMajor, vec![handle], None, None, None, None,
)?;
let payload = [7u8; 64];
// Bound, not a temporary: the operation borrows the payloads until the item is
// written, which is ADR-041 § 3's lifetime rule showing up in the type system.
let buffers: [&[u8]; 1] = [&payload];

/// Drives one operation to completion over any `io::Write`.
fn drive(mut op: hurray_io::file::FileWriteOp<'_>, sink: &mut Vec<u8>) -> hurray_io::Result<()> {
    while let WriteStep::Output { .. } = op.step()? {
        let src = op.output_src();
        sink.write_all(src).map_err(hurray_io::Error::Io)?;
        let n = src.len();
        op.report_written(n)?;
    }
    Ok(())
}

let mut file = Vec::new();
let mut core = FileWriterCore::new();
drive(core.header()?, &mut file)?;
drive(core.tensor("embeddings", &descriptor, &buffers)?, &mut file)?;
drive(core.finish(vec![])?, &mut file)?;

assert_eq!(&file[0..8], b"HRRYFILE");
assert_eq!(&file[file.len() - 4..], b"HRRY");

Implementations

impl FileWriterCore

pub fn new() -> Self

Creates a core with default options: 4096-byte data alignment, unsorted index.

Examples

See the type-level example.

pub fn with_options(options: FileWriterOptions) -> Result<Self>

Creates a core with custom options.

Errors

Error::InvalidHeader — data_buffer_alignment is not a power of two in [4096, 2097152].

Examples

use hurray_io::file::{FileWriterCore, FileWriterOptions};

let core = FileWriterCore::with_options(FileWriterOptions {
    data_buffer_alignment: 8192,
    sorted_index: true,
})?;
assert!(!core.is_unusable());

pub fn header(&mut self) -> Result<FileWriteOp<'_>>

Emits the 64-byte file header.

Drive this before the first tensor: a core with no header written refuses tensor and finish.

Errors

• Error::CoreUnusable — the core is poisoned
• Error::CoreFailed — the header is already written, or the file is finished

Examples

See the type-level example.

pub fn tensor<'a>( &'a mut self, name: &str, descriptor: &TensorDescriptor, buffers: &'a [&'a [u8]], ) -> Result<FileWriteOp<'a>>

Submits one tensor: its descriptor, its alignment padding, and its payloads.

The returned operation borrows buffers until it reports WriteStep::ItemWritten; the core holds no pointer into them afterwards (§ 3). The tensor’s index entry is recorded at that same point.

Errors

• Error::TensorNameEmpty / Error::TensorNameTooLong — invalid name
• Error::DuplicateTensorName — the name is already in this file
• Error::MultiBufferLengthMismatch — buffers.len() is not the descriptor’s buffer count
• Error::BufferSizeMismatch — a slice length differs from its handle’s byte_size
• Error::Core — the descriptor does not encode
• Error::CoreFailed — the footer has already been written
• Error::CoreUnusable — the sink was poisoned mid-item

Examples

See the type-level example.

pub fn finish(&mut self, kv: Vec<(String, KvValue)>) -> Result<FileWriteOp<'_>>

Submits the footer: the KV section, the index, and the 40-byte trailer.

Errors

• Error::KvKeyEmpty / Error::KvKeyTooLong / Error::DuplicateKvKey — invalid KV keys
• Error::InvalidHeader — a KV array is empty, nested, or not homogeneous
• Error::CoreFailed — the footer has already been written
• Error::CoreUnusable — the sink was poisoned mid-item

Examples

See the type-level example.

pub fn is_unusable(&self) -> bool

Whether a write put an unknown number of bytes on the sink, leaving the file torn.

Examples

use hurray_io::file::FileWriterCore;

assert!(!FileWriterCore::new().is_unusable());

pub fn poison(&mut self, reason: Unusable)

Retires the core because the output can no longer be completed.

For a unit larger than one item — a composite group, whose head is already on the sink when a member fails. A FileWriteOp cannot express that: the failure may come from submission, with no operation in hand.

Public because the driver that needs it may live in another crate. ADR-041 § 1 puts drivers at the edge, and a group spanning several operations is the driver’s unit, not the core’s — so a binding outside hurray-io writing composites needs the same tool FileWriter uses, exactly as WriteOp::mark_unusable is public for the single-operation case.

Examples

use hurray_io::file::FileWriterCore;
use hurray_io::stream::Unusable;

let mut core = FileWriterCore::new();
core.poison(Unusable::IncompleteOutput);
assert!(core.is_unusable());

Trait Implementations

impl Debug for FileWriterCore

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for FileWriterCore

fn default() -> Self

Returns the “default value” for a type.