Struct FileReaderCore 

pub struct FileReaderCore<A: DestinationAlloc = SystemDestinations> { /* private fields */ }

Reads a Hurray file from bytes a driver fetches by range.

The core owns the memory every payload byte lands in, so the transport’s write into it is the only copy (ADR-041 § 2), and that memory carries the alignment the buffer handle declares (§ 5).

Every request is absolute, which is what makes this core unloseable. The read position advances only when a count is reported, so a driver that abandons a transfer — because its future was cancelled, or because a count came back it could not trust — simply calls step again and is handed the same range. Re-filling a partly filled destination from the start of the range is idempotent. The stream core has an Unusable state for a lost byte count; this one needs none, and for the same reason a job that fails leaves the core usable: the next begin_* starts from the index, not from wherever the file pointer happened to stop.

Examples

Reading a file with no runtime anywhere — the driver is three std::io calls:

use std::io::{Read, Seek, SeekFrom};

use hurray_io::file::{FileReaderCore, FileStep};
let mut file = std::io::Cursor::new(bytes);
let size = file.seek(SeekFrom::End(0))?;

let mut core = FileReaderCore::new(size);
// Open: header, trailer, index, KV.
loop {
    match core.step()? {
        FileStep::Range { offset, len } => {
            file.seek(SeekFrom::Start(offset))?;
            let n = file.read(&mut core.input_dst()[..len])?;
            core.report_filled(n)?;
        }
        FileStep::Ready => break,
        other => panic!("unexpected {other:?} while opening"),
    }
}
assert_eq!(core.index().len(), 1);

// Then one tensor, by name.
core.begin_tensor("embeddings")?;
let tensor = loop {
    match core.step()? {
        FileStep::Range { offset, len } => {
            file.seek(SeekFrom::Start(offset))?;
            let n = file.read(&mut core.input_dst()[..len])?;
            core.report_filled(n)?;
        }
        FileStep::Tensor => break core.take_tensor().expect("a tensor is ready"),
        other => panic!("unexpected {other:?} while reading"),
    }
};
assert_eq!(tensor.buffers[0].as_ref(), &payload[..]);
// The destination carried the declared alignment, so a SIMD load is safe on it.
assert_eq!(tensor.buffers[0].as_ptr() as usize % 64, 0);

Implementations

impl FileReaderCore<SystemDestinations>

pub fn new(size: u64) -> Self

Creates a core for a file of size bytes, reading into host allocations.

The size is a constructor argument rather than a request the core makes: every driver already knows it — a seek to the end, a stat, or the transport vtable’s size callback — and a size the core cannot act on buys nothing.

Examples

See the type-level example.

impl<A: DestinationAlloc> FileReaderCore<A>

pub fn with_destinations(size: u64, destinations: A) -> Self

Creates a core whose read destinations come from destinations.

The seam ADR-041 left open: a binding may supply the allocation so a read lands directly in memory its runtime owns, as long as the destination satisfies the alignment the handle declares.

Examples

use hurray_io::file::FileReaderCore;
use hurray_io::stream::SystemDestinations;

let core = FileReaderCore::with_destinations(4096, SystemDestinations);
assert!(!core.is_open());

pub fn with_max_composite_depth(self, max_composite_depth: usize) -> Self

Sets the maximum composite nesting depth accepted by begin_composite. Default: DEFAULT_MAX_COMPOSITE_DEPTH.

Examples

use hurray_io::file::FileReaderCore;

let core = FileReaderCore::new(4096).with_max_composite_depth(8);
assert!(!core.is_open());

pub fn step(&mut self) -> Result<FileStep>

Advances the machine and returns what it needs next.

Never blocks and never waits: every transition here is a parse, an allocation or bookkeeping, which is what lets one core serve a blocking driver and an event loop alike (ADR-041 § 4).

Errors

• Error::InvalidFileMagic, Error::UnsupportedContainerVersion, Error::ReservedFileFlagBits, Error::InvalidTrailerMagic, Error::IndexOverrunsTrailer, Error::IndexCrc32cMismatch, Error::DuplicateTensorName — the file’s own structure is wrong
• Error::Core — a descriptor failed to decode
• Error::UnallocatableBuffer — a destination the allocator cannot provide
• Error::TornComposite, Error::CompositeNestingTooDeep, Error::NotAComposite — a composite walk could not be completed
• Error::CoreFailed — the open sequence failed earlier

Examples

See the type-level example.

pub fn input_dst(&mut self) -> &mut [u8]

The memory the current FileStep::Range should be read into.

Exactly as long as the outstanding FileStep::Range’s len, and its address does not move while that request stands, so a driver may hold a pointer into it across a suspension (§ 6a).

Examples

See the type-level example.

pub fn report_filled(&mut self, n: usize) -> Result<()>

Reports that n bytes of the outstanding range were read.

Records a count and nothing else: every parse, allocation and state change happens in step, which keeps the window a cancellation can land in as narrow as possible (§ 6).

Errors

• Error::UnexpectedEof — n was 0: the file is shorter than its index claims
• Error::InvalidProgressReport — n exceeded the bytes offered, or no range was outstanding

Examples

See the type-level example.

pub fn begin_descriptor(&mut self, name: &str) -> Result<()>

Starts reading the descriptor for name, without reading any buffer data.

Supersedes any job already in flight: the index makes every offset absolute, so there is nothing to unwind.

Errors

• Error::TensorNotFound — no tensor of that name is in the index
• Error::CoreFailed — the open sequence failed, or has not finished

Examples

See the type-level example.

pub fn begin_tensor(&mut self, name: &str) -> Result<()>

Starts reading the complete tensor named name: descriptor, then every buffer.

Supersedes any job already in flight.

Errors

As begin_descriptor.

Examples

See the type-level example.

pub fn begin_composite(&mut self, head_name: &str) -> Result<()>

Starts reading the composite whose head is named head_name, reassembling the head with its members.

Errors

• Error::TensorNotFound — no tensor of that name is in the index
• Error::CoreFailed — the open sequence failed, or has not finished

Walking it may further fail with Error::NotAComposite, Error::TornComposite or Error::CompositeNestingTooDeep, reported from step.

Examples

core.begin_composite("weight")?;

pub fn take_descriptor(&mut self) -> Option<TensorDescriptor>

Takes the descriptor FileStep::Descriptor announced, leaving the core ready.

Examples

See the type-level example.

pub fn take_tensor(&mut self) -> Option<FileTensor>

Takes the tensor FileStep::Tensor announced, leaving the core ready.

Examples

See the type-level example.

pub fn take_composite(&mut self) -> Option<FileComposite>

Takes the composite FileStep::Composite announced, leaving the core ready.

Examples

if let Some(composite) = core.take_composite() {
    println!("{} member(s)", composite.members.len());
}

pub fn index(&self) -> &[IndexEntry]

The footer index: one entry per tensor, in the order the file records it.

Empty until FileStep::Ready.

Examples

See the type-level example.

pub fn kv(&self) -> &[(String, KvValue)]

The file-level KV metadata, empty if the file has none.

Examples

use hurray_io::file::FileReaderCore;

let core = FileReaderCore::new(4096);
assert!(core.kv().is_empty());

pub fn is_open(&self) -> bool

Whether the open sequence has finished, so the index and KV are available.

Examples

use hurray_io::file::FileReaderCore;

assert!(!FileReaderCore::new(4096).is_open());

Trait Implementations

impl<A: DestinationAlloc> Debug for FileReaderCore<A>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.