grain.sources.ArrayRecordDataSource

grain.sources.ArrayRecordDataSource#

class grain.sources.ArrayRecordDataSource(*args, **kwargs)#

Data source for ArrayRecord files.

Parameters:

  • paths (array_record.python.array_record_data_source.PathLikeOrFileInstruction | Sequence[array_record.python.array_record_data_source.PathLikeOrFileInstruction])

  • reader_options (dict[str, str] | None)

__init__(paths, reader_options=None)#

Creates a new ArrayRecordDataSource object.

See array_record.ArrayRecordDataSource for more details.

Parameters:

  • paths (array_record.python.array_record_data_source.PathLikeOrFileInstruction | Sequence[array_record.python.array_record_data_source.PathLikeOrFileInstruction]) – A single path/FileInstruction or list of paths/FileInstructions.

  • reader_options (dict[str, str] | None) – a dict[str, str] to be passed when creating a reader. For example, {index_storage_option:”in_memory”} stores the reader indices in memory versus {index_storage_option:”offloaded”} stores the indices on disk to save memory usage.

Methods

__init__(paths[, reader_options])

Creates a new ArrayRecordDataSource object.

Attributes

paths

Contents