Manage large vectors of bits types in Julia. A thin wrapper for mmapped binary data, with a few sanity checks and convenience functions.
For each dataset, the columns (vectors of equal length) and metadata are stored in a directory like this:
dir/ ├── layout.jld2 ├── meta/ │ └ ... ├── 1.bin ├── 2.bin ├── ... ├── ... └── ...
Additional metadata can be saved as in files in the directory
meta. This is ignored by this library; use the function
to calculate paths relative to
Two interfaces are provided. Use
SinkColumns for an ex ante
unknown number of elements, written sequentially. This is useful for
MmappedColumns is useful when the number of records is known and
Types for the columns are specified as
Tuples. See the docstrings
for both interfaces and the unit tests for examples.
Work on this library was supported by the Austrian National Bank Jubiläumsfonds grant #17378.