DiskDataProviders.jl

Disk based, buffered data structures for machine learning
Author baggepinnen
Popularity
8 Stars
Updated Last
7 Months Ago
Started In
August 2019

Build Status codecov

DiskDataProviders.jl

This package implements datastructures that are iterable and backed by a buffer that is fed by data from disk. The reading is done on a separate thread, so make sure Julia is started with at least two threads.

Intended usage: To buffer data reading from disk when training convolutional neural networks (1d or 2d) using Flux.jl. This allows the CPU to work with the disk and data while the GPU is working on the training. This package might be useful for other things as well.

For usage example, see the documentation
For lower-level buffered iterators, see LengthChannels.jl

Used By Packages

No packages found.