StatefulIterators
Typed, stream-like iterables for Julia.
The following stream-like commands are supported: read(), read!(), readbytes(), readbytes!(), position(), skip(), seek(), seekstart(), seekend(), eof(), readuntil() and readline().
In addition, available() and peek() have been added.
Example
julia> using StatefulIterators
julia> s = StatefulIterator([1,2,3,4,5])
StatefulIterators.StatefulIterator{Array{Int64,1},Int64}([1,2,3,4,5],1)
julia> collect(take(s, 2))
2-element Array{Any,1}:
1
2
julia> eltype(s)
Int64
julia> read(s)
3
julia> readbytes(s)
16-element Array{UInt8,1}:
0x04
0x00
0x00
0x00
0x00
0x00
0x00
0x00
0x05
0x00
0x00
0x00
0x00
0x00
0x00
0x00
Types
Unlike Julia's IOStream
, stateful iterators have an underlying type,
given by eltype(s)
. This affects the behaviour of various methods
as follows:
-
default type for read(s, dims...) and similar methods is the underlying type.
-
iterator chunking is on the underlying type.
The second point is subtle and affects how types that are smaller than the underlying type are read:
julia> s = StatefulIterator(Int16[1,2,3])
StatefulIterators.StatefulIterator{Array{Int16,1},Int64}(Int16[1,2,3],1)
julia> read(s, UInt8, 2)
2-element Array{UInt8,1}:
0x01
0x00
julia> read(s, UInt8)
0x02
julia> read(s, UInt8)
0x03
Note above how the second and third reads start from the next
UInt16
. In comparison, the first read extracts both bytes (UInt8
)
from a single UInt16
.
Explicit State
Not all iterables follow the "spirit" of the iter interface - the most
common exception is Task
. These types do not have a state that can
be saved and restored, and so some methods - copy()
, peek()
,
position()
, seek()
, seekstart()
, and available()
- are not
supported:
julia> s = StatefulIterator(Task(() -> (for i in 1:3; produce(i); end)))
StatefulIterators.StatefulIterator{Task,Void}(Task (runnable) @0x00007f04c42e8fb0,nothing)
julia> read(s)
1
julia> available(s)
ERROR: Task lacks explicit state
in available at /home/andrew/.julia/v0.4/StatefulIterators/src/StatefulIterators.jl:289
Bits Types
Methods involving type conversion are only supported when the underlying type is a bits type:
julia> s = StatefulIterator([0x01, 0x02])
StatefulIterators.StatefulIterator{Array{UInt8,1},Int64}(UInt8[0x01,0x02],1)
julia> read(s, UInt16)
0x0201
julia> s = StatefulIterator([0x01, "hello world"])
StatefulIterators.StatefulIterator{Array{Any,1},Int64}(Any[0x01,"hello world"],1)
julia> read(s, UInt16)
ERROR: argument is an abstract type; size is indeterminate
in read at /home/andrew/.julia/v0.4/StatefulIterators/src/StatefulIterators.jl:199
State Is Not (Always) An Offset
The value returned by position()
and taken by seek()
is the
state of the corresponding iterable. It may not be an integer.
The function skip()
, however, does take an integer. This is the
number of instances of the underlying type (not necessarily bytes) to
move over.
Warnings
This is less efficient than using normal iterators
(ref).
A simple summation (using sum()
) of 1 million elements is about
twice as slow when using a stateful iterator, compared to using a bare
array (but allocates no more memory).
Credits
Thanks to okvs for a more efficient data structure, a more efficient inner loop, and various other good ideas.