K-SVD is an algorithm for creating overcomplete dictionaries for sparse representations.

This package implements:

• K-SVD as described in the original paper: K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation
• Matching Pursuit for representing signals using a given dictionary.

In particular, substantial effort has been put in speeding up this implementation. This includes:

• Custom parallel dense-sparse matrix multiplication via ThreadedDenseSparseMul.jl
• Custom pipelined GPU offloading for matching pursuit (CUDAAcceleratedMatchingPursuit)
• Threaded matching pursuit (ParallelMatchingPursuit) (mostly "embarassingly parallel")
• Threaded ksvd implementation (ParallelKSVD) (not "embarassingly parallel")
• Threaded and batched ksvd implementation (BatchedParallelKSVD) (not "embarassingly parallel")
• Extensive efforts removing allocations by preallocating buffers
• Extensive benchmark-driven optimizations utilizing ProfileView.jl
• Many other modification experiments.

Assume that each column of Y represents a feature vector (or an input signal from some system).
D is a dictionary. Each column of D represents an atom.
K-SVD derives D and X such that DX ≈ Y from only Y.

(; D, X) = ksvd(Y, 256)

# we can control the matching pursuit stage and ksvd stage through method structs
ksvd_update_method = BatchedParallelKSVD{false, Float64}(shuffle_indices=true, batch_size_per_thread=1)
sparse_coding_method = ParallelMatchingPursuit(max_nnz=25, rtol=5e-2)
result = ksvd(Y, 256;
              ksvd_update_method,
              sparse_coding_method,
              maxiters=100,
              abstol=1e-6,
              reltol=1e-6,
              show_trace=true)

# Access additional information
println("Termination condition: ", result.termination_condition)
println("Norm results: ", result.norm_results)
println("NNZ per column results: ", result.nnz_per_col_results)
println("Timing information: ", result.timer)

Of course we can also just run one step of matching pursuit/sparse coding, or one step of the ksvd update:

basis = KSVD.init_dictionary(size(Y, 1), 2*size(Y,2))
X = sparse_coding(OrthogonalMatchingPursuit(max_nnz=25), Y, basis)

(; D, X) = ksvd_update(ksvd_update_method, Y, basis, X)

Matching Pursuit derives X from D and Y such that DX = Y in constraint that X be as sparse as possible.

Here is an overview of the performance improvements in the ksvd_update provided in this package, broken down by computation type. The data is computed using different commits on the experiments branch. More details will be added later.