Time-series analysis using the matrix profile. The matrix profile
P tells you which sub-sequences of a time series
T are similar to each other, and which are most dissimilar from all other. This will allow you to find repeated patterns, or motifs, as well as finding outliers and anomalies. Here's a blog post that introduces the matrix profile with lots of nice figures and examples: https://towardsdatascience.com/introduction-to-matrix-profiles-5568f3375d90
This package is registered and can be installed with
using Pkg pkg"add MatrixProfile"
matrix_profile returns the matrix profile and profile indices. Here's an example where we insert a repeated pattern in an otherwise random time series.
using MatrixProfile, Plots t = range(0, stop=1, step=1/10) y0 = sin.(2pi .* t) T = [randn(20); y0; randn(20); y0; randn(20)] window_length = length(y0) profile = matrix_profile(T, window_length) plot(profile) # Should have minima at 21 and 52
The matrix profile have two sharp minima at the onsets of the repeated pattern. The parameter
window_length determines how long pattern to search for.
Analysis across different time-series
If called like
profile = matrix_profile(A, B, m, [dist])
consecutive windows of
A will be compared to the entire
B. The resulting matrix profile will have a length that depends on
B, and indicate with small values when a window of
A appeared in
B, and with large values when no window in
A matched the corresponding window in
B. This is not a symmetric function, in general,
matrix_profile(A, B) != matrix_profile(B, A).
matrix_profile benefits greatly in speed from the use of
Float32 instead of
Float64, but may accumulate some error for very long time series (> 10⁶ perhaps). The computational time scales as the square of the length of
T, but is invariant to the window length. Calculating the matrix profile of
2^17 ≈ 100k points takes less than minute on a laptop.
dist is provided, a generic (slow) method is used. If
dist is not provided and the inputs
A,B are one dimensional vectors of numbers, a fast method is used. The fast method handles long time series,
length(A) = length(B) = 100k takes less than 30s.
If the time-series is sampled very fast in relation to the time scale on which interesting things happen, you may try the function
resample(T, fraction::Real) to reduce the amount of data to process. Example,
Using the fake data from the example above, we can do
k = 2 mot = motifs(profile, k; r=2, th=5) plot(profile, mot) # plot(mot) # Motifs can be plotted on their own for a different view.
kis the number of motifs to extract
rcontrols how similar two windows must be to belong to the same motif. A higher value leads to more windows being grouped together.
this a threshold on how nearby in time two motifs are allowed to be.
Also see the function
anomalies(profile) to find anomalies (or outliers) in the data, sometimes called discords.
Arbitrary metrics and input types
The matrix profile can be computed for any sequence of things that has a "time axis" and a notion of distance. The examples so far have dealt with one-dimensional arrays of real numbers with the Euclidean metric, for which the matrix profile is particularly efficient to compute. We do not have to limit ourselves to this setting, though, and
matrix_profile accepts any array-like object and any distance function on the form
dist(x,y). The interface looks like this
profile = matrix_profile(T, m, dist)
T is a high-dimensional array, time is considered to be the last axis.
T can also be a vector of any arbitrary julia objects for which the function
dist(x,y) is defined. Note that if
T has a long time dimensions, the matrix profile will be expensive to compute, 𝒪(n²log(n)). This method does not make use of the STOMP algorithm, since this is limited to one-dimensional data under the Euclidean metric.
Segmentation / change-point detection
The most likely segmentation of a time series into two is calculated using
segment(p::Profile). A more detailed analysis can be performed using
sp = segment_profile(p::Profile) which returns a vector of the same length as
p, where a low value at index
i indicates that few nearest-neighbor arcs pass over index
sp thus form sort-of a "segmentation profile".
Time series snippets
To summarize a time series in the form of a small number of snippets, we have the function
snips = snippets(T, 3, 100) plot(snips)
The arguments to
- The time series
- The desired number of snippets
- The length of each snippet
m: the length of a small subsequence to be used internally, defaults to 10% of the snippet length.
This function can take a while to run for long time-series, for
length(T) = 15k, it takes less than a minute on a laptop. The time depends strongly on the internal window length parameter.
- The STOMP algorithm used in
matrix_profileis detailed in the paper Matrix profile II.
- The algorithm used in
segment_profilecomes from Matrix Profile VIII
- The MP distance is described in Matrix profile XII
- The algorithm for extraction of time-series snippets comes from Matrix profile XIII