ParameterSchedulers.jl

Common hyperparameter scheduling for ML
Author FluxML
Popularity
26 Stars
Updated Last
10 Months Ago
Started In
December 2020

ParameterSchedulers

Dev Build Status

ParameterSchedulers.jl provides common machine learning (ML) schedulers for hyper-parameters. Though this package is framework agnostic, a convenient interface for pairing schedules with Flux.jl optimizers is available. Using this package with Flux is as simple as:

using Flux, ParameterSchedulers
using ParameterSchedulers: Scheduler

opt = Scheduler(Exp= 1e-2, γ = 0.8), Momentum())

Available Schedules

This is a table of the common schedules implemented, but ParameterSchedulers provides utilities for creating more exotic schedules as well. The higher order schedules should make it so that you will rarely need to write a schedule from scratch.

You can read this paper for more information on the schedules below.

{cell=table, display=false, output=false, results=false}

using UnicodePlots, ParameterSchedulers
Schedule Description Type Example

Step(;λ, γ, step_sizes)

Exponential decay by γ every step in step_sizes

Decay

{cell=table, display=false}

using UnicodePlots, ParameterSchedulers
t = 1:10 |> collect
s = Step= 1.0, γ = 0.8, step_sizes = [2, 3, 2])
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)

Exp(;λ, γ)

Exponential decay by γ every iteration

Decay

{cell=table, display=false}

using UnicodePlots, ParameterSchedulers
t = 1:10 |> collect
s = Exp= 1.0, γ = 0.5)
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)

CosAnneal(;λ0, λ1, period)

Cosine annealing

Cyclic

{cell=table, display=false}

using UnicodePlots, ParameterSchedulers
t = 1:10 |> collect
s = CosAnneal(λ0 = 0.0, λ1 = 1.0, period = 4)
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)

Triangle(;λ0, λ1, period)

Triangle wave function

Cyclic

{cell=table, display=false}

using UnicodePlots, ParameterSchedulers
t = 1:10 |> collect
s = Triangle(λ0 = 0.0, λ1 = 1.0, period = 2)
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)

TriangleDecay2(;λ0, λ1, period)

Triangle wave function with half the amplitude every period

Cyclic

{cell=table, display=false}

using UnicodePlots, ParameterSchedulers
t = 1:10 |> collect
s = TriangleDecay2(λ0 = 0.0, λ1 = 1.0, period = 2)
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)

TriangleExp(;λ0, λ1, period, γ)

Triangle wave function with exponential amplitude decay at rate γ

Cyclic

{cell=table, display=false}

using UnicodePlots, ParameterSchedulers
t = 1:10 |> collect
s = TriangleExp(λ0 = 0.0, λ1 = 1.0, period = 2, γ = 0.8)
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)

Poly(;λ, p, max_iter)

Polynomial decay at degree p

Decay

{cell=table, display=false}

using UnicodePlots, ParameterSchedulers
t = 1:10 |> collect
s = Poly= 1.0, p = 2, max_iter = t[end])
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)

Inv(;λ, γ, p)

Inverse decay at rate (1 + tγ)^p

Decay

{cell=table, display=false}

using UnicodePlots, ParameterSchedulers
t = 1:10 |> collect
s = Inv= 1.0, p = 2, γ = 0.8)
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)

Sin(;λ0, λ1, period)

Sine function

Cyclic

{cell=table, display=false}

using UnicodePlots, ParameterSchedulers
t = 1:10 |> collect
s = Sin(λ0 = 0.0, λ1 = 1.0, period = 2)
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)

SinDecay2(;λ0, λ1, period)

Sine function with half the amplitude every period

Cyclic

{cell=table, display=false}

using UnicodePlots, ParameterSchedulers
t = 1:10 |> collect
s = SinDecay2(λ0 = 0.0, λ1 = 1.0, period = 2)
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)

SinExp(;λ0, λ1, period)

Sine function with exponential amplitude decay at rate γ

Cyclic

{cell=table, display=false}

using UnicodePlots, ParameterSchedulers
t = 1:10 |> collect
s = SinExp(λ0 = 0.0, λ1 = 1.0, period = 2, γ = 0.8)
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)