ParameterSchedulers.jl

Common hyperparameter scheduling for ML
Author darsnack
Popularity
13 Stars
Updated Last
1 Year Ago
Started In
December 2020

ParameterSchedulers

Dev Build Status

ParameterSchedulers.jl provides common machine learning (ML) schedulers for hyper-parameters. Though this package is framework agnostic, a convenient interface for pairing schedules with Flux.jl optimizers is available. Using this package with Flux is as simple as:

using Flux, ParameterSchedulers
using ParameterSchedulers: Scheduler

opt = Scheduler(Exp= 1e-2, γ = 0.8), Momentum())

Available Schedules

This is a table of the common schedules implemented, but ParameterSchedulers provides utilities for creating more exotic schedules as well. You can read this paper for more information on the schedules below.

{cell=table, display=false, output=false, results=false}

using UnicodePlots, ParameterSchedulers
Schedule Description Type Example

Step(;λ, γ, step_sizes)

Exponential decay by γ every step in step_sizes

Decay

{cell=table, display=false}

using UnicodePlots, ParameterSchedulers
t = 1:10 |> collect
s = Step= 1.0, γ = 0.8, step_sizes = [2, 3, 2])
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)

Exp(;λ, γ)

Exponential decay by γ every iteration

Decay

{cell=table, display=false}

using UnicodePlots, ParameterSchedulers
t = 1:10 |> collect
s = Exp= 1.0, γ = 0.5)
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)

Poly(;λ, p, max_iter)

Polynomial decay at degree p

Decay

{cell=table, display=false}

using UnicodePlots, ParameterSchedulers
t = 1:10 |> collect
s = Poly= 1.0, p = 2, max_iter = t[end])
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)

Inv(;λ, γ, p)

Inverse decay at rate (1 + tγ)^p

Decay

{cell=table, display=false}

using UnicodePlots, ParameterSchedulers
t = 1:10 |> collect
s = Inv= 1.0, p = 2, γ = 0.8)
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)

Triangle(;λ0, λ1, period)

Triangle wave function

Cyclic

{cell=table, display=false}

using UnicodePlots, ParameterSchedulers
t = 1:10 |> collect
s = Triangle(λ0 = 0.0, λ1 = 1.0, period = 2)
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)

TriangleDecay2(;λ0, λ1, period)

Triangle wave function with half the amplitude every period

Cyclic

{cell=table, display=false}

using UnicodePlots, ParameterSchedulers
t = 1:10 |> collect
s = TriangleDecay2(λ0 = 0.0, λ1 = 1.0, period = 2)
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)

TriangleExp(;λ0, λ1, period, γ)

Triangle wave function with exponential amplitude decay at rate γ

Cyclic

{cell=table, display=false}

using UnicodePlots, ParameterSchedulers
t = 1:10 |> collect
s = TriangleExp(λ0 = 0.0, λ1 = 1.0, period = 2, γ = 0.8)
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)

Sin(;λ0, λ1, period)

Sine function

Cyclic

{cell=table, display=false}

using UnicodePlots, ParameterSchedulers
t = 1:10 |> collect
s = Sin(λ0 = 0.0, λ1 = 1.0, period = 2)
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)

SinDecay2(;λ0, λ1, period)

Sine function with half the amplitude every period

Cyclic

{cell=table, display=false}

using UnicodePlots, ParameterSchedulers
t = 1:10 |> collect
s = SinDecay2(λ0 = 0.0, λ1 = 1.0, period = 2)
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)

SinExp(;λ0, λ1, period)

Sine function with exponential amplitude decay at rate γ

Cyclic

{cell=table, display=false}

using UnicodePlots, ParameterSchedulers
t = 1:10 |> collect
s = SinExp(λ0 = 0.0, λ1 = 1.0, period = 2, γ = 0.8)
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)

Cos(;λ0, λ1, period)

Cosine annealing

Cyclic

{cell=table, display=false}

using UnicodePlots, ParameterSchedulers
t = 1:10 |> collect
s = Cos(λ0 = 0.0, λ1 = 1.0, period = 4)
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)

Used By Packages

No packages found.