ParameterSchedulers

ParameterSchedulers.jl provides common machine learning (ML) schedulers for hyper-parameters. Though this package is framework agnostic, a convenient interface for pairing schedules with Flux.jl optimizers is available. Using this package with Flux is as simple as:

using Flux, ParameterSchedulers
using ParameterSchedulers: Scheduler

opt = Scheduler(Exp(λ = 1e-2, γ = 0.8), Momentum())

Available Schedules

This is a table of the common schedules implemented, but ParameterSchedulers provides utilities for creating more exotic schedules as well. You can read this paper for more information on the schedules below.

{cell=table, display=false, output=false, results=false}

using UnicodePlots, ParameterSchedulers

Schedule	Description	Type	Example
`Step(;λ, γ, step_sizes)`	Exponential decay by `γ` every step in `step_sizes`	Decay	{cell=table, display=false} using UnicodePlots, ParameterSchedulers t = 1:10 \|> collect s = Step(λ = 1.0, γ = 0.8, step_sizes = [2, 3, 2]) lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)
`Exp(;λ, γ)`	Exponential decay by `γ` every iteration	Decay	{cell=table, display=false} using UnicodePlots, ParameterSchedulers t = 1:10 \|> collect s = Exp(λ = 1.0, γ = 0.5) lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)
`Poly(;λ, p, max_iter)`	Polynomial decay at degree `p`	Decay	{cell=table, display=false} using UnicodePlots, ParameterSchedulers t = 1:10 \|> collect s = Poly(λ = 1.0, p = 2, max_iter = t[end]) lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)
`Inv(;λ, γ, p)`	Inverse decay at rate `(1 + tγ)^p`	Decay	{cell=table, display=false} using UnicodePlots, ParameterSchedulers t = 1:10 \|> collect s = Inv(λ = 1.0, p = 2, γ = 0.8) lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)
`Triangle(;λ0, λ1, period)`	Triangle wave function	Cyclic	{cell=table, display=false} using UnicodePlots, ParameterSchedulers t = 1:10 \|> collect s = Triangle(λ0 = 0.0, λ1 = 1.0, period = 2) lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)
`TriangleDecay2(;λ0, λ1, period)`	Triangle wave function with half the amplitude every `period`	Cyclic	{cell=table, display=false} using UnicodePlots, ParameterSchedulers t = 1:10 \|> collect s = TriangleDecay2(λ0 = 0.0, λ1 = 1.0, period = 2) lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)
`TriangleExp(;λ0, λ1, period, γ)`	Triangle wave function with exponential amplitude decay at rate `γ`	Cyclic	{cell=table, display=false} using UnicodePlots, ParameterSchedulers t = 1:10 \|> collect s = TriangleExp(λ0 = 0.0, λ1 = 1.0, period = 2, γ = 0.8) lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)
`Sin(;λ0, λ1, period)`	Sine function	Cyclic	{cell=table, display=false} using UnicodePlots, ParameterSchedulers t = 1:10 \|> collect s = Sin(λ0 = 0.0, λ1 = 1.0, period = 2) lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)
`SinDecay2(;λ0, λ1, period)`	Sine function with half the amplitude every `period`	Cyclic	{cell=table, display=false} using UnicodePlots, ParameterSchedulers t = 1:10 \|> collect s = SinDecay2(λ0 = 0.0, λ1 = 1.0, period = 2) lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)
`SinExp(;λ0, λ1, period)`	Sine function with exponential amplitude decay at rate `γ`	Cyclic	{cell=table, display=false} using UnicodePlots, ParameterSchedulers t = 1:10 \|> collect s = SinExp(λ0 = 0.0, λ1 = 1.0, period = 2, γ = 0.8) lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)
`Cos(;λ0, λ1, period)`	Cosine annealing	Cyclic	{cell=table, display=false} using UnicodePlots, ParameterSchedulers t = 1:10 \|> collect s = Cos(λ0 = 0.0, λ1 = 1.0, period = 4) lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)