Estimation of calibration errors.
This package implements different estimators of the expected calibration error (ECE), the squared kernel calibration error (SKCE), and the unnormalized calibration mean embedding (UCME) in the Julia language.
Calibration errors can be estimated from a data set of predicted probability distributions and a set of corresponding observed targets by executing
The sets of predictions and targets have to be provided as vectors.
This package implements the estimator
ECE of the ECE, the estimators
BlockUnbiasedSKCE for the SKCE, and
UCME for the
CalibrationErrorsDistributions.jl extends calibration error estimation in this package to more general probabilistic predictive models that output arbitrary probability distributions.
CalibrationTests.jl implements statistical hypothesis tests of calibration.
pycalibration is a Python interface for CalibrationErrors.jl, CalibrationErrorsDistributions.jl, and CalibrationTests.jl.
rcalibration is an R interface for CalibrationErrors.jl, CalibrationErrorsDistributions.jl, and CalibrationTests.jl.
If you use CalibrationsErrors.jl as part of your research, teaching, or other activities, please consider citing the following publications:
Widmann, D., Lindsten, F., & Zachariah, D. (2019). Calibration tests in multi-class classification: A unifying framework. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019) (pp. 12257–12267).
Widmann, D., Lindsten, F., & Zachariah, D. (2021). Calibration tests beyond classification. To be presented at ICLR 2021.