ReliabilityDiagrams
Visualization of model calibration
Related packages
- CalibrationErrors.jl: Estimation of calibration errors such as the expected calibration error (ECE), the squared kernel calibration error (SKCE), and the unnormalized calibration mean embedding (UCME).
- CalibrationErrorsDistributions.jl: Estimation of calibration errors for probabilistic predictive models that output probability distributions from Distributions.jl.
- CalibrationTests.jl: Statistical hypothesis tests of calibration.
References
Murphy, A., & Winkler, R. (1977). Reliability of Subjective Probability Forecasts of Precipitation and Temperature. Journal of the Royal Statistical Society. Series C (Applied Statistics), 26(1), 41-47.
Vaicenavicius, J., Widmann, D., Andersson, C., Lindsten, F., Roll, J. & Schön, T. B. (2019). Evaluating model calibration in classification. Proceedings of Machine Learning Research, in Proceedings of Machine Learning Research 89:3459-3467 (AISTATS 2019).