Visualization of model calibration
- CalibrationErrors.jl: Estimation of calibration errors such as the expected calibration error (ECE), the squared kernel calibration error (SKCE), and the unnormalized calibration mean embedding (UCME).
- CalibrationTests.jl: Statistical hypothesis tests of calibration.
Bröcker, J., & Smith, L. A. (2007). Increasing the reliability of reliability diagrams. Weather and forecasting, 22(3), 651-661.
Murphy, A., & Winkler, R. (1977). Reliability of Subjective Probability Forecasts of Precipitation and Temperature. Journal of the Royal Statistical Society. Series C (Applied Statistics), 26(1), 41-47.
Vaicenavicius, J., Widmann, D., Andersson, C., Lindsten, F., Roll, J. & Schön, T. B. (2019). Evaluating model calibration in classification. Proceedings of Machine Learning Research, in Proceedings of Machine Learning Research 89:3459-3467 (AISTATS 2019).