Generic function to calibrate binary classification models.
Arguments
- x
ClassificationorClassificationResobject to calibrate.- hyperparameters
Optional
Hyperparametersobject: Setup using one ofsetup_*functions. Defines the algorithm used to train the calibration model. NULL uses setup_Isotonic.- verbosity
Integer: Verbosity level.
- ...
Additional arguments passed to specific methods.
Details
The goal of calibration is to adjust the predicted probabilities of a binary classification model so that they better reflect the true probabilities (i.e. empirical risk) of the positive class.
Method-specific parameters
For Classification objects:
predicted_probabilities: Numeric vector of the positive class's predicted probabilities, one per casetrue_labels: Factor of true class labels
For ClassificationRes objects:
resampler_config:ResamplerConfigobject for calibration trainingtrain_verbosity: Integer controlling calibration model training output
Choosing a calibrator
A calibration map must be monotonic non-decreasing. A map that reorders
scores changes the ranking of the predictions, and so changes AUC; a
non-decreasing one cannot. available_calibration lists the algorithms
that carry that guarantee. Any other Hyperparameters object is accepted
and trained like any other model, but nothing then constrains the map.
setup_Isotonic (the default) fits isotonic regression. The map is a step function, so it merges nearby scores into ties, which moves AUC slightly in either direction. Fitted probabilities are held at least
1 / (2 * n)away from 0 and 1,nbeing the number of calibration cases, so a block of uniformly labelled cases does not assert certainty.setup_MonotonicHAL fits a monotonic Highly Adaptive Lasso on the logit scale. At its default
smoothness_orders = 1the map is continuous and strictly increasing, so it preserves AUC exactly – but the constraint that achieves this also makes the map convex on the logit scale, which costs accuracy whenever the correction needed is concave.smoothness_orders = 0withpenalized = FALSElifts that restriction, at the cost of ties and a markedly slower fit.
Isotonic is the default because no monotonic lasso configuration is
uniformly better and every one of them is several times slower.
data-raw/benchmark_calibrators.R reproduces the comparison.
The calibrator that ran is recorded on the returned object's @calibrator
property, is shown by print(), and is serialized by to_json().
Examples
# --- Calibrate Classification ---
dat <- iris[51:150, ]
res <- resample(dat)
#> 2026-08-09 13:22:20
#> Using max n bins possible = 2.
#> [kfold]
dat$Species <- factor(dat$Species)
dat_train <- dat[res[[1]], ]
dat_test <- dat[-res[[1]], ]
# Train GLM on a training/test split
mod_c_glm <- train(
x = dat_train,
dat_test = dat_test,
hyperparameters = setup_GLM()
)
#> 2026-08-09 13:22:20
#> Checking data is ready for training...
#>
#> ✔
#> [check_supervised]
#> 2026-08-09 13:22:20
#> ▶
#> [train]
#> 2026-08-09 13:22:20
#> Training set: 90 cases x 4 features.
#> [summarize_supervised]
#> 2026-08-09 13:22:20
#> Test set: 10 cases x 4 features.
#> [summarize_supervised]
#> 2026-08-09 13:22:20
#> // Max workers: c(system = 7) { Algorithm: 1; Tuning: 1; Outer Resampling: 1 }
#> [get_n_workers]
#> 2026-08-09 13:22:20
#> Training GLM Classification...
#> [train]
#> 2026-08-09 13:22:20
#> Checking data is ready for training...
#>
#> ✔
#> [check_supervised]
#>
#> <Classification>
#> GLM (Generalized Linear Model)
#>
#> <Training Classification Metrics>
#> Predicted
#> Reference virginica versicolor
#> virginica 44 1
#> versicolor 1 44
#>
#> Overall
#> Sensitivity 0.978
#> Specificity 0.978
#> Balanced Accuracy 0.978
#> Ppv 0.978
#> Npv 0.978
#> F1 0.978
#> Accuracy 0.978
#> Auc 0.997
#> Brier Score 0.021
#>
#> Positive Class virginica
#>
#> <Test Classification Metrics>
#> Predicted
#> Reference virginica versicolor
#> virginica 5 0
#> versicolor 0 5
#>
#> Overall
#> Sensitivity 1.000
#> Specificity 1.000
#> Balanced Accuracy 1.000
#> Ppv 1.000
#> Npv 1.000
#> F1 1.000
#> Accuracy 1.000
#> Auc 1.000
#> Brier Score 2e-07
#>
#> Positive Class virginica
#>
#> 2026-08-09 13:22:20
#> Done in 0.15 seconds.
#> [train]
# Calibrate the `Classification` by defining `predicted_probabilities` and `true_labels`,
# in this case using the training data, but it could be a separate calibration dataset.
mod_c_glm_cal <- calibrate(
mod_c_glm,
predicted_probabilities = mod_c_glm$predicted_prob_training[, 1L],
true_labels = mod_c_glm$y_training
)
#> 2026-08-09 13:22:20
#> <> Calibrating GLM classification...
#> [calibrate]
#> 2026-08-09 13:22:20
#> Checking data is ready for training...
#>
#> ✔
#> [check_supervised]
#> 2026-08-09 13:22:20
#> ▶
#> [train]
#> 2026-08-09 13:22:20
#> Training set: 90 cases x 1 features.
#> [summarize_supervised]
#> 2026-08-09 13:22:20
#> Test set: 10 cases x 1 features.
#> [summarize_supervised]
#> 2026-08-09 13:22:20
#> // Max workers: c(system = 7) { Algorithm: 1; Tuning: 1; Outer Resampling: 1 }
#> [get_n_workers]
#> 2026-08-09 13:22:20
#> Training Isotonic Classification...
#> [train]
#> 2026-08-09 13:22:20
#> Checking data is ready for training...
#>
#> ✔
#> [check_supervised]
#>
#> <Classification>
#> Isotonic (Isotonic Regression)
#>
#> <Training Classification Metrics>
#> Predicted
#> Reference virginica versicolor
#> virginica 44 1
#> versicolor 1 44
#>
#> Overall
#> Sensitivity 0.978
#> Specificity 0.978
#> Balanced Accuracy 0.978
#> Ppv 0.978
#> Npv 0.978
#> F1 0.978
#> Accuracy 0.978
#> Auc 0.998
#> Brier Score 0.017
#>
#> Positive Class virginica
#>
#> <Test Classification Metrics>
#> Predicted
#> Reference virginica versicolor
#> virginica 5 0
#> versicolor 0 5
#>
#> Overall
#> Sensitivity 1.000
#> Specificity 1.000
#> Balanced Accuracy 1.000
#> Ppv 1.000
#> Npv 1.000
#> F1 1.000
#> Accuracy 1.000
#> Auc 1.000
#> Brier Score 3.1e-05
#>
#> Positive Class virginica
#>
#> 2026-08-09 13:22:20
#> Done in 0.02 seconds.
#> [train]
#>
#> <Classification>
#> GLM (Generalized Linear Model)
#> ⟋ Calibrated using Isotonic Regression.
#>
#> <Training Classification Metrics (Pre => Post Calibration)>
#> Predicted
#> Reference virginica versicolor
#> virginica 44 => 44 1 => 1
#> versicolor 1 => 1 44 => 44
#>
#> 1
#> Sensitivity 0.98 => 0.98
#> Specificity 0.98 => 0.98
#> Balanced Accuracy 0.98 => 0.98
#> Ppv 0.98 => 0.98
#> Npv 0.98 => 0.98
#> F1 0.98 => 0.98
#> Accuracy 0.98 => 0.98
#> Auc 1.00 => 1.00
#> Brier Score 0.02 => 0.02
#>
#> Positive Class virginica
#>
#> <Test Classification Metrics (Pre => Post Calibration)>
#> Predicted
#> Reference virginica versicolor
#> virginica 5 => 5 0 => 0
#> versicolor 0 => 0 5 => 5
#>
#> 1
#> Sensitivity 1.00 => 1.00
#> Specificity 1.00 => 1.00
#> Balanced Accuracy 1.00 => 1.00
#> Ppv 1.00 => 1.00
#> Npv 1.00 => 1.00
#> F1 1.00 => 1.00
#> Accuracy 1.00 => 1.00
#> Auc 1.00 => 1.00
#> Brier Score 2e-07 => 3.1e-05
#>
#> Positive Class virginica
#>
#> 2026-08-09 13:22:20
#> </> Calibration done.
#> [calibrate]
mod_c_glm_cal
#> <Classification>
#> GLM (Generalized Linear Model)
#> ⟋ Calibrated using Isotonic Regression.
#>
#> <Training Classification Metrics (Pre => Post Calibration)>
#> Predicted
#> Reference virginica versicolor
#> virginica 44 => 44 1 => 1
#> versicolor 1 => 1 44 => 44
#>
#> 1
#> Sensitivity 0.98 => 0.98
#> Specificity 0.98 => 0.98
#> Balanced Accuracy 0.98 => 0.98
#> Ppv 0.98 => 0.98
#> Npv 0.98 => 0.98
#> F1 0.98 => 0.98
#> Accuracy 0.98 => 0.98
#> Auc 1.00 => 1.00
#> Brier Score 0.02 => 0.02
#>
#> Positive Class virginica
#>
#> <Test Classification Metrics (Pre => Post Calibration)>
#> Predicted
#> Reference virginica versicolor
#> virginica 5 => 5 0 => 0
#> versicolor 0 => 0 5 => 5
#>
#> 1
#> Sensitivity 1.00 => 1.00
#> Specificity 1.00 => 1.00
#> Balanced Accuracy 1.00 => 1.00
#> Ppv 1.00 => 1.00
#> Npv 1.00 => 1.00
#> F1 1.00 => 1.00
#> Accuracy 1.00 => 1.00
#> Auc 1.00 => 1.00
#> Brier Score 2e-07 => 3.1e-05
#>
#> Positive Class virginica
# --- Calibrate ClassificationRes ---
# Train GLM with cross-validation
resmod_c_glm <- train(
x = dat,
hyperparameters = setup_GLM(),
outer_resampling_config = setup_Resampler(n_resamples = 3L, type = "KFold")
)
#> 2026-08-09 13:22:20
#> Checking data is ready for training...
#>
#> ✔
#> [check_supervised]
#> 2026-08-09 13:22:20
#> ▶
#> [train]
#> 2026-08-09 13:22:20
#> Training set: 100 cases x 4 features.
#> [summarize_supervised]
#> 2026-08-09 13:22:20
#> // Max workers: c(system = 7) { Algorithm: 1; Tuning: 1; Outer Resampling: c(system = 7) }
#> [get_n_workers]
#> 2026-08-09 13:22:20
#> <> Training GLM Classification using 3 independent folds...
#> [train]
#> 2026-08-09 13:22:20
#> Using max n bins possible = 2.
#> [kfold]
#> 2026-08-09 13:22:20
#> Outer resamples started (total: 3)
#>
#> Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
#> Warning: glm.fit: algorithm did not converge
#> Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
#> 2026-08-09 13:22:20
#> ✔ Outer resamples 3/3 done in 0:00
#>
#> 2026-08-09 13:22:20
#> </> Outer resampling done.
#> [train]
#>
#> <Resampled Classification Model>
#> GLM (Generalized Linear Model)
#> ⟳ Tested using 3 independent folds.
#>
#> <Resampled Classification Training Metrics>
#> Aggregate Confusion Matrix across resamples.
#> Predicted
#> Reference virginica versicolor
#> virginica 98 2
#> versicolor 2 98
#>
#> Showing mean (sd) across resamples.
#> Sensitivity: 0.980 (0.017)
#> Specificity: 0.980 (0.017)
#> Balanced Accuracy: 0.980 (0.017)
#> Ppv: 0.980 (0.017)
#> Npv: 0.980 (0.017)
#> F1: 0.980 (0.017)
#> Accuracy: 0.980 (0.017)
#> Auc: 0.998 (1.9e-03)
#> Brier Score: 0.014 (0.013)
#>
#> <Resampled Classification Test Metrics>
#> Aggregate Confusion Matrix across resamples.
#> Predicted
#> Reference virginica versicolor
#> virginica 47 3
#> versicolor 2 48
#>
#> Showing mean (sd) across resamples.
#> Sensitivity: 0.940 (0.059)
#> Specificity: 0.960 (0.035)
#> Balanced Accuracy: 0.950 (0.045)
#> Ppv: 0.958 (0.036)
#> Npv: 0.942 (0.056)
#> F1: 0.949 (0.047)
#> Accuracy: 0.950 (0.045)
#> Auc: 0.973 (0.024)
#> Brier Score: 0.052 (0.042)
#>
#> 2026-08-09 13:22:20
#> Done in 0.07 seconds.
#> [train]
# Calibrate the `ClassificationRes` using the same resampling configuration as used for training.
resmod_c_glm_cal <- calibrate(resmod_c_glm)
#> 2026-08-09 13:22:20
#> <> Calibrating GLM resampled classification...
#> [calibrate]
#>
#> <Resampled Classification Model>
#> GLM (Generalized Linear Model)
#> ⟳ Tested using 3 independent folds.
#> ⟋ Calibrated using Isotonic Regression with 5 independent folds.
#>
#> <Resampled Classification Training Metrics (Pre => Post Calibration)>
#> Showing mean (sd) across resamples, Pre => Post calibration.
#> sensitivity: 0.98 (0.02) => 0.96 (0.04)
#> specificity: 0.98 (0.02) => 0.96 (0.04)
#> balanced_accuracy: 0.98 (0.02) => 0.96 (0.03)
#> ppv: 0.98 (0.02) => 0.97 (0.04)
#> npv: 0.98 (0.02) => 0.96 (0.04)
#> f1: 0.98 (0.02) => 0.96 (0.03)
#> accuracy: 0.98 (0.02) => 0.96 (0.03)
#> auc: 1.00 (1.9e-03) => 0.98 (0.02)
#> brier_score: 0.01 (0.01) => 0.03 (0.02)
#>
#> <Resampled Classification Test Metrics (Pre => Post Calibration)>
#> Showing mean (sd) across resamples, Pre => Post calibration.
#> sensitivity: 0.94 (0.06) => 0.93 (0.14)
#> specificity: 0.96 (0.04) => 0.90 (0.15)
#> balanced_accuracy: 0.95 (0.05) => 0.92 (0.08)
#> ppv: 0.96 (0.04) => 0.92 (0.11)
#> npv: 0.94 (0.06) => 0.95 (0.10)
#> f1: 0.95 (0.05) => 0.92 (0.09)
#> accuracy: 0.95 (0.05) => 0.92 (0.08)
#> auc: 0.97 (0.02) => 0.94 (0.08)
#> brier_score: 0.05 (0.04) => 0.07 (0.08)
#>
#> 2026-08-09 13:22:21
#> </> Calibration done.
#> [calibrate]
resmod_c_glm_cal
#> <Resampled Classification Model>
#> GLM (Generalized Linear Model)
#> ⟳ Tested using 3 independent folds.
#> ⟋ Calibrated using Isotonic Regression with 5 independent folds.
#>
#> <Resampled Classification Training Metrics (Pre => Post Calibration)>
#> Showing mean (sd) across resamples, Pre => Post calibration.
#> sensitivity: 0.98 (0.02) => 0.96 (0.04)
#> specificity: 0.98 (0.02) => 0.96 (0.04)
#> balanced_accuracy: 0.98 (0.02) => 0.96 (0.03)
#> ppv: 0.98 (0.02) => 0.97 (0.04)
#> npv: 0.98 (0.02) => 0.96 (0.04)
#> f1: 0.98 (0.02) => 0.96 (0.03)
#> accuracy: 0.98 (0.02) => 0.96 (0.03)
#> auc: 1.00 (1.9e-03) => 0.98 (0.02)
#> brier_score: 0.01 (0.01) => 0.03 (0.02)
#>
#> <Resampled Classification Test Metrics (Pre => Post Calibration)>
#> Showing mean (sd) across resamples, Pre => Post calibration.
#> sensitivity: 0.94 (0.06) => 0.93 (0.14)
#> specificity: 0.96 (0.04) => 0.90 (0.15)
#> balanced_accuracy: 0.95 (0.05) => 0.92 (0.08)
#> ppv: 0.96 (0.04) => 0.92 (0.11)
#> npv: 0.94 (0.06) => 0.95 (0.10)
#> f1: 0.95 (0.05) => 0.92 (0.09)
#> accuracy: 0.95 (0.05) => 0.92 (0.08)
#> auc: 0.97 (0.02) => 0.94 (0.08)
#> brier_score: 0.05 (0.04) => 0.07 (0.08)