Generic function to calibrate binary classification models.
Arguments
- x
ClassificationorClassificationResobject to calibrate.- hyperparameters
Optional
Hyperparametersobject: Setup using one ofsetup_*functions. Defines the algorithm used to train the calibration model. NULL uses setup_Isotonic.- execution_config
Optional
ExecutionConfigobject: Setup using setup_SerialExecution, setup_FutureExecution or setup_MiraiExecution. The calibration models are trained under it. NULL trains them serially.- verbosity
Integer: Verbosity level.
- ...
Additional arguments passed to specific methods.
Details
The goal of calibration is to adjust the predicted probabilities of a binary classification model so that they better reflect the true probabilities (i.e. empirical risk) of the positive class.
Method-specific parameters
For Classification objects:
predicted_probabilities: Numeric vector of the positive class's predicted probabilities, one per casetrue_labels: Factor of true class labels
For ClassificationRes objects:
resampler_config:ResamplerConfigobject for calibration trainingtrain_verbosity: Integer controlling calibration model training output
Choosing a calibrator
A calibration map must be monotonic non-decreasing. A map that reorders
scores changes the ranking of the predictions, and so changes AUC; a
non-decreasing one cannot. available_calibration lists the algorithms
that carry that guarantee. Any other Hyperparameters object is accepted
and trained like any other model, but nothing then constrains the map.
setup_Isotonic (the default) fits isotonic regression. The map is a step function, so it merges nearby scores into ties, which moves AUC slightly in either direction. Fitted probabilities are held at least
1 / (2 * n)away from 0 and 1,nbeing the number of calibration cases, so a block of uniformly labelled cases does not assert certainty.setup_MonotonicHAL fits a monotonic Highly Adaptive Lasso on the logit scale. At its default
smoothness_orders = 1the map is continuous and strictly increasing, so it preserves AUC exactly – but the constraint that achieves this also makes the map convex on the logit scale, which costs accuracy whenever the correction needed is concave.smoothness_orders = 0withpenalized = FALSElifts that restriction, at the cost of ties and a markedly slower fit.
Isotonic is the default because no monotonic lasso configuration is
uniformly better and every one of them is several times slower.
data-raw/benchmark_calibrators.R reproduces the comparison.
The calibrator that ran is recorded on the returned object's @calibrator
property, is shown by print(), and is serialized by to_json().
Examples
# --- Calibrate Classification ---
dat <- iris[51:150, ]
res <- resample(dat)
dat[["Species"]] <- factor(dat[["Species"]])
dat_train <- dat[res[[1]], ]
dat_test <- dat[-res[[1]], ]
# Train GLM on a training/test split
mod_c_glm <- train(
x = dat_train,
dat_test = dat_test,
hyperparameters = setup_GLM()
)#> 2026-10-20 09:07:35
#> Checking data is ready for training...
#>
#> ✔
#> [check_supervised]
#> 2026-10-20 09:07:35
#> ▶
#> [train]
#> 2026-10-20 09:07:35
#> Training set: 90 cases x 4 features.
#> [summarize_supervised]
#> 2026-10-20 09:07:35
#> Test set: 10 cases x 4 features.
#> [summarize_supervised]
#> 2026-10-20 09:07:35
#> // CPU | future (mirai_multisession) | 7 workers: algorithm 1 thread, tuning 1, outer resampling 1
#> [msg_resources]
#> 2026-10-20 09:07:35
#> Shared memory (auto): not sharing training data (5.4 kB) -- this run transfers nothing to workers.
#> [report_shared_memory]
#> 2026-10-20 09:07:35
#> Training GLM Classification...
#> [train]
#> 2026-10-20 09:07:35
#> Checking data is ready for training...
#>
#> ✔
#> [check_supervised]
#>
#> <Classification>
#> GLM (GLM (generalized linear model))
#>
#> <Training Classification Metrics>
#> Predicted
#> Reference virginica versicolor
#> virginica 44 1
#> versicolor 1 44
#>
#> Overall
#> Sensitivity 0.978
#> Specificity 0.978
#> Balanced Accuracy 0.978
#> PPV 0.978
#> NPV 0.978
#> F1 0.978
#> Accuracy 0.978
#> AUC 0.997
#> Brier Score 0.020
#>
#> Positive Class virginica
#>
#> <Test Classification Metrics>
#> Predicted
#> Reference virginica versicolor
#> virginica 5 0
#> versicolor 0 5
#>
#> Overall
#> Sensitivity 1.000
#> Specificity 1.000
#> Balanced Accuracy 1.000
#> PPV 1.000
#> NPV 1.000
#> F1 1.000
#> Accuracy 1.000
#> AUC 1.000
#> Brier Score 0.022
#>
#> Positive Class virginica
#>
#> 2026-10-20 09:07:35
#> Done in 0.42 seconds.
#> [train]
# Calibrate the `Classification` by defining `predicted_probabilities` and `true_labels`,
# in this case using the training data, but it could be a separate calibration dataset.
mod_c_glm_cal <- calibrate(
mod_c_glm,
predicted_probabilities = mod_c_glm[["predicted_prob_training"]][, 1L],
true_labels = mod_c_glm[["y_training"]]
)#> 2026-10-20 09:07:35
#> <> Calibrating GLM classification...
#> [calibrate]
#> 2026-10-20 09:07:35
#> Checking data is ready for training...
#>
#> ✔
#> [check_supervised]
#> 2026-10-20 09:07:35
#> ▶
#> [train]
#> 2026-10-20 09:07:35
#> Training set: 90 cases x 1 features.
#> [summarize_supervised]
#> 2026-10-20 09:07:35
#> Test set: 10 cases x 1 features.
#> [summarize_supervised]
#> 2026-10-20 09:07:35
#> // CPU | serial | 1 worker: algorithm 1 thread, tuning 1, outer resampling 1
#> [msg_resources]
#> 2026-10-20 09:07:35
#> Shared memory (auto): not sharing training data (3 kB) -- this run transfers nothing to workers.
#> [report_shared_memory]
#> 2026-10-20 09:07:35
#> Training Isotonic Classification...
#> [train]
#> 2026-10-20 09:07:35
#> Checking data is ready for training...
#>
#> ✔
#> [check_supervised]
#>
#> <Classification>
#> Isotonic (Isotonic regression)
#>
#> <Training Classification Metrics>
#> Predicted
#> Reference virginica versicolor
#> virginica 44 1
#> versicolor 1 44
#>
#> Overall
#> Sensitivity 0.978
#> Specificity 0.978
#> Balanced Accuracy 0.978
#> PPV 0.978
#> NPV 0.978
#> F1 0.978
#> Accuracy 0.978
#> AUC 0.998
#> Brier Score 0.016
#>
#> Positive Class virginica
#>
#> <Test Classification Metrics>
#> Predicted
#> Reference virginica versicolor
#> virginica 5 0
#> versicolor 1 4
#>
#> Overall
#> Sensitivity 1.000
#> Specificity 0.800
#> Balanced Accuracy 0.900
#> PPV 0.833
#> NPV 1.000
#> F1 0.909
#> Accuracy 0.900
#> AUC 1.000
#> Brier Score 0.056
#>
#> Positive Class virginica
#>
#> 2026-10-20 09:07:35
#> Done in 0.03 seconds.
#> [train]
#>
#> <Classification>
#> GLM (GLM (generalized linear model))
#> ⟋ Calibrated using Isotonic regression.
#>
#> <Training Classification Metrics (Pre => Post Calibration)>
#> Predicted
#> Reference virginica versicolor
#> virginica 44 => 44 1 => 1
#> versicolor 1 => 1 44 => 44
#>
#> 1
#> Sensitivity 0.98 => 0.98
#> Specificity 0.98 => 0.98
#> Balanced Accuracy 0.98 => 0.98
#> PPV 0.98 => 0.98
#> NPV 0.98 => 0.98
#> F1 0.98 => 0.98
#> Accuracy 0.98 => 0.98
#> AUC 1.00 => 1.00
#> Brier Score 0.02 => 0.02
#>
#> Positive Class virginica
#>
#> <Test Classification Metrics (Pre => Post Calibration)>
#> Predicted
#> Reference virginica versicolor
#> virginica 5 => 5 0 => 0
#> versicolor 0 => 1 5 => 4
#>
#> 1
#> Sensitivity 1.00 => 1.00
#> Specificity 1.00 => 0.80
#> Balanced Accuracy 1.00 => 0.90
#> PPV 1.00 => 0.83
#> NPV 1.00 => 1.00
#> F1 1.00 => 0.91
#> Accuracy 1.00 => 0.90
#> AUC 1.00 => 1.00
#> Brier Score 0.02 => 0.06
#>
#> Positive Class virginica
#>
#> 2026-10-20 09:07:35
#> </> Calibration done.
#> [calibrate]mod_c_glm_cal#> <Classification>
#> GLM (GLM (generalized linear model))
#> ⟋ Calibrated using Isotonic regression.
#>
#> <Training Classification Metrics (Pre => Post Calibration)>
#> Predicted
#> Reference virginica versicolor
#> virginica 44 => 44 1 => 1
#> versicolor 1 => 1 44 => 44
#>
#> 1
#> Sensitivity 0.98 => 0.98
#> Specificity 0.98 => 0.98
#> Balanced Accuracy 0.98 => 0.98
#> PPV 0.98 => 0.98
#> NPV 0.98 => 0.98
#> F1 0.98 => 0.98
#> Accuracy 0.98 => 0.98
#> AUC 1.00 => 1.00
#> Brier Score 0.02 => 0.02
#>
#> Positive Class virginica
#>
#> <Test Classification Metrics (Pre => Post Calibration)>
#> Predicted
#> Reference virginica versicolor
#> virginica 5 => 5 0 => 0
#> versicolor 0 => 1 5 => 4
#>
#> 1
#> Sensitivity 1.00 => 1.00
#> Specificity 1.00 => 0.80
#> Balanced Accuracy 1.00 => 0.90
#> PPV 1.00 => 0.83
#> NPV 1.00 => 1.00
#> F1 1.00 => 0.91
#> Accuracy 1.00 => 0.90
#> AUC 1.00 => 1.00
#> Brier Score 0.02 => 0.06
#>
#> Positive Class virginica
# --- Calibrate ClassificationRes ---
# Train GLM with cross-validation
resmod_c_glm <- train(
x = dat,
hyperparameters = setup_GLM(),
outer_resampling_config = setup_KFold(n_resamples = 3L)
)#> 2026-10-20 09:07:35
#> Checking data is ready for training...
#>
#> ✔
#> [check_supervised]
#> 2026-10-20 09:07:35
#> ▶
#> [train]
#> 2026-10-20 09:07:35
#> Training set: 100 cases x 4 features.
#> [summarize_supervised]
#> 2026-10-20 09:07:35
#> // CPU | future (mirai_multisession) | 7 workers: algorithm 1 thread, tuning 1, outer resampling 7
#> [msg_resources]
#> 2026-10-20 09:07:36
#> Started 7 workers (future) for this run.
#> [worker_pool_start]
#> 2026-10-20 09:07:36
#> Shared memory (auto): training data (5.8 kB) shared across 7 workers.
#> [report_shared_memory]
#> 2026-10-20 09:07:36
#> <> Training GLM Classification using 3 independent folds...
#> [train]
#> 2026-10-20 09:07:36
#> Dispatching 3 tasks to 7 workers (future).
#> [execute]
#> 2026-10-20 09:07:36
#> Outer resamples started (total: 3)
#>
#> Warning: glm.fit: algorithm did not converge
#> Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
#> Warning: glm.fit: algorithm did not converge
#> Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
#> 2026-10-20 09:07:38
#> ✔ Outer resamples 3/3 done in 0:01
#>
#> 2026-10-20 09:07:38
#> </> Outer resampling done.
#> [train]
#>
#> <Resampled Classification Model>
#> GLM (GLM (generalized linear model))
#> ⟳ Tested using 3 independent folds.
#>
#> <Resampled Classification Training Metrics>
#> Aggregate Confusion Matrix across resamples.
#> Predicted
#> Reference virginica versicolor
#> virginica 99 1
#> versicolor 1 99
#>
#> Showing mean (sd) across resamples.
#> Sensitivity: 0.990 (0.017)
#> Specificity: 0.990 (0.017)
#> Balanced Accuracy: 0.990 (0.017)
#> PPV: 0.990 (0.017)
#> NPV: 0.990 (0.017)
#> F1: 0.990 (0.017)
#> Accuracy: 0.990 (0.017)
#> AUC: 0.999 (2e-03)
#> Brier Score: 0.007 (0.012)
#>
#> <Resampled Classification Test Metrics>
#> Aggregate Confusion Matrix across resamples.
#> Predicted
#> Reference virginica versicolor
#> virginica 47 3
#> versicolor 2 48
#>
#> Showing mean (sd) across resamples.
#> Sensitivity: 0.941 (0.102)
#> Specificity: 0.960 (0.035)
#> Balanced Accuracy: 0.950 (0.033)
#> PPV: 0.962 (0.033)
#> NPV: 0.950 (0.087)
#> F1: 0.948 (0.039)
#> Accuracy: 0.950 (0.033)
#> AUC: 0.961 (0.045)
#> Brier Score: 0.045 (0.038)
#>
#> 2026-10-20 09:07:38
#> Done in 2.07 seconds.
#> [train]
# Calibrate the `ClassificationRes` using the same resampling configuration as used for training.
resmod_c_glm_cal <- calibrate(resmod_c_glm)#> 2026-10-20 09:07:38
#> <> Calibrating GLM resampled classification...
#> [calibrate]
#>
#> <Resampled Classification Model>
#> GLM (GLM (generalized linear model))
#> ⟳ Tested using 3 independent folds.
#> ⟋ Calibrated using Isotonic regression with 5 independent folds.
#>
#> <Resampled Classification Training Metrics (Pre => Post Calibration)>
#> Showing mean (sd) across resamples, Pre => Post calibration.
#> sensitivity: 0.99 (0.02) => 0.94 (0.09)
#> specificity: 0.99 (0.02) => 0.98 (0.03)
#> balanced_accuracy: 0.99 (0.02) => 0.96 (0.04)
#> ppv: 0.99 (0.02) => 0.98 (0.03)
#> npv: 0.99 (0.02) => 0.95 (0.08)
#> f1: 0.99 (0.02) => 0.96 (0.05)
#> accuracy: 0.99 (0.02) => 0.96 (0.04)
#> auc: 1.00 (2e-03) => 0.96 (0.04)
#> brier_score: 0.01 (0.01) => 0.03 (0.03)
#>
#> <Resampled Classification Test Metrics (Pre => Post Calibration)>
#> Showing mean (sd) across resamples, Pre => Post calibration.
#> sensitivity: 0.94 (0.10) => 0.94 (0.15)
#> specificity: 0.96 (0.04) => 0.97 (0.09)
#> balanced_accuracy: 0.95 (0.03) => 0.96 (0.08)
#> ppv: 0.96 (0.03) => 0.97 (0.07)
#> npv: 0.95 (0.09) => 0.96 (0.10)
#> f1: 0.95 (0.04) => 0.95 (0.10)
#> accuracy: 0.95 (0.03) => 0.96 (0.08)
#> auc: 0.96 (0.04) => 0.96 (0.08)
#> brier_score: 0.04 (0.04) => 0.04 (0.07)
#>
#> 2026-10-20 09:07:38
#> </> Calibration done.
#> [calibrate]resmod_c_glm_cal#> <Resampled Classification Model>
#> GLM (GLM (generalized linear model))
#> ⟳ Tested using 3 independent folds.
#> ⟋ Calibrated using Isotonic regression with 5 independent folds.
#>
#> <Resampled Classification Training Metrics (Pre => Post Calibration)>
#> Showing mean (sd) across resamples, Pre => Post calibration.
#> sensitivity: 0.99 (0.02) => 0.94 (0.09)
#> specificity: 0.99 (0.02) => 0.98 (0.03)
#> balanced_accuracy: 0.99 (0.02) => 0.96 (0.04)
#> ppv: 0.99 (0.02) => 0.98 (0.03)
#> npv: 0.99 (0.02) => 0.95 (0.08)
#> f1: 0.99 (0.02) => 0.96 (0.05)
#> accuracy: 0.99 (0.02) => 0.96 (0.04)
#> auc: 1.00 (2e-03) => 0.96 (0.04)
#> brier_score: 0.01 (0.01) => 0.03 (0.03)
#>
#> <Resampled Classification Test Metrics (Pre => Post Calibration)>
#> Showing mean (sd) across resamples, Pre => Post calibration.
#> sensitivity: 0.94 (0.10) => 0.94 (0.15)
#> specificity: 0.96 (0.04) => 0.97 (0.09)
#> balanced_accuracy: 0.95 (0.03) => 0.96 (0.08)
#> ppv: 0.96 (0.03) => 0.97 (0.07)
#> npv: 0.95 (0.09) => 0.96 (0.10)
#> f1: 0.95 (0.04) => 0.95 (0.10)
#> accuracy: 0.95 (0.03) => 0.96 (0.08)
#> auc: 0.96 (0.04) => 0.96 (0.08)
#> brier_score: 0.04 (0.04) => 0.04 (0.07)