Skip to contents

Calibrate Classification & ClassificationRes Models

Source: R/00_S7init.R
calibrate.Rd

Generic function to calibrate binary classification models.

Usage

calibrate(
  x,
  algorithm = "isotonic",
  hyperparameters = NULL,
  verbosity = 1L,
  ...
)

Arguments

x

Classification or ClassificationRes object to calibrate.

algorithm

Character: Algorithm to use to train calibration model.

hyperparameters

Hyperparameters object: Setup using one of setup_* functions.

verbosity

Integer: Verbosity level.

...

Additional arguments passed to specific methods.

Value

Calibrated model object.

Details

The goal of calibration is to adjust the predicted probabilities of a binary classification model so that they better reflect the true probabilities (i.e. empirical risk) of the positive class.

Method-specific parameters

For Classification objects:

  • predicted_probabilities: Numeric vector of predicted probabilities

  • true_labels: Factor of true class labels

For ClassificationRes objects:

  • resampler_config: ResamplerConfig object for calibration training

  • train_verbosity: Integer controlling calibration model training output

Author

EDG

Examples

# --- Calibrate Classification ---
dat <- iris[51:150, ]
res <- resample(dat)
#> 2026-03-15 07:49:40 
#> Input contains more than one column; stratifying on last.
#>  [resample]
#> 2026-03-15 07:49:40 
#> Using max n bins possible = 2.
#>  [kfold]
dat$Species <- factor(dat$Species)
dat_train <- dat[res[[1]], ]
dat_test <- dat[-res[[1]], ]

# Train GLM on a training/test split
mod_c_glm <- train(
  x = dat_train,
  dat_test = dat_test,
  algorithm = "glm"
)
#> 2026-03-15 07:49:40 
#> ▶
#>  [train]
#> 2026-03-15 07:49:40 
#> Training set: 90 cases x 4 features.
#>  [summarize_supervised]
#> 2026-03-15 07:49:40 
#>     Test set: 10 cases x 4 features.
#>  [summarize_supervised]
#> 2026-03-15 07:49:40 
#> // Max workers: 7 => Algorithm: 1; Tuning: 1; Outer Resampling: 1
#>  [get_n_workers]
#> 2026-03-15 07:49:40 
#> Training GLM Classification...
#>  [train]
#> 2026-03-15 07:49:40 
#> Checking data is ready for training...
#>  
#> ✓ 
#> [check_supervised]
#> 
#> Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
#> 
#> <Classification>
#> GLM (Generalized Linear Model)
#> 
#>   <Training Classification Metrics>
#>                      Predicted
#>           Reference  virginica  versicolor  
#>           virginica         44           1
#>          versicolor          1          44
#> 
#>                      Overall  
#>         Sensitivity  0.978  
#>         Specificity  0.978  
#>   Balanced_Accuracy  0.978  
#>                 PPV  0.978  
#>                 NPV  0.978  
#>                  F1  0.978  
#>            Accuracy  0.978  
#>                 AUC  0.997  
#>         Brier_Score  0.021  
#> 
#>      Positive Class virginica
#> 
#>   <Test Classification Metrics>
#>                      Predicted
#>           Reference  virginica  versicolor  
#>           virginica          5           0
#>          versicolor          0           5
#> 
#>                      Overall  
#>         Sensitivity  1.000  
#>         Specificity  1.000  
#>   Balanced_Accuracy  1.000  
#>                 PPV  1.000  
#>                 NPV  1.000  
#>                  F1  1.000  
#>            Accuracy  1.000  
#>                 AUC  1.000  
#>         Brier_Score  0.016  
#> 
#>      Positive Class virginica
#> 
#> 2026-03-15 07:49:40 
#> ✓ Done in 0.22 seconds.
#>  [train]

# Calibrate the `Classification` by defining `predicted_probabilities` and `true_labels`,
# in this case using the training data, but it could be a separate calibration dataset.
mod_c_glm_cal <- calibrate(
  mod_c_glm,
  predicted_probabilities = mod_c_glm$predicted_prob_training,
  true_labels = mod_c_glm$y_training
)
#> 2026-03-15 07:49:40 
#> <> Calibrating GLM classification...
#>  [calibrate]
#> 2026-03-15 07:49:40 
#> ▶
#>  [train]
#> 2026-03-15 07:49:40 
#> Training set: 90 cases x 1 features.
#>  [summarize_supervised]
#> 2026-03-15 07:49:40 
#>     Test set: 10 cases x 1 features.
#>  [summarize_supervised]
#> 2026-03-15 07:49:40 
#> // Max workers: 7 => Algorithm: 1; Tuning: 1; Outer Resampling: 1
#>  [get_n_workers]
#> 2026-03-15 07:49:40 
#> Training Isotonic Classification...
#>  [train]
#> 2026-03-15 07:49:40 
#> Checking data is ready for training...
#>  
#> ✓ 
#> [check_supervised]
#> 
#> 
#> <Classification>
#> Isotonic (Isotonic Regression)
#> 
#>   <Training Classification Metrics>
#>                      Predicted
#>           Reference  virginica  versicolor  
#>           virginica         44           1
#>          versicolor          1          44
#> 
#>                      Overall  
#>         Sensitivity  0.978  
#>         Specificity  0.978  
#>   Balanced_Accuracy  0.978  
#>                 PPV  0.978  
#>                 NPV  0.978  
#>                  F1  0.978  
#>            Accuracy  0.978  
#>                 AUC  0.998  
#>         Brier_Score  0.016  
#> 
#>      Positive Class virginica
#> 
#>   <Test Classification Metrics>
#>                      Predicted
#>           Reference  virginica  versicolor  
#>           virginica          5           0
#>          versicolor          0           5
#> 
#>                      Overall  
#>         Sensitivity  1.000  
#>         Specificity  1.000  
#>   Balanced_Accuracy  1.000  
#>                 PPV  1.000  
#>                 NPV  1.000  
#>                  F1  1.000  
#>            Accuracy  1.000  
#>                 AUC  1.000  
#>         Brier_Score  0.006  
#> 
#>      Positive Class virginica
#> 
#> 2026-03-15 07:49:40 
#> ✓ Done in 0.04 seconds.
#>  [train]
#> 
#> <Classification>
#> GLM (Generalized Linear Model)
#> ⟋ Calibrated using Isotonic Regression.
#> 
#>   <Training Classification Metrics (Pre => Post Calibration)>
#>                      Predicted
#>           Reference  virginica  versicolor  
#>           virginica   44 => 44      1 => 1
#>          versicolor     1 => 1    44 => 44
#> 
#>                      Overall       
#>         Sensitivity  0.98 => 0.98
#>         Specificity  0.98 => 0.98
#>   Balanced_Accuracy  0.98 => 0.98
#>                 PPV  0.98 => 0.98
#>                 NPV  0.98 => 0.98
#>                  F1  0.98 => 0.98
#>            Accuracy  0.98 => 0.98
#>                 AUC  1.00 => 1.00
#>         Brier_Score  0.02 => 0.02
#> 
#>      Positive Class virginica
#> 
#>   <Test Classification Metrics (Pre => Post Calibration)>
#>                      Predicted
#>           Reference  virginica  versicolor  
#>           virginica     5 => 5      0 => 0
#>          versicolor     0 => 0      5 => 5
#> 
#>                      Overall       
#>         Sensitivity  1.00 => 1.00
#>         Specificity  1.00 => 1.00
#>   Balanced_Accuracy  1.00 => 1.00
#>                 PPV  1.00 => 1.00
#>                 NPV  1.00 => 1.00
#>                  F1  1.00 => 1.00
#>            Accuracy  1.00 => 1.00
#>                 AUC  1.00 => 1.00
#>         Brier_Score  0.02 => 0.01
#> 
#>      Positive Class virginica
#> 
#> 2026-03-15 07:49:40 
#> </> Calibration done.
#>  [calibrate]
mod_c_glm_cal
#> <Classification>
#> GLM (Generalized Linear Model)
#> ⟋ Calibrated using Isotonic Regression.
#> 
#>   <Training Classification Metrics (Pre => Post Calibration)>
#>                      Predicted
#>           Reference  virginica  versicolor  
#>           virginica   44 => 44      1 => 1
#>          versicolor     1 => 1    44 => 44
#> 
#>                      Overall       
#>         Sensitivity  0.98 => 0.98
#>         Specificity  0.98 => 0.98
#>   Balanced_Accuracy  0.98 => 0.98
#>                 PPV  0.98 => 0.98
#>                 NPV  0.98 => 0.98
#>                  F1  0.98 => 0.98
#>            Accuracy  0.98 => 0.98
#>                 AUC  1.00 => 1.00
#>         Brier_Score  0.02 => 0.02
#> 
#>      Positive Class virginica
#> 
#>   <Test Classification Metrics (Pre => Post Calibration)>
#>                      Predicted
#>           Reference  virginica  versicolor  
#>           virginica     5 => 5      0 => 0
#>          versicolor     0 => 0      5 => 5
#> 
#>                      Overall       
#>         Sensitivity  1.00 => 1.00
#>         Specificity  1.00 => 1.00
#>   Balanced_Accuracy  1.00 => 1.00
#>                 PPV  1.00 => 1.00
#>                 NPV  1.00 => 1.00
#>                  F1  1.00 => 1.00
#>            Accuracy  1.00 => 1.00
#>                 AUC  1.00 => 1.00
#>         Brier_Score  0.02 => 0.01
#> 
#>      Positive Class virginica

# --- Calibrate ClassificationRes ---

# Train GLM with cross-validation
resmod_c_glm <- train(
  x = dat,
  algorithm = "glm",
  outer_resampling_config = setup_Resampler(n_resamples = 3L, type = "KFold")
)
#> 2026-03-15 07:49:40 
#> ▶
#>  [train]
#> 2026-03-15 07:49:40 
#> Training set: 100 cases x 4 features.
#>  [summarize_supervised]
#> 2026-03-15 07:49:40 
#> // Max workers: 7 => Algorithm: 1; Tuning: 1; Outer Resampling: 7
#>  [get_n_workers]
#> 2026-03-15 07:49:40 
#> <> Training GLM Classification using 3 independent folds...
#>  [train]
#> 2026-03-15 07:49:41 
#> Input contains more than one column; stratifying on last.
#>  [resample]
#> 2026-03-15 07:49:41 
#> Using max n bins possible = 2.
#>  [kfold]
#> Warning: glm.fit: algorithm did not converge
#> Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
#> Warning: glm.fit: algorithm did not converge
#> Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
#> 2026-03-15 07:49:41 
#> </> Outer resampling done.
#>  [train]
#> 
#> <Resampled Classification Model>
#> GLM (Generalized Linear Model)
#> ⟳ Tested using 3 independent folds.
#> 
#>   <Resampled Classification Training Metrics>
#>     Showing mean (sd) across resamples.
#>           Sensitivity: 0.990 (0.017)
#>           Specificity: 0.990 (0.017)
#>     Balanced_Accuracy: 0.990 (0.017)
#>                   PPV: 0.990 (0.017)
#>                   NPV: 0.990 (0.017)
#>                    F1: 0.990 (0.017)
#>              Accuracy: 0.990 (0.017)
#>                   AUC: 0.999 (2e-03)
#>           Brier_Score: 0.008 (0.014)
#> 
#>   <Resampled Classification Test Metrics>
#>     Showing mean (sd) across resamples.
#>           Sensitivity: 0.961 (0.068)
#>           Specificity: 0.980 (0.034)
#>     Balanced_Accuracy: 0.971 (0.029)
#>                   PPV: 0.981 (0.032)
#>                   NPV: 0.965 (0.061)
#>                    F1: 0.970 (0.031)
#>              Accuracy: 0.971 (0.029)
#>                   AUC: 0.969 (0.031)
#>           Brier_Score: 0.033 (0.025)
#> 
#> 2026-03-15 07:49:41 
#> ✓ Done in 0.09 seconds.
#>  [train]

# Calibrate the `ClassificationRes` using the same resampling configuration as used for training.
resmod_c_glm_cal <- calibrate(resmod_c_glm)
#> 2026-03-15 07:49:41 
#> <> Calibrating GLM resampled classification...
#>  [calibrate]
#> 2026-03-15 07:49:41 
#> // Max workers: 7 => Algorithm: 1; Tuning: 1; Outer Resampling: 1
#>  [get_n_workers]
#> 2026-03-15 07:49:41 
#> Training Isotonic Classification...
#>  [train]
#> 2026-03-15 07:49:41 
#> // Max workers: 7 => Algorithm: 1; Tuning: 1; Outer Resampling: 1
#>  [get_n_workers]
#> 2026-03-15 07:49:41 
#> Training Isotonic Classification...
#>  [train]
#> 2026-03-15 07:49:41 
#> // Max workers: 7 => Algorithm: 1; Tuning: 1; Outer Resampling: 1
#>  [get_n_workers]
#> 2026-03-15 07:49:41 
#> Training Isotonic Classification...
#>  [train]
#> 2026-03-15 07:49:41 
#> // Max workers: 7 => Algorithm: 1; Tuning: 1; Outer Resampling: 1
#>  [get_n_workers]
#> 2026-03-15 07:49:41 
#> Training Isotonic Classification...
#>  [train]
#> 2026-03-15 07:49:41 
#> // Max workers: 7 => Algorithm: 1; Tuning: 1; Outer Resampling: 1
#>  [get_n_workers]
#> 2026-03-15 07:49:41 
#> Training Isotonic Classification...
#>  [train]
#> 2026-03-15 07:49:41 
#> // Max workers: 7 => Algorithm: 1; Tuning: 1; Outer Resampling: 1
#>  [get_n_workers]
#> 2026-03-15 07:49:41 
#> Training Isotonic Classification...
#>  [train]
#> 2026-03-15 07:49:41 
#> // Max workers: 7 => Algorithm: 1; Tuning: 1; Outer Resampling: 1
#>  [get_n_workers]
#> 2026-03-15 07:49:41 
#> Training Isotonic Classification...
#>  [train]
#> 2026-03-15 07:49:41 
#> // Max workers: 7 => Algorithm: 1; Tuning: 1; Outer Resampling: 1
#>  [get_n_workers]
#> 2026-03-15 07:49:41 
#> Training Isotonic Classification...
#>  [train]
#> 2026-03-15 07:49:41 
#> // Max workers: 7 => Algorithm: 1; Tuning: 1; Outer Resampling: 1
#>  [get_n_workers]
#> 2026-03-15 07:49:41 
#> Training Isotonic Classification...
#>  [train]
#> 2026-03-15 07:49:41 
#> // Max workers: 7 => Algorithm: 1; Tuning: 1; Outer Resampling: 1
#>  [get_n_workers]
#> 2026-03-15 07:49:41 
#> Training Isotonic Classification...
#>  [train]
#> 2026-03-15 07:49:41 
#> // Max workers: 7 => Algorithm: 1; Tuning: 1; Outer Resampling: 1
#>  [get_n_workers]
#> 2026-03-15 07:49:41 
#> Training Isotonic Classification...
#>  [train]
#> 2026-03-15 07:49:41 
#> // Max workers: 7 => Algorithm: 1; Tuning: 1; Outer Resampling: 1
#>  [get_n_workers]
#> 2026-03-15 07:49:41 
#> Training Isotonic Classification...
#>  [train]
#> 2026-03-15 07:49:41 
#> // Max workers: 7 => Algorithm: 1; Tuning: 1; Outer Resampling: 1
#>  [get_n_workers]
#> 2026-03-15 07:49:41 
#> Training Isotonic Classification...
#>  [train]
#> 2026-03-15 07:49:41 
#> // Max workers: 7 => Algorithm: 1; Tuning: 1; Outer Resampling: 1
#>  [get_n_workers]
#> 2026-03-15 07:49:41 
#> Training Isotonic Classification...
#>  [train]
#> 2026-03-15 07:49:41 
#> // Max workers: 7 => Algorithm: 1; Tuning: 1; Outer Resampling: 1
#>  [get_n_workers]
#> 2026-03-15 07:49:41 
#> Training Isotonic Classification...
#>  [train]
#> 
#> <Resampled Classification Model>
#> GLM (Generalized Linear Model)
#> ⟳ Tested using 3 independent folds.
#> ⟋ Calibrated using Isotonic Regression with 5 independent folds.
#> 
#>   <Resampled Classification Training Metrics (Pre => Post Calibration)>
#>     Showing mean (sd) across resamples, Pre => Post calibration.
#>           Sensitivity: 0.99 (0.02) => 0.96 (0.06)
#>           Specificity: 0.99 (0.02) => 0.98 (0.03)
#>     Balanced_Accuracy: 0.99 (0.02) => 0.97 (0.03)
#>                   PPV: 0.99 (0.02) => 0.98 (0.03)
#>                   NPV: 0.99 (0.02) => 0.96 (0.05)
#>                    F1: 0.99 (0.02) => 0.97 (0.03)
#>              Accuracy: 0.99 (0.02) => 0.97 (0.03)
#>                   AUC: 1.00 (2e-03) => 0.97 (0.03)
#>           Brier_Score: 0.01 (0.01) => 0.03 (0.03)
#> 
#>   <Resampled Classification Test Metrics (Pre => Post Calibration)>
#>     Showing mean (sd) across resamples, Pre => Post calibration.
#>           Sensitivity: 0.96 (0.07) => 0.96 (0.12)
#>           Specificity: 0.98 (0.03) => 0.92 (0.14)
#>     Balanced_Accuracy: 0.97 (0.03) => 0.94 (0.08)
#>                   PPV: 0.98 (0.03) => 0.94 (0.10)
#>                   NPV: 0.96 (0.06) => 0.97 (0.09)
#>                    F1: 0.97 (0.03) => 0.94 (0.08)
#>              Accuracy: 0.97 (0.03) => 0.94 (0.08)
#>                   AUC: 0.97 (0.03) => 0.94 (0.08)
#>           Brier_Score: 0.03 (0.03) => 0.06 (0.07)
#> 
#> 2026-03-15 07:49:41 
#> </> Calibration done.
#>  [calibrate]
resmod_c_glm_cal
#> <Resampled Classification Model>
#> GLM (Generalized Linear Model)
#> ⟳ Tested using 3 independent folds.
#> ⟋ Calibrated using Isotonic Regression with 5 independent folds.
#> 
#>   <Resampled Classification Training Metrics (Pre => Post Calibration)>
#>     Showing mean (sd) across resamples, Pre => Post calibration.
#>           Sensitivity: 0.99 (0.02) => 0.96 (0.06)
#>           Specificity: 0.99 (0.02) => 0.98 (0.03)
#>     Balanced_Accuracy: 0.99 (0.02) => 0.97 (0.03)
#>                   PPV: 0.99 (0.02) => 0.98 (0.03)
#>                   NPV: 0.99 (0.02) => 0.96 (0.05)
#>                    F1: 0.99 (0.02) => 0.97 (0.03)
#>              Accuracy: 0.99 (0.02) => 0.97 (0.03)
#>                   AUC: 1.00 (2e-03) => 0.97 (0.03)
#>           Brier_Score: 0.01 (0.01) => 0.03 (0.03)
#> 
#>   <Resampled Classification Test Metrics (Pre => Post Calibration)>
#>     Showing mean (sd) across resamples, Pre => Post calibration.
#>           Sensitivity: 0.96 (0.07) => 0.96 (0.12)
#>           Specificity: 0.98 (0.03) => 0.92 (0.14)
#>     Balanced_Accuracy: 0.97 (0.03) => 0.94 (0.08)
#>                   PPV: 0.98 (0.03) => 0.94 (0.10)
#>                   NPV: 0.96 (0.06) => 0.97 (0.09)
#>                    F1: 0.97 (0.03) => 0.94 (0.08)
#>              Accuracy: 0.97 (0.03) => 0.94 (0.08)
#>                   AUC: 0.97 (0.03) => 0.94 (0.08)
#>           Brier_Score: 0.03 (0.03) => 0.06 (0.07)