21  Probability calibration

Probability calibration fits a map from a binary model’s predicted probabilities to observed outcome frequencies. Fit that map using cases separate from model fitting, then assess it on a further held-out sample.

21.1 Fit a model and calibrator

The example uses generated binary data with distinct training, calibration, and assessment samples.

set.seed(2026L)
dat <- data.frame(x1 = rnorm(300), x2 = rnorm(300))
p <- plogis(dat[["x1"]] - 0.5 * dat[["x2"]])
dat[["outcome"]] <- factor(rbinom(300, 1L, p), levels = c(0L, 1L))
training <- dat[1:180, ]
calibration <- dat[181:240, ]
assessment <- dat[241:300, ]
mod <- train(
  training,
  hyperparameters = setup_CART(maxdepth = 3L),
  execution_config = setup_SerialExecution(seed = 2026L)
)

2026-10-08 02:53:56 Checking data is ready for training... ✔ [check_supervised]
2026-10-08 02:53:56 ▶ [train]
2026-10-08 02:53:56 Training set: 180 cases x 2 features. [summarize_supervised]
2026-10-08 02:53:56 // CPU | serial | 1 worker: algorithm 1 thread, tuning 1, outer resampling 1 [msg_resources]
2026-10-08 02:53:56 Shared memory (auto): not sharing training data (5.1 kB) -- this run transfers nothing to workers. [report_shared_memory]
2026-10-08 02:53:56 Training CART Classification... [train]
2026-10-08 02:53:56 Checking data is ready for training... ✔ [check_supervised]

<Classification>
CART (CART decision tree (rpart))

  <Training Classification Metrics>
                     Predicted
          Reference  1   0   
                  1  69  19
                  0  17  75

                     Overall  
        Sensitivity  0.784  
        Specificity  0.815  
  Balanced Accuracy  0.800  
                PPV  0.802  
                NPV  0.798  
                 F1  0.793  
           Accuracy  0.800  
                AUC  0.806  
        Brier Score  0.160  

     Positive Class 1

2026-10-08 02:53:57 Done in 0.93 seconds. [train]

calibrated <- calibrate(
  mod,
  predicted_probabilities = predict(mod, features(calibration))[, 1L],
  true_labels = calibration[["outcome"]],
  hyperparameters = setup_Isotonic()
)

2026-10-08 02:53:57 <> Calibrating CART classification... [calibrate]
2026-10-08 02:53:57 Checking data is ready for training... ✔ [check_supervised]
2026-10-08 02:53:57 ▶ [train]
2026-10-08 02:53:57 Training set: 60 cases x 1 features. [summarize_supervised]
2026-10-08 02:53:57 // CPU | future (mirai_multisession) | 7 workers: algorithm 1 thread, tuning 1, outer resampling 1 [msg_resources]
2026-10-08 02:53:57 Shared memory (auto): not sharing training data (2.6 kB) -- this run transfers nothing to workers. [report_shared_memory]
2026-10-08 02:53:57 Training Isotonic Classification... [train]
2026-10-08 02:53:57 Checking data is ready for training... ✔ [check_supervised]

<Classification>
Isotonic (Isotonic regression)

  <Training Classification Metrics>
                     Predicted
          Reference  1   0  
                  1  33  0
                  0  27  0

                     Overall  
        Sensitivity  1.000  
        Specificity  0.000  
  Balanced Accuracy  0.500  
                PPV  0.550  
                NPV  NA     
                 F1  0.710  
           Accuracy  0.550  
                AUC  0.522  
        Brier Score  0.247  

     Positive Class 1

2026-10-08 02:53:57 Done in 0.03 seconds. [train]

<Classification>
CART (CART decision tree (rpart))
⟋ Calibrated using Isotonic regression.

  <Training Classification Metrics (Pre => Post Calibration)>
                     Predicted
          Reference  1         0        
                  1  69 => 88  19 => 0
                  0  17 => 92  75 => 0

                     1             
        Sensitivity  0.78 => 1.00
        Specificity  0.82 => 0.00
  Balanced Accuracy  0.80 => 0.50
                PPV  0.80 => 0.49
                NPV  0.80 => NA  
                 F1  0.79 => 0.66
           Accuracy  0.80 => 0.49
                AUC  0.81 => 0.75
        Brier Score  0.16 => 0.24

     Positive Class 1

2026-10-08 02:53:57 </> Calibration done. [calibrate]

head(predict(calibrated, features(assessment)))

             1
[1,] 0.5217391
[2,] 0.5217391
[3,] 0.5675676
[4,] 0.5675676
[5,] 0.5675676
[6,] 0.5217391

setup_Isotonic() fits a non-decreasing step function. Its ties can change AUC; its probability bounds depend on calibration sample size. available_calibration() lists supported calibration algorithms, including monotonic HAL.

21.2 Resampled models

calibrate() also accepts ClassificationRes and a calibration resampler_config. The returned object records the calibrator and supports prediction aggregation. An independent assessment sample remains necessary when evaluating the complete calibrated procedure.

Probability calibration concerns the interpretation of probabilities. Conformal prediction uses separate calibration information to construct intervals or label sets. Use separate rows for probability and conformal calibration.