22  Conformal prediction

conformal() produces prediction intervals for regression and prediction sets for classification. Split conformal calibration uses cases held apart from model fitting and selection. Its coverage is marginal over exchangeable calibration and future observations; it is not a per-case or per-subgroup guarantee.

22.1 Split conformal intervals

set.seed(2026L)
dat <- data.frame(x1 = rnorm(300), x2 = rnorm(300))
dat[["outcome"]] <- 2 * dat[["x1"]] + dat[["x2"]] + rnorm(300)
training <- dat[1:180, ]
calibration <- dat[181:240, ]
assessment <- dat[241:300, ]
mod <- train(
  training,
  hyperparameters = setup_GLM(),
  execution_config = setup_SerialExecution(seed = 2026L)
)

2026-10-08 02:53:58 Checking data is ready for training... ✔ [check_supervised]
2026-10-08 02:53:58 ▶ [train]
2026-10-08 02:53:59 Training set: 180 cases x 2 features. [summarize_supervised]
2026-10-08 02:53:59 // CPU | serial | 1 worker: algorithm 1 thread, tuning 1, outer resampling 1 [msg_resources]
2026-10-08 02:53:59 Shared memory (auto): not sharing training data (5.3 kB) -- this run transfers nothing to workers. [report_shared_memory]
2026-10-08 02:53:59 Training GLM Regression... [train]
2026-10-08 02:53:59 Checking data is ready for training... ✔ [check_supervised]

<Regression>
GLM (GLM (generalized linear model))

  <Training Regression Metrics>
   MAE: 0.71
   MSE: 0.78
  RMSE: 0.88
    R²: 0.89

2026-10-08 02:53:59 Done in 0.92 seconds. [train]

region <- conformal(
  mod,
  newdata = features(assessment),
  calibration = calibration,
  config = setup_SplitConformal(alpha = 0.1)
)

2026-10-08 02:53:59 Calibrating on  60  supplied cases... [conformal_calibration_predi...]

head(region@lower)

[1] -1.07185463 -0.99737161  1.13969631 -0.08048922 -3.58821753 -2.65490098

head(region@upper)

[1] 3.1078986 3.1823816 5.3194495 4.0992640 0.5915357 1.5248523

conformal_metrics(region, assessment[["outcome"]])

   coverage mean_width median_width  n
1 0.9666667   4.179753     4.179753 60

For split conformal inference, alpha = 0.1 targets marginal coverage of at least 0.9 under exchangeability. Read interval width together with empirical coverage. Distribution shift or data-dependent selection using the calibration cases changes the conditions supporting that guarantee.

A finite split-conformal threshold requires enough calibration cases for the requested quantile. For example, alpha = 0.05 requires at least 19 cases. An explicit calibration dataset makes its role visible. When omitted, a single fitted model can use its stored test split; that split then serves as calibration data and requires a separate sample for coverage assessment.

22.2 Classification sets

setup_SplitConformal(score = "APS", seed = ...) uses accumulated probability scores. score = "LAC" uses one minus the true-class probability. APS is randomized, so its seed forms part of the configuration. conformal_metrics() reports coverage and set size for a PredictionSet.

22.3 Resampled and quantile methods

setup_CVPlus() uses stored out-of-fold predictions from a resampled model. The assessment folds must partition the cases exactly once; repeated subsampling and bootstrap complements do not satisfy this requirement. The construction and coverage bound depend on the method: jackknife+, finite-fold CV+, and cross-conformal classification have distinct guarantees. Consult the method-specific result and the CV+ reference when interpreting coverage.

setup_CQR() calibrates lower and upper conditional quantiles. The supported Ranger configuration uses setup_Ranger(quantreg = TRUE). CQR requires explicit calibration data because it queries the fitted forest for quantiles at those cases.