Conformal prediction regions — conformal • rtemis

A prediction interval for a regression or a set of labels for a classification, covering the truth with probability at least 1 - alpha under exchangeability alone – in finite samples, for any model, including a misspecified one.

Usage

conformal(x, newdata, calibration = NULL, config = NULL, verbosity = 1L, ...)

Arguments

x: Supervised or SupervisedRes object.
newdata: tabular data: Cases to bound. Predictors only, in training order, as stats::predict() requires.
calibration: Optional tabular data: Calibration cases, predictors and outcome, in the shape train() was given. NULL uses the model's stored test split where it has one.
config: Optional ConformalConfig object: Built by setup_SplitConformal, setup_CVPlus or setup_CQR. NULL takes the method the object supports.
verbosity: Integer: Verbosity level.
...: Additional arguments passed to methods.

Value

PredictionInterval object for a regression, PredictionSet for a classification.

Details

The guarantee is marginal. P(Y in C(X)) >= 1 - alpha averages over the draw of both the calibration set and the test case. It is not conditional: 90% marginal coverage is compatible with 99% coverage on easy cases and 60% on a hard subgroup, and averaged over everyone is not the same as guaranteed for anyone. It is also void, silently, under distribution shift – exchangeability is the whole assumption.

Width is the quality measure; coverage is the correctness check. A useless model attains valid coverage by returning intervals wide enough to be uninformative, so read the two together. The region reports its widths (or set sizes); conformal_metrics() scores coverage against outcomes the region did not see.

Which data calibrates. calibration is a tabular dataset holding the predictors and the outcome, in training shape. Left NULL, a model trained with dat_test calibrates on that split, whose residuals it already stores: train() never fits, tunes or early-stops on dat_test. Nothing else is assumed clean – a validation split may have driven early stopping, and a model chosen by its test metric has used that split for selection, which nothing in the object records.

Conformal calibration is unrelated to the probability calibration calibrate performs, and the two must not share rows. Where rtemis can see that they do, it refuses.

Which method. config NULL takes the method the object supports: split conformal for a Supervised, CV+ for a SupervisedRes. See setup_SplitConformal, setup_CVPlus and setup_CQR.

Author

EDG

Examples

x <- data.frame(age = rnorm(300), bmi = rnorm(300))
x[["y"]] <- x[["age"]] * 2 + rnorm(300, sd = 0.3)
# Three splits: one fits, one calibrates, one scores what the other two did.
mod <- train(
  x[1:200, ],
  dat_test = x[201:250, ],
  hyperparameters = setup_GLM(),
  verbosity = 0L
)

# The stored test split calibrates; 90% intervals for five fresh cases.
region <- conformal(mod, x[1:5, c("age", "bmi")], verbosity = 0L)
region

#> <PredictionInterval>
#> split conformal intervals for 5 cases, 90% guaranteed, mean width 0.92

region@lower

#> [1] -0.6287758  1.2317692 -3.9750545 -2.7923820 -0.3453527

region@upper

#> [1]  0.2919234  2.1524684 -3.0543554 -1.8716828  0.5753464


# Coverage, against outcomes nothing in the pipeline has seen.
held_out <- x[251:300, ]
conformal_metrics(
  conformal(mod, held_out[, c("age", "bmi")], verbosity = 0L),
  held_out[["y"]]
)

#>   coverage mean_width median_width  n
#> 1     0.88  0.9206991    0.9206991 50