Score a Conformal Prediction Region — conformal

Empirical coverage and region size for a PredictionRegion, against the outcomes of the cases it bounds.

Usage

conformal_metrics(region, true_outcome)

Arguments

region: PredictionRegion object, from conformal.
true_outcome: Numeric or Factor: The outcomes of the cases region bounds, in the same order.

Value

One-row data.frame. coverage and n for either shape, plus mean_width and median_width for an interval, or mean_set_size, singleton_rate and empty_rate for a set.

Details

Coverage on the calibration data is not a check. The threshold is chosen as the 1 - alpha empirical quantile of the calibration scores, so coverage measured back on those same cases is 1 - alpha by construction. Score against outcomes nothing in the pipeline has seen, or the number means nothing.

Read coverage and size together. A model with no signal attains valid coverage by returning intervals wide enough to be useless, so coverage alone cannot separate a good model from a wide one. Width – or set size, and the rate of singleton and empty sets – is the part that carries information about the model.

Coverage is a proportion over the cases supplied, so it carries the Monte Carlo error of that sample: 100 cases at a nominal 90% have a standard error near 3 points, and a single run landing at 87% is not evidence of a broken guarantee.

Author

EDG

Examples

x <- data.frame(age = rnorm(300), bmi = rnorm(300))
x[["y"]] <- x[["age"]] * 2 + rnorm(300, sd = 0.3)
mod <- train(
  x[1:200, ],
  dat_test = x[201:250, ],
  hyperparameters = setup_GLM(),
  verbosity = 0L
)
held_out <- x[251:300, ]
region <- conformal(mod, held_out[, c("age", "bmi")], verbosity = 0L)
conformal_metrics(region, held_out[["y"]])

#>   coverage mean_width median_width  n
#> 1     0.98   1.128317     1.128317 50