Decompose a learner's error into the part that comes from being systematically wrong and the part that comes from being unstable, by refitting it over many resamples of the training set and watching what its predictions do at a fixed test set.
Usage
bias_variance(
x,
hyperparameters,
dat_test = NULL,
true_values = NULL,
resampler_config = NULL,
test_p = 0.3,
execution_config = setup_FutureExecution(),
verbosity = 1L
)Arguments
- x
tabular data: Full dataset, outcome in the last column. Resamples are drawn from it after the test set is held out.
- hyperparameters
Hyperparametersobject: Must be fixed, not a search space – this measures one model, not a tuning run.- dat_test
Optional tabular data: Test set. NULL holds one out of
x.- true_values
Optional Numeric vector: The true function at the test cases, in their order. Regression:
f(x). Classification: the probability of the positive class. NULL measures bias against the observed outcome, which includes irreducible noise.- resampler_config
Optional
ResamplerConfigobject: How training sets are drawn. NULL uses 100 stratified bootstraps.- test_p
Numeric (0, 1): Fraction of
xheld out as the test set. Ignored whendat_testis given.- execution_config
ExecutionConfigobject: Supplies the seed and the workers the resamples are fitted over.- verbosity
Integer: If > 0, print messages.
Details
For squared-error loss, the expected error at a point decomposes as
$$E[(y - \hat{f}(x))^2] = \sigma^2 + (E[\hat{f}(x)] - f(x))^2 + Var[\hat{f}(x)]$$
– irreducible noise, squared bias, and variance. This function estimates the
second and third per test case, by fitting the model on n_resamples
resamples of the training set and reading the spread and the average of the
predictions each makes at the same test cases.
The test set is held out once and every resample is drawn from what is left, which is what makes the predictions comparable across resamples: a test set that moved would confound the model's variability with the sample's.
Whether the bias is really the bias
Squared bias is measured against true_values when you have them, and
against the observed outcome when you do not. The difference is not
cosmetic:
- With
true_values bias_squaredis the bias, exactly. Only simulated data has this, which is what makes a simulation study the place to ask how a learner divides its error.- Without
The observed outcome carries the irreducible noise, so what is reported is \(bias^2 + \sigma^2\) and no amount of resampling separates them. It stays comparable between learners on the same data – they share the same \(\sigma^2\) – so a ranking is still meaningful even though the level is inflated.
What the numbers are, exactly
variance is the sample variance across resamples (dividing by
n_resamples - 1), which is unbiased for the variance of the fitted
function. bias_squared is the plug-in estimate, and is inflated by roughly
variance / n_resamples because the mean prediction is itself estimated –
so prefer many resamples over few, and treat a bias_squared of the same
order as variance / n_resamples as indistinguishable from zero.
Classification
The decomposition above is for squared error, and 0-1 loss does not
decompose additively into bias and variance. For a binary outcome this
function therefore decomposes the squared error of the predicted
probability of the positive class, which does. true_values is then the
true probability, not the class. Multiclass is not supported.
References
Geman S, Bienenstock E, Doursat R (1992). Neural Networks and the Bias/Variance Dilemma. Neural Computation, 4(1), 1-58. doi:10.1162/neco.1992.4.1.1
Examples
set.seed(2026)
n <- 200L
features <- data.frame(a = rnorm(n), b = rnorm(n))
dat <- data.frame(features, y = 2 * features[["a"]] + rnorm(n))
bv <- bias_variance(
dat,
hyperparameters = setup_CART(),
resampler_config = setup_StratBoot(n_resamples = 10L),
verbosity = 0L
)
bv#> <BiasVariance>
#> Mean squared bias: 1.23 (1.52)
#> Mean variance: 0.27 (0.22)