12  Resampled classifiers

library(rtemis.draw)

Attaching package: 'rtemis.draw'

The following object is masked from 'package:graphics':

    Axis

A cross-validated classifier gives several useful views of performance. Confusion counts summarize class decisions, fold metrics show variation between fits, and ROC curves describe discrimination across thresholds. These examples use the same rtemis model object throughout.

12.1 Fit a cross-validated model

Use sepal length and width to distinguish virginica from the other iris species. Explicit factor levels define the two classes. Five-fold cross-validation fits a logistic regression on four folds and predicts the remaining fold:

flowers <- iris[c("Sepal.Length", "Sepal.Width")]
flowers[["Species"]] <- factor(
  ifelse(iris[["Species"]] == "virginica", "Virginica", "Other"),
  levels = c("Other", "Virginica")
)

classifier_cv <- rtemis::train(
  flowers,
  hyperparameters = rtemis::setup_GLM(),
  outer_resampling_config = rtemis::setup_KFold(n_resamples = 5L, seed = 31L),
  execution_config = rtemis::setup_SerialExecution(), verbosity = 0L
)

Here what = "test" selects each fold’s held-out predictions. Every flower appears once in that pooled test sample and in four training samples. The held-out folds are parts of this cross-validation run, not an additional external test dataset. The model settings are fixed before this run.

12.2 Pool the classification decisions

cv_confusion <- rtemis::plot_true_pred(
  classifier_cv, what = "test", digits = 2L,
  title = "Out-of-fold classifications"
)
cv_confusion

Rows are reference classes and columns are predicted classes. The cells count the 150 held-out decisions across the five folds. The surrounding metrics are calculated from these pooled counts, so they need not equal the mean of the corresponding fold metrics.

With repeated cross-validation or other resampling designs, an observation may contribute several predictions. The pooled view retains those repeats; it does not select or average a prediction for each unique observation.

12.3 Inspect variation between folds

cv_metrics <- rtemis::plot_metric(
  classifier_cv, metric = "balanced_accuracy",
  ylab = "Balanced accuracy",
  title = "Balanced accuracy by fold"
)
cv_metrics

Each point is a stored score from one fold. For these two classes, balanced accuracy is the mean of the two class recalls. Training and test boxes summarize their respective fold scores with equal weight per fold. Their spread describes this run; it is not a confidence interval. Hover a point to identify its fold.

12.4 Pool the predicted probabilities

ROC curves use stored probabilities instead of the final class decisions. Name the positive class explicitly:

cv_roc_pooled <- rtemis::plot_roc(
  classifier_cv, what = "test", positive = "Virginica",
  variant = "aggregate", digits = 2L,
  title = "Pooled held-out ROC"
)
cv_roc_pooled

The pooled curve combines the held-out probabilities from all five fitted models. Its AUC compares positive and negative observations across that combined sample, including pairs scored by different fits. Interpreting it therefore involves the comparability of scores across fits.

12.5 Inspect each fold’s ROC curve

cv_roc_folds <- rtemis::plot_roc(
  classifier_cv, what = "test", positive = "Virginica",
  variant = "per_resample", digits = 2L,
  title = "Held-out ROC by fold"
)
cv_roc_folds

Each curve uses one fold’s held-out observations. The legend reports the unweighted mean and sample standard deviation of the fold AUCs, followed by the number of defined curves. This mean AUC makes comparisons within each fold before averaging; it is a different summary from the pooled AUC.

Hover a curve to inspect its fold and AUC. Click the legend entry to hide or restore the group; the chance diagonal remains visible. legend_position can right-align the outside row with "top-right"; add legend_placement = "inside" to place it within the plotting area.

12.6 Save a figure

save_drawing(cv_roc_folds, "cross-validation-roc.svg", width = 700, height = 700)

For single fitted models, see Plotting model objects. For ROC plots made directly from labels and probabilities, see Classification diagnostics.