hyperparameters <- setup_CART(cp = tune_over(0.001, 0.01, 0.05), maxdepth = tune_over(2L, 4L))
tuning_grid(hyperparameters) cp maxdepth
1 0.001 2
2 0.010 2
3 0.050 2
4 0.001 4
5 0.010 4
6 0.050 412 Hyperparameter tuning
Tuning selects a configuration using inner resampling. The tuner_config argument to train() specifies the selection procedure. Outer resampling or an independent test set assesses the selected procedure.
12.1 Inspect a search space
tune_over() explicitly marks candidate values. A bare vector retains its meaning as one value; a scalar property rejects a vector of alternatives.
tuning_grid() returns the combinations the search considers. Applicability rules remove irrelevant settings and duplicate combinations.
12.2 Configure the tuner
Pass a configuration from setup_GridSearch() as tuner_config to train(). Its arguments control the selection procedure:
| Argument | Purpose |
|---|---|
resampler_config |
Internal resampling method and its settings, including fold count, repetitions, and seed where applicable. |
search_type |
Exhaustive evaluation or randomized sampling of candidate combinations. |
randomize_p |
Proportion of combinations evaluated in a randomized search. |
metric |
Performance metric used to compare candidates. |
maximize |
Whether selection maximizes or minimizes the metric. |
metrics_aggregate_fn |
Function used to aggregate the metric across resamples. |
For example, resampler_config = setup_KFold(n_resamples = 5L) requests 5-fold cross-validation. Other resampling designs and their controls are introduced in partitions and resampling.
12.3 Run nested resampling
mod <- train(
iris,
hyperparameters = hyperparameters,
tuner_config = setup_GridSearch(
resampler_config = setup_KFold(n_resamples = 3L, seed = 2026L)
),
outer_resampling_config = setup_KFold(n_resamples = 3L, seed = 2027L),
execution_config = setup_SerialExecution(seed = 2028L)
)<Resampled Classification Model>
CART (CART decision tree (rpart))
⚙ Tuned using exhaustive grid search.
⟳ Tested using 3 independent folds.
<Resampled Classification Training Metrics>
Aggregate Confusion Matrix across resamples.
Predicted
Reference setosa versicolor virginica
setosa 100 0 0
versicolor 0 97 3
virginica 0 1 99
Showing mean (sd) across resamples.
Balanced Accuracy: 0.987 (0.015)
F1: 0.987 (0.015)
Accuracy: 0.987 (0.015)
<Resampled Classification Test Metrics>
Aggregate Confusion Matrix across resamples.
Predicted
Reference setosa versicolor virginica
setosa 50 0 0
versicolor 0 43 7
virginica 0 3 47
Showing mean (sd) across resamples.
Balanced Accuracy: 0.933 (0.010)
F1: 0.933 (0.010)
Accuracy: 0.933 (0.010)
review(mod)<SupervisedReview>
CART decision tree (rpart) was used for classification. Hyperparameter tuning was performed using exhaustive grid search. Mean balanced accuracy was 0.99 in the training set and 0.93 in the test set across 3 independent folds.
3 resamples: 150 out-of-sample predictions of 150 cases, at least 99 training cases per resample; 4 predictors (8.2 minority-class training cases per predictor).
Performance, mean (SD) over 3 resamples
Training Test Training - test
Balanced Accuracy 0.987 (0.015) 0.933 (0.010) 0.053
F1 0.987 (0.015) 0.933 (0.010) 0.053
Accuracy 0.987 (0.015) 0.933 (0.010) 0.053
Baseline comparisons, mean over resamples
Accuracy 0.933 vs 0.333 (most common training class); better in 3 of 3 resamples compared
Balanced Accuracy 0.933 vs 0.333 (chance level); better in 3 of 3 resamples compared
Tuning
cp: searched 0.001 to 0.05 (3 values), 3 of 3 resamples at an extendable edge
Findings
* FOLD_VARIATION (note): Test balanced accuracy ranged from 0.922 to 0.941 across 3 resamples (mean 0.933, SD 0.010); 3 of 3 resamples in which the metric is defined outperformed their baseline. Resamples share training cases, so their results are not independent, and no confidence interval or test is computed from them.
* FEW_CASES_PER_PREDICTOR (note): There are 8.250 minority-class training cases per learner column (4 columns), fewer than 10. This is a rule of thumb from logistic regression (events per variable), not a validated sample-size requirement for every algorithm.
Suggestion: Judge the model by held-out performance. Consider a regularized or sparse algorithm, or a decomposition step in train(), which is fitted on training cases only.
* PRESELECTION_RISK (note): With few cases per predictor, predictors are often selected before modeling. Selection or filtering of predictors done on all the data before it was passed to train() biases evaluation; this review cannot determine whether that happened. Preprocessing, decomposition and tuning within train() use training cases only.
Suggestion: Avoid selecting predictors on all the data before train(). To reduce the predictors, use a sparse algorithm or a decomposition step in train(); any selection done outside rtemis must use only the training cases of each split.
* TUNING_GRID_EDGE (note): 3 of 3 resamples selected cp at the edge of the values searched (0.001 to 0.050); the best value may lie beyond it.
Suggestion: Extend the search range of cp past that edge.
Limitations
- Whether the model is useful depends on the question it addresses, the costs of different errors, and how its predictions will be used; performance metrics cannot establish this.
- Preprocessing, decomposition and tuning done by rtemis are fitted on the training cases of each split and only applied to its test cases. Steps taken before the data was passed to rtemis -- selecting or filtering predictors, transforming or removing cases using all the data, or choosing among models by their test performance -- can bias evaluation; this review cannot determine whether that happened.
- Performance estimates apply to cases from the population the data was drawn from; performance on a different population or setting may differ.
Each outer training set undergoes its own inner search. The selected configuration is refitted on that outer training set and assessed on its held-out fold. The tuner controls the selection metric, its direction, and aggregation across inner resamples.
setup_GridSearch(search_type = "randomized", randomize_p = ...) samples a proportion of the grid. Search size, inner resampling, outer resampling, and learner complexity jointly determine computational cost.
12.4 Named variants
A named list represents alternative configurations of the same algorithm. Each entry can carry its own candidate values.
variants <- list(
shallow = setup_CART(maxdepth = 2L, cp = tune_over(0.01, 0.05)),
deeper = setup_CART(maxdepth = 5L, cp = tune_over(0.001, 0.01))
)
lapply(variants, tuning_grid)$shallow
cp
1 0.01
2 0.05
$deeper
cp
1 0.001
2 0.010Pass this list as hyperparameters to train(). The selected fitted configuration records its variant.
selected <- train(
iris,
hyperparameters = variants,
tuner_config = setup_GridSearch(
resampler_config = setup_KFold(n_resamples = 3L, seed = 2026L)
),
execution_config = setup_SerialExecution(seed = 2027L)
)<Classification>
CART (CART decision tree (rpart))
⚙ Tuned using exhaustive grid search.
<Training Classification Metrics>
Predicted
Reference setosa versicolor virginica
setosa 50 0 0
versicolor 0 50 0
virginica 0 0 50
Overall
Balanced Accuracy 1.000
F1 1.000
Accuracy 1.000
Setosa Versicolor Virginica
Sensitivity 1.000 1.000 1.000
Specificity 1.000 1.000 1.000
Balanced Accuracy 1.000 1.000 1.000
PPV 1.000 1.000 1.000
NPV 1.000 1.000 1.000
F1 1.000 1.000 1.000
selected@hyperparameters@variant[1] "deeper"A run record retains selection results for each fitted outer fold.
Selection at the edge of a search range is reported by review(). Expanding the search uses training information; repeated selection by held-out test performance also uses that test set for model selection.
Vector-valued settings can also be candidates. For example, setup_MLP(hidden_units = tune_over(c(16L, 8L), c(32L, 16L))) searches over two complete network architectures. Serialized search spaces use a candidates object; named configuration alternatives use a variants map.
