dat <- set_outcome(mtcars[, c("mpg", "wt", "hp", "disp")], "mpg")
split <- resample(
dat,
setup_StratSub(n_resamples = 1L, train_p = 0.75, seed = 2026L)
)
training_rows <- split[[1L]]
dat_training <- dat[training_rows, ]
dat_test <- dat[-training_rows, ]10 Regression
Regression uses a numeric outcome. This example predicts fuel consumption from vehicle measurements in mtcars.
10.1 Define training and test data
10.2 Fit and predict
mod <- train(
dat_training,
dat_test = dat_test,
hyperparameters = setup_GLM(),
execution_config = setup_SerialExecution(seed = 2026L)
)<Regression>
GLM (GLM (generalized linear model))
<Training Regression Metrics>
MAE: 1.88
MSE: 6.18
RMSE: 2.49
R²: 0.84
<Test Regression Metrics>
MAE: 1.92
MSE: 6.25
RMSE: 2.50
R²: 0.77
predict(mod, features(dat_test)) Datsun 710 Hornet Sportabout Merc 230 Merc 280
25.00797 17.76391 22.00378 20.01420
Merc 450SL Merc 450SLC Fiat 128 Camaro Z28
16.84267 16.67122 26.38884 14.12145
Pontiac Firebird Maserati Bora
16.26241 12.22946
predict() estimates the outcome for new cases. The model also stores the true and predicted values of each sample it was trained and tested on: get_predictions() returns them, and get_metrics() the metrics computed from them.
head(get_predictions(mod, "test")) sample true predicted
<char> <num> <num>
1: test 22.8 25.00797
2: test 18.7 17.76391
3: test 22.8 22.00378
4: test 19.2 20.01420
5: test 17.3 16.84267
6: test 15.2 16.67122
get_metrics(mod, "test")<Test Regression Metrics>
MAE: 1.92
MSE: 6.25
RMSE: 2.50
R²: 0.77
10.3 Assess the fit
review(mod)<SupervisedReview>
GLM (generalized linear model) was used for regression. R-squared was 0.84 on the training set and 0.77 on the test set.
22 training and 10 test cases; 3 predictors (7.3 training cases per predictor).
Performance
Training Test Training - test
MAE 1.878 1.922 -0.045
MSE 6.177 6.254 -0.076
RMSE 2.485 2.501 -0.015
R² 0.840 0.769 0.071
Baseline comparisons
MSE 6.254 vs 27.625 (training mean); loss reduction 21.372 (95% CI -3.669 to 46.412): indistinguishable
MAE 1.922 vs 4.137 (training mean); loss reduction 2.215 (95% CI 0.378 to 4.052): better
R² 0.769 vs -0.020 (training mean)
Findings
* SINGLE_SPLIT (note): Performance was estimated on a single split, with 10 test cases. The test intervals reflect the number of test cases, not how much the estimate would change with a different split.
Suggestion: Use outer resampling to test on every case and to measure variation between splits; repeat it when cases are few.
* TEST_PRECISION (note): Test mean squared error is 6.254 (95% CI 0.000 to 14.085), from 10 test cases.
* FEW_CASES_PER_PREDICTOR (note): There are 7.333 training cases per learner column (3 columns), fewer than 10. This is a rule of thumb from logistic regression (events per variable), not a validated sample-size requirement for every algorithm.
Suggestion: Judge the model by held-out performance. Consider a regularized or sparse algorithm, or a decomposition step in train(), which is fitted on training cases only.
* PRESELECTION_RISK (note): With few cases per predictor, predictors are often selected before modeling. Selection or filtering of predictors done on all the data before it was passed to train() biases evaluation; this review cannot determine whether that happened. Preprocessing, decomposition and tuning within train() use training cases only.
Suggestion: Avoid selecting predictors on all the data before train(). To reduce the predictors, use a sparse algorithm or a decomposition step in train(); any selection done outside rtemis must use only the training cases of each split.
* BASELINE_MSE (warning): Test mean squared error of 6.254 (mean reduction from the baseline's 27.625: 21.372, 95% CI -3.669 to 46.412) does not provide clear evidence of a difference from predicting the training mean for every case; limited precision can cause this.
Limitations
- Whether the model is useful depends on the question it addresses, the costs of different errors, and how its predictions will be used; performance metrics cannot establish this.
- Preprocessing, decomposition and tuning done by rtemis are fitted on the training cases of each split and only applied to its test cases. Steps taken before the data was passed to rtemis -- selecting or filtering predictors, transforming or removing cases using all the data, or choosing among models by their test performance -- can bias evaluation; this review cannot determine whether that happened.
- Performance estimates apply to cases from the population the data was drawn from; performance on a different population or setting may differ.
plot_true_pred(mod)Read error measures together with sample size, the outcome’s units, and the baseline comparison. A small test set gives an imprecise assessment even when its point estimate appears favorable.
10.4 Outer resampling
resampled <- train(
dat,
hyperparameters = setup_GLM(),
outer_resampling_config = setup_KFold(n_resamples = 4L, seed = 2026L),
execution_config = setup_SerialExecution(seed = 2027L)
)<Resampled Regression Model>
GLM (GLM (generalized linear model))
⟳ Tested using 4 independent folds.
<Resampled Regression Training Metrics>
Showing mean (sd) across resamples.
MAE: 1.853 (0.177)
MSE: 5.857 (0.845)
RMSE: 2.415 (0.177)
R²: 0.833 (0.027)
<Resampled Regression Test Metrics>
Showing mean (sd) across resamples.
MAE: 2.091 (0.465)
MSE: 7.363 (2.931)
RMSE: 2.672 (0.547)
R²: 0.787 (0.094)
review(resampled)<SupervisedReview>
GLM (generalized linear model) was used for regression. Mean R-squared was 0.83 on the training set and 0.79 on the test set across 4 independent folds.
4 resamples: 32 out-of-sample predictions of 32 cases, at least 22 training cases per resample; 3 predictors (7.3 training cases per predictor).
Performance, mean (SD) over 4 resamples
Training Test Training - test
MAE 1.853 (0.177) 2.091 (0.465) -0.238
MSE 5.857 (0.845) 7.363 (2.931) -1.507
RMSE 2.415 (0.177) 2.672 (0.547) -0.257
R² 0.833 (0.027) 0.787 (0.094) 0.046
Baseline comparisons, mean over resamples
MSE 7.363 vs 35.495 (training mean); better in 4 of 4 resamples compared
MAE 2.091 vs 4.743 (training mean); better in 4 of 4 resamples compared
R² 0.787 vs -0.007 (training mean); better in 4 of 4 resamples compared
Findings
* FOLD_VARIATION (note): Test mean squared error ranged from 4.324 to 10.711 across 4 resamples (mean 7.363, SD 2.931); 4 of 4 resamples in which the metric is defined outperformed their baseline. Resamples share training cases, so their results are not independent, and no confidence interval or test is computed from them.
* FEW_CASES_PER_PREDICTOR (note): There are 7.333 training cases per learner column (3 columns), fewer than 10. This is a rule of thumb from logistic regression (events per variable), not a validated sample-size requirement for every algorithm.
Suggestion: Judge the model by held-out performance. Consider a regularized or sparse algorithm, or a decomposition step in train(), which is fitted on training cases only.
* PRESELECTION_RISK (note): With few cases per predictor, predictors are often selected before modeling. Selection or filtering of predictors done on all the data before it was passed to train() biases evaluation; this review cannot determine whether that happened. Preprocessing, decomposition and tuning within train() use training cases only.
Suggestion: Avoid selecting predictors on all the data before train(). To reduce the predictors, use a sparse algorithm or a decomposition step in train(); any selection done outside rtemis must use only the training cases of each split.
Limitations
- Whether the model is useful depends on the question it addresses, the costs of different errors, and how its predictions will be used; performance metrics cannot establish this.
- Preprocessing, decomposition and tuning done by rtemis are fitted on the training cases of each split and only applied to its test cases. Steps taken before the data was passed to rtemis -- selecting or filtering predictors, transforming or removing cases using all the data, or choosing among models by their test performance -- can bias evaluation; this review cannot determine whether that happened.
- Performance estimates apply to cases from the population the data was drawn from; performance on a different population or setting may differ.
RegressionRes retains the fitted resamples and their metrics. For prediction, type = "avg" aggregates predictions across fitted models and type = "all" returns their individual predictions.
predict(resampled, features(dat)[1:3, ], type = "avg") Mazda RX4 Mazda RX4 Wag Datsun 710
23.58495 22.57305 25.28262
se() computes model-specific uncertainty where the learner supports it. Conformal prediction constructs prediction intervals using held-out calibration information.