Skip to main content

10  Regression

Regression uses a numeric outcome. This example predicts fuel consumption from vehicle measurements in mtcars.

10.1 Define training and test data

dat <- set_outcome(mtcars[, c("mpg", "wt", "hp", "disp")], "mpg")
split <- resample(
  dat,
  setup_StratSub(n_resamples = 1L, train_p = 0.75, seed = 2026L)
)
training_rows <- split[[1L]]
dat_training <- dat[training_rows, ]
dat_test <- dat[-training_rows, ]

10.2 Fit and predict

mod <- train(
  dat_training,
  dat_test = dat_test,
  hyperparameters = setup_GLM(),
  execution_config = setup_SerialExecution(seed = 2026L)
)
2026-10-25 11:45:34 Checking data is ready for training... ✔ [check_supervised]
2026-10-25 11:45:34 ▶ [train]
2026-10-25 11:45:34 Training set: 22 cases x 3 features. [summarize_supervised]
2026-10-25 11:45:34     Test set: 10 cases x 3 features. [summarize_supervised]
2026-10-25 11:45:34 // CPU | serial | 1 worker: algorithm 1 thread, tuning 1, outer resampling 1 [msg_resources]
2026-10-25 11:45:34 Shared memory (auto): not sharing training data (3.4 kB) -- this run transfers nothing to workers. [report_shared_memory]
2026-10-25 11:45:34 Training GLM Regression... [train]
2026-10-25 11:45:34 Checking data is ready for training... ✔ [check_supervised]

<Regression>
GLM (GLM (generalized linear model))

  <Training Regression Metrics>
   MAE: 1.88
   MSE: 6.18
  RMSE: 2.49
    R²: 0.84

  <Test Regression Metrics>
   MAE: 1.92
   MSE: 6.25
  RMSE: 2.50
    R²: 0.77

2026-10-25 11:45:35 Done in 0.35 seconds. [train]
predict(mod, features(dat_test))
       Datsun 710 Hornet Sportabout          Merc 230          Merc 280 
         25.00797          17.76391          22.00378          20.01420 
       Merc 450SL       Merc 450SLC          Fiat 128        Camaro Z28 
         16.84267          16.67122          26.38884          14.12145 
 Pontiac Firebird     Maserati Bora 
         16.26241          12.22946 

predict() estimates the outcome for new cases. The model also stores the true and predicted values of each sample it was trained and tested on: get_predictions() returns them, and get_metrics() the metrics computed from them.

head(get_predictions(mod, "test"))
   sample  true predicted
   <char> <num>     <num>
1:   test  22.8  25.00797
2:   test  18.7  17.76391
3:   test  22.8  22.00378
4:   test  19.2  20.01420
5:   test  17.3  16.84267
6:   test  15.2  16.67122
get_metrics(mod, "test")
<Test Regression Metrics>
 MAE: 1.92
 MSE: 6.25
RMSE: 2.50
  R²: 0.77

10.3 Assess the fit

review(mod)
<SupervisedReview>
  GLM (generalized linear model) was used for regression. R-squared was 0.84 on the training set and 0.77 on the test set.
  22 training and 10 test cases; 3 predictors (7.3 training cases per predictor).

  Performance
          Training   Test  Training - test
    MAE      1.878  1.922           -0.045
    MSE      6.177  6.254           -0.076
    RMSE     2.485  2.501           -0.015
    R²       0.840  0.769            0.071

  Baseline comparisons
    MSE 6.254 vs 27.625 (training mean); loss reduction 21.372 (95% CI -3.669 to 46.412): indistinguishable
    MAE 1.922 vs 4.137 (training mean); loss reduction 2.215 (95% CI 0.378 to 4.052): better
    R² 0.769 vs -0.020 (training mean)

  Findings
    * SINGLE_SPLIT (note): Performance was estimated on a single split, with 10 test cases. The test intervals reflect the number of test cases, not how much the estimate would change with a different split.
      Suggestion: Use outer resampling to test on every case and to measure variation between splits; repeat it when cases are few.
    * TEST_PRECISION (note): Test mean squared error is 6.254 (95% CI 0.000 to 14.085), from 10 test cases.
    * FEW_CASES_PER_PREDICTOR (note): There are 7.333 training cases per learner column (3 columns), fewer than 10. This is a rule of thumb from logistic regression (events per variable), not a validated sample-size requirement for every algorithm.
      Suggestion: Judge the model by held-out performance. Consider a regularized or sparse algorithm, or a decomposition step in train(), which is fitted on training cases only.
    * PRESELECTION_RISK (note): With few cases per predictor, predictors are often selected before modeling. Selection or filtering of predictors done on all the data before it was passed to train() biases evaluation; this review cannot determine whether that happened. Preprocessing, decomposition and tuning within train() use training cases only.
      Suggestion: Avoid selecting predictors on all the data before train(). To reduce the predictors, use a sparse algorithm or a decomposition step in train(); any selection done outside rtemis must use only the training cases of each split.
    * BASELINE_MSE (warning): Test mean squared error of 6.254 (mean reduction from the baseline's 27.625: 21.372, 95% CI -3.669 to 46.412) does not provide clear evidence of a difference from predicting the training mean for every case; limited precision can cause this.

  Limitations
    - Whether the model is useful depends on the question it addresses, the costs of different errors, and how its predictions will be used; performance metrics cannot establish this.
    - Preprocessing, decomposition and tuning done by rtemis are fitted on the training cases of each split and only applied to its test cases. Steps taken before the data was passed to rtemis -- selecting or filtering predictors, transforming or removing cases using all the data, or choosing among models by their test performance -- can bias evaluation; this review cannot determine whether that happened.
    - Performance estimates apply to cases from the population the data was drawn from; performance on a different population or setting may differ.
plot_true_pred(mod)

Read error measures together with sample size, the outcome’s units, and the baseline comparison. A small test set gives an imprecise assessment even when its point estimate appears favorable.

10.4 Outer resampling

resampled <- train(
  dat,
  hyperparameters = setup_GLM(),
  outer_resampling_config = setup_KFold(n_resamples = 4L, seed = 2026L),
  execution_config = setup_SerialExecution(seed = 2027L)
)
2026-10-25 11:45:35 Checking data is ready for training... ✔ [check_supervised]
2026-10-25 11:45:35 ▶ [train]
2026-10-25 11:45:35 Training set: 32 cases x 3 features. [summarize_supervised]
2026-10-25 11:45:35 // CPU | serial | 1 worker: algorithm 1 thread, tuning 1, outer resampling 1 [msg_resources]
2026-10-25 11:45:35 Shared memory (auto): not sharing training data (4.5 kB) -- this run transfers nothing to workers. [report_shared_memory]
2026-10-25 11:45:35 <> Training GLM Regression using 4 independent folds... [train]
2026-10-25 11:45:35 ✔ Outer resamples 4/4 done in 0:00
2026-10-25 11:45:35 </> Outer resampling done. [train]
<Resampled Regression Model>
GLM (GLM (generalized linear model))
⟳ Tested using 4 independent folds.

  <Resampled Regression Training Metrics>
  Showing mean (sd) across resamples.
   MAE: 1.853 (0.177)
   MSE: 5.857 (0.845)
  RMSE: 2.415 (0.177)
    R²: 0.833 (0.027)

  <Resampled Regression Test Metrics>
  Showing mean (sd) across resamples.
   MAE: 2.091 (0.465)
   MSE: 7.363 (2.931)
  RMSE: 2.672 (0.547)
    R²: 0.787 (0.094)

2026-10-25 11:45:35 Done in 0.05 seconds. [train]
review(resampled)
<SupervisedReview>
  GLM (generalized linear model) was used for regression. Mean R-squared was 0.83 on the training set and 0.79 on the test set across 4 independent folds.
  4 resamples: 32 out-of-sample predictions of 32 cases, at least 22 training cases per resample; 3 predictors (7.3 training cases per predictor).

  Performance, mean (SD) over 4 resamples
               Training           Test  Training - test
    MAE   1.853 (0.177)  2.091 (0.465)           -0.238
    MSE   5.857 (0.845)  7.363 (2.931)           -1.507
    RMSE  2.415 (0.177)  2.672 (0.547)           -0.257
    R²    0.833 (0.027)  0.787 (0.094)            0.046

  Baseline comparisons, mean over resamples
    MSE 7.363 vs 35.495 (training mean); better in 4 of 4 resamples compared
    MAE 2.091 vs 4.743 (training mean); better in 4 of 4 resamples compared
    R² 0.787 vs -0.007 (training mean); better in 4 of 4 resamples compared

  Findings
    * FOLD_VARIATION (note): Test mean squared error ranged from 4.324 to 10.711 across 4 resamples (mean 7.363, SD 2.931); 4 of 4 resamples in which the metric is defined outperformed their baseline. Resamples share training cases, so their results are not independent, and no confidence interval or test is computed from them.
    * FEW_CASES_PER_PREDICTOR (note): There are 7.333 training cases per learner column (3 columns), fewer than 10. This is a rule of thumb from logistic regression (events per variable), not a validated sample-size requirement for every algorithm.
      Suggestion: Judge the model by held-out performance. Consider a regularized or sparse algorithm, or a decomposition step in train(), which is fitted on training cases only.
    * PRESELECTION_RISK (note): With few cases per predictor, predictors are often selected before modeling. Selection or filtering of predictors done on all the data before it was passed to train() biases evaluation; this review cannot determine whether that happened. Preprocessing, decomposition and tuning within train() use training cases only.
      Suggestion: Avoid selecting predictors on all the data before train(). To reduce the predictors, use a sparse algorithm or a decomposition step in train(); any selection done outside rtemis must use only the training cases of each split.

  Limitations
    - Whether the model is useful depends on the question it addresses, the costs of different errors, and how its predictions will be used; performance metrics cannot establish this.
    - Preprocessing, decomposition and tuning done by rtemis are fitted on the training cases of each split and only applied to its test cases. Steps taken before the data was passed to rtemis -- selecting or filtering predictors, transforming or removing cases using all the data, or choosing among models by their test performance -- can bias evaluation; this review cannot determine whether that happened.
    - Performance estimates apply to cases from the population the data was drawn from; performance on a different population or setting may differ.

RegressionRes retains the fitted resamples and their metrics. For prediction, type = "avg" aggregates predictions across fitted models and type = "all" returns their individual predictions.

predict(resampled, features(dat)[1:3, ], type = "avg")
    Mazda RX4 Mazda RX4 Wag    Datsun 710 
     23.58495      22.57305      25.28262 

se() computes model-specific uncertainty where the learner supports it. Conformal prediction constructs prediction intervals using held-out calibration information.

© 2026 E.D. Gennatas