16  Gradient boosting

Gradient boosting fits an additive sequence of learners under a specified loss. In rtemis, setup_LightGBM() selects LightGBM and its tree, sampling, regularization, and objective settings.

16.1 Fit a model

The example requires the lightgbm package. Internal resampling selects the number of boosting rounds; held-out test data supply performance assessment.

set.seed(2026L)
dat <- data.frame(x1 = rnorm(200), x2 = rnorm(200))
dat[["outcome"]] <- dat[["x1"]]^2 + dat[["x2"]] + rnorm(200)
mod <- train(
  dat[1:160, ],
  dat_test = dat[161:200, ],
  hyperparameters = setup_LightGBM(max_nrounds = 30L, num_leaves = 4L),
  execution_config = setup_SerialExecution(n_workers_algorithm = 1L, seed = 2026L)
)

2026-10-08 03:41:49 Checking data is ready for training... ✔ [check_supervised]
2026-10-08 03:41:49 ▶ [train]
2026-10-08 03:41:49 Training set: 160 cases x 2 features. [summarize_supervised]
2026-10-08 03:41:49     Test set: 40 cases x 2 features. [summarize_supervised]
2026-10-08 03:41:49 // CPU (auto-selected) | serial | 1 worker: algorithm 1 thread, tuning 1, outer resampling 1 (as set) [msg_resources]
2026-10-08 03:41:49 Shared memory (auto): not sharing training data (4.8 kB) -- this run transfers nothing to workers. [report_shared_memory]
2026-10-08 03:41:49 <> Tuning LightGBM by exhaustive grid search with 5 independent folds... [tuning_intro]
2026-10-08 03:41:49 1 parameter combination x 5 resamples: 5 models total (aarch64-apple-darwin23). [tuning_intro]

⠋ Tuning 1/5
⠙ Tuning 2/5 ETA 0:03
⠹ Tuning 4/5 ETA 0:01
⠸ Tuning 5/5         
            
2026-10-08 03:41:50 ✔ Tuning 5/5 done in 0:01
2026-10-08 03:41:50 Best config to minimize mse: [tune_GridSearch]
                      nrounds: {30} => 30
2026-10-08 03:41:50 </> Tuning done. [tune_GridSearch]
2026-10-08 03:41:50 Training LightGBM Regression with tuned hyperparameters... [train]
2026-10-08 03:41:50 Checking data is ready for training... ✔ [check_supervised]

<Regression>
LightGBM (LightGBM gradient boosting)
⚙ Tuned using exhaustive grid search.

  <Training Regression Metrics>
   MAE: 1.34
   MSE: 3.00
  RMSE: 1.73
    R²: 0.19

  <Test Regression Metrics>
   MAE: 1.27
   MSE: 2.36
  RMSE: 1.54
    R²: 0.15

2026-10-08 03:41:50 Done in 1.88 seconds. [train]

review(mod)

<SupervisedReview>
  LightGBM gradient boosting was used for regression. Hyperparameter tuning was performed using exhaustive grid search. R-squared was 0.19 on the training set and 0.15 on the test set.
  160 training and 40 test cases; 2 predictors (80.0 training cases per predictor).

  Performance
          Training   Test  Training - test
    MAE      1.336  1.273            0.063
    MSE      3.001  2.364            0.637
    RMSE     1.732  1.537            0.195
    R²       0.192  0.150            0.041

  Baseline comparisons
    MSE 2.364 vs 2.903 (training mean); loss reduction 0.539 (95% CI 0.244 to 0.833): better
    MAE 1.273 vs 1.405 (training mean); loss reduction 0.132 (95% CI 0.053 to 0.211): better
    R² 0.150 vs -0.043 (training mean)

  Findings
    * SINGLE_SPLIT (note): Performance was estimated on a single split, with 40 test cases. The test intervals reflect the number of test cases, not how much the estimate would change with a different split.
      Suggestion: Use outer resampling to test on every case and to measure variation between splits; repeat it when cases are few.
    * TEST_PRECISION (note): Test mean squared error is 2.364 (95% CI 1.464 to 3.264), from 40 test cases.
    * BASELINE_MSE (note): Test mean squared error of 2.364 (mean reduction from the baseline's 2.903: 0.539, 95% CI 0.244 to 0.833) is better than predicting the training mean for every case.

  Limitations
    - Whether the model is useful depends on the question it addresses, the costs of different errors, and how its predictions will be used; performance metrics cannot establish this.
    - Preprocessing, decomposition and tuning done by rtemis are fitted on the training cases of each split and only applied to its test cases. Steps taken before the data was passed to rtemis -- selecting or filtering predictors, transforming or removing cases using all the data, or choosing among models by their test performance -- can bias evaluation; this review cannot determine whether that happened.
    - Performance estimates apply to cases from the population the data was drawn from; performance on a different population or setting may differ.

16.2 Configure selection

Learning rate, tree complexity, number of boosting rounds, and regularization interact. tune_over() on tunable properties requests an inner search; nested resampling assesses that search procedure.

LightGBM configurations use bagging_fraction, bagging_freq, and boosting. boosting selects the boosting mode; data_sample_strategy selects the sampling strategy where applicable. Objective-specific parameters apply only to compatible objectives. The setup constructor validates those relationships.

Device and thread allocation come from execution configuration. get_learning_curve() and plot_learning() inspect recorded training progress. A fitted model records fit-resolved settings, including the number of rounds actually used.