library(rtemis) .:rtemis 1.3.5 🌊 aarch64-apple-darwin23library(data.table)
Attaching package: 'data.table'The following object is masked from 'package:base':
%notin%11 Classification
11.1 Setup
11.1.1 Packages
11.1.2 Data
For this example, we shall use the BreastCancer dataset from the mlbench package:
data(BreastCancer, package = "mlbench")In rtemis, the last column is the outcome variable.
We optionally convert the dataset to a data.table:
train() supports data.frame, data.table, or tibble inputs.
dat <- as.data.table(BreastCancer)
dat Id Cl.thickness Cell.size Cell.shape Marg.adhesion Epith.c.size
<char> <ord> <ord> <ord> <ord> <ord>
1: 1000025 5 1 1 1 2
2: 1002945 5 4 4 5 7
3: 1015425 3 1 1 1 2
4: 1016277 6 8 8 1 3
5: 1017023 4 1 1 3 2
---
695: 776715 3 1 1 1 3
696: 841769 2 1 1 1 2
697: 888820 5 10 10 3 7
698: 897471 4 8 6 4 3
699: 897471 4 8 8 5 4
Bare.nuclei Bl.cromatin Normal.nucleoli Mitoses Class
<fctr> <fctr> <fctr> <fctr> <fctr>
1: 1 3 1 1 benign
2: 10 3 2 1 benign
3: 2 3 1 1 benign
4: 4 3 7 1 benign
5: 1 3 1 1 benign
---
695: 2 1 1 1 benign
696: 1 1 1 1 benign
697: 3 8 10 2 malignant
698: 4 10 6 1 malignant
699: 5 10 4 1 malignantAlso optionally, we clean the dataset, in this case to replace periods with underscores in column names:
dt_set_clean_all(dat)
datdt_* functions operate on data.table objects. dt_set_* functions modify their input in-place.
Class is already the last column, otherwise we could use set_outcome() to move it.
For classification, the outcome variable must be a factor. For binary classification, the second factor level is considered the positive case.
The first column, “Id”, is not a predictor, so we remove it:
dat[, Id := NULL]11.2 Check data
check_data(dat) dat: A data.table with 699 rows and 10 columns.
Data types
* 0 numeric features
* 0 integer features
* 10 factors, of which 5 are ordered
* 0 character features
* 0 date features
Issues
* 0 constant features
* 236 duplicate cases
* 1 feature includes 'NA' values; 16 'NA' values total
* 1 factor
Recommendations
* Consider removing the duplicate cases.
* Consider using algorithms that can handle missingness or imputing missing values.11.3 Train a single model
11.3.1 Resample
res <- resample(dat, setup_Resampler(1L, "StratSub"))res<Resampler>
type: StratSub
resamples:
Subsample_1: 1, 2, 3, 4, ...
config:
<StratSubConfig>
n_resamples: 1
train_p: 0.75
stratify_var: NULL
strat_n_bins: 2
id_strat: NULL
seed: NULLdat_training <- dat[res$Subsample_1, ]
dat_test <- dat[-res$Subsample_1, ]
size(dat_training)523 x 10 size(dat_test)176 x 10 11.3.2 Train model
Using LightRF as an example to train a random forest model:
mod_lightrf <- train(
dat_training,
dat_test = dat_test,
hyperparameters = setup_LightRF()
)<Classification>
LightRF (LightGBM Random Forest)
<Training Classification Metrics>
Predicted
Reference malignant benign
malignant 162 18
benign 11 332
Overall
Sensitivity 0.900
Specificity 0.968
Balanced Accuracy 0.934
Ppv 0.936
Npv 0.949
F1 0.918
Accuracy 0.945
Auc 0.983
Brier Score 0.074
Positive Class malignant
<Test Classification Metrics>
Predicted
Reference malignant benign
malignant 56 5
benign 2 113
Overall
Sensitivity 0.918
Specificity 0.983
Balanced Accuracy 0.950
Ppv 0.966
Npv 0.958
F1 0.941
Accuracy 0.960
Auc 0.995
Brier Score 0.069
Positive Class malignant11.3.3 Describe model
describe(mod_lightrf)LightGBM Random Forest was used for classification. Balanced accuracy was 0.93 in the training set and 0.95 in the test set. 11.3.4 Plot Confusion Matrix
plot_true_pred(mod_lightrf)11.3.5 Plot ROC Curve
plot_roc(mod_lightrf)11.3.6 Present model
present() combines describe() and plot() or plot_roc() (default):
present(mod_lightrf)LightGBM Random Forest was used for classification. Balanced accuracy was 0.93 in the training set and 0.95 in the test set. type defaults to "ROC", but can be set to "confusion" to show training and test confusion matrices side by side:
present(mod_lightrf, type = "confusion")LightGBM Random Forest was used for classification. Balanced accuracy was 0.93 in the training set and 0.95 in the test set. 11.3.7 Plot Variable Importance
plot_varimp(mod_lightrf)11.3.8 Predict on new data
For this example, we’ll use the dat_test we created. Remember that if the dataset includes the outcome variable, it must be removed before predicting. You can either delete the column, or use indexing to exclude it. rtemis includes a convenience function features() which excludes the last column of data.frames, data.tables, or tibbles:
head(features(dat_test)) Cl_thickness Cell_size Cell_shape Marg_adhesion Epith_c_size Bare_nuclei
<ord> <ord> <ord> <ord> <ord> <fctr>
1: 4 1 1 3 2 1
2: 8 10 10 8 7 10
3: 2 1 2 1 2 1
4: 2 1 1 1 2 1
5: 4 2 1 1 2 1
6: 8 7 5 10 7 9
Bl_cromatin Normal_nucleoli Mitoses
<fctr> <fctr> <fctr>
1: 3 1 1
2: 9 7 1
3: 3 1 1
4: 1 1 5
5: 2 1 1
6: 5 5 4In binary classification, the output of predict() is a vector of probabilities for the positive class:
pred <- predict(mod_lightrf, features(dat_test))head(pred) malignant
[1,] 0.1582693
[2,] 0.7405205
[3,] 0.1529617
[4,] 0.1514779
[5,] 0.1693546
[6,] 0.738735811.4 Train on multiple training/test resamples
To train on multiple resamples, we use the outer_resampling_config argument:
resmod_lightrf <- train(
dat,
hyperparameters = setup_LightRF(),
outer_resampling_config = setup_Resampler(n_resamples = 10L, type = "KFold")
)<Resampled Classification Model>
LightRF (LightGBM Random Forest)
⟳ Tested using 10 independent folds.
<Resampled Classification Training Metrics>
Aggregate Confusion Matrix across resamples.
Predicted
Reference malignant benign
malignant 2026 143
benign 130 3992
Showing mean (sd) across resamples.
Sensitivity: 0.934 (0.014)
Specificity: 0.968 (3.4e-03)
Balanced Accuracy: 0.951 (0.006)
Ppv: 0.940 (0.006)
Npv: 0.965 (0.007)
F1: 0.937 (0.006)
Accuracy: 0.957 (3.7e-03)
Auc: 0.987 (1.2e-03)
Brier Score: 0.067 (1.2e-03)
<Resampled Classification Test Metrics>
Aggregate Confusion Matrix across resamples.
Predicted
Reference malignant benign
malignant 225 16
benign 15 443
Showing mean (sd) across resamples.
Sensitivity: 0.934 (0.066)
Specificity: 0.967 (0.026)
Balanced Accuracy: 0.950 (0.031)
Ppv: 0.940 (0.043)
Npv: 0.966 (0.033)
F1: 0.935 (0.036)
Accuracy: 0.956 (0.024)
Auc: 0.985 (0.010)
Brier Score: 0.068 (0.009)Now, train() produced a ClassificationRes object:
class(resmod_lightrf)[1] "rtemis::ClassificationRes" "rtemis::SupervisedRes"
[3] "S7_object" 11.4.1 Describe
describe(resmod_lightrf)LightGBM Random Forest was used for classification. Mean balanced accuracy was 0.95 in the training set and 0.95 in the test set across 10 independent folds. 11.4.2 Plot
The plot() method for ClassificationRes objects plots boxplots of the training and test set metrics:
plot_true_pred(resmod_lightrf)11.4.3 Present
The present() method for ClassificationRes objects combines the describe() and plot() methods:
present(resmod_lightrf)LightGBM Random Forest was used for classification. Mean balanced accuracy was 0.95 in the training set and 0.95 in the test set across 10 independent folds. 
