library(rtemis) .:rtemis 1.0.0 🌊 aarch64-apple-darwin20library(data.table)11 Classification
11.1 Setup
11.1.1 Packages
11.1.2 Data
For this example, we shall use the BreastCancer dataset from the mlbench package:
data(BreastCancer, package = "mlbench")In rtemis, the last column is the outcome variable.
We optionally convert the dataset to a data.table:
train() supports data.frame, data.table, or tibble inputs.
dat <- as.data.table(BreastCancer)
dat Id Cl.thickness Cell.size Cell.shape Marg.adhesion Epith.c.size
<char> <ord> <ord> <ord> <ord> <ord>
1: 1000025 5 1 1 1 2
2: 1002945 5 4 4 5 7
3: 1015425 3 1 1 1 2
4: 1016277 6 8 8 1 3
5: 1017023 4 1 1 3 2
---
695: 776715 3 1 1 1 3
696: 841769 2 1 1 1 2
697: 888820 5 10 10 3 7
698: 897471 4 8 6 4 3
699: 897471 4 8 8 5 4
Bare.nuclei Bl.cromatin Normal.nucleoli Mitoses Class
<fctr> <fctr> <fctr> <fctr> <fctr>
1: 1 3 1 1 benign
2: 10 3 2 1 benign
3: 2 3 1 1 benign
4: 4 3 7 1 benign
5: 1 3 1 1 benign
---
695: 2 1 1 1 benign
696: 1 1 1 1 benign
697: 3 8 10 2 malignant
698: 4 10 6 1 malignant
699: 5 10 4 1 malignantAlso optionally, we clean the dataset, in this case to replace periods with underscores in column names:
dt_set_clean_all(dat)
datdt_* functions operate on data.table objects. dt_set_* functions modify their input in-place.
Class is already the last column, otherwise we could use set_outcome() to move it.
For classification, the outcome variable must be a factor. For binary classification, the second factor level is considered the positive case.
The first column, “Id”, is not a predictor, so we remove it:
dat[, Id := NULL]11.2 Check data
check_data(dat) dat: A data.table with 699 rows and 10 columns.
Data types
* 0 numeric features
* 0 integer features
* 10 factors, of which 5 are ordered
* 0 character features
* 0 date features
Issues
* 0 constant features
* 236 duplicate cases
* 1 feature includes 'NA' values; 16 'NA' values total
* 1 factor
Recommendations
* Consider removing the duplicate cases.
* Consider using algorithms that can handle missingness or imputing missing values.
11.3 Train a single model
11.3.1 Resample
res <- resample(dat, setup_Resampler(1L, "StratSub"))res<Resampler>
type: StratSub
resamples:
Subsample_1: 1, 2, 5, 6...
config:
<StratSubConfig>
n: 1
train_p: 0.75
stratify_var: NULL
strat_n_bins: 2
id_strat: NULL
seed: NULL
dat_training <- dat[res$Subsample_1, ]
dat_test <- dat[-res$Subsample_1, ]
size(dat_training)523 x 10
size(dat_test)176 x 10
11.3.2 Train model
Using LightRF as an example to train a random forest model:
mod_lightrf <- train(
dat_training,
dat_test = dat_test,
algorithm = "LightRF"
)<Classification>
LightRF (LightGBM Random Forest)
<Training Classification Metrics>
Predicted
Reference malignant benign
malignant 168 12
benign 9 334
Overall
Sensitivity 0.933
Specificity 0.974
Balanced_Accuracy 0.954
PPV 0.949
NPV 0.965
F1 0.941
Accuracy 0.960
AUC 0.989
Brier_Score 0.070
Positive Class malignant
<Test Classification Metrics>
Predicted
Reference malignant benign
malignant 57 4
benign 4 111
Overall
Sensitivity 0.934
Specificity 0.965
Balanced_Accuracy 0.950
PPV 0.934
NPV 0.965
F1 0.934
Accuracy 0.955
AUC 0.979
Brier_Score 0.077
Positive Class malignant
11.3.3 Describe model
describe(mod_lightrf)LightGBM Random Forest was used for classification.
Balanced accuracy was 0.95 on the training set and 0.95 in the test set.11.3.4 Plot Confusion Matrix
plot_true_pred(mod_lightrf)11.3.5 Plot ROC Curve
plot_roc(mod_lightrf)11.3.6 Present model
present() combines describe() and plot() or plot_roc() (default):
present(mod_lightrf)LightGBM Random Forest was used for classification.
Balanced accuracy was 0.95 on the training set and 0.95 in the test set.type defaults to "ROC", but can be set to "confusion" to show training and test confusion matrices side by side:
present(mod_lightrf, type = "confusion")LightGBM Random Forest was used for classification.
Balanced accuracy was 0.95 on the training set and 0.95 in the test set.11.3.7 Plot Variable Importance
plot_varimp(mod_lightrf)11.3.8 Predict on new data
For this example, we’ll use the dat_test we created. Remember that if the dataset includes the outcome variable, it must be removed before predicting. You can either delete the column, or use indexing to exclude it. rtemis includes a convenience function features() which excludes the last column of data.frames, data.tables, or tibbles:
head(features(dat_test)) Cl_thickness Cell_size Cell_shape Marg_adhesion Epith_c_size Bare_nuclei
<ord> <ord> <ord> <ord> <ord> <fctr>
1: 3 1 1 1 2 2
2: 6 8 8 1 3 4
3: 1 1 1 1 1 1
4: 5 3 3 3 2 3
5: 8 7 5 10 7 9
6: 10 5 5 3 6 7
Bl_cromatin Normal_nucleoli Mitoses
<fctr> <fctr> <fctr>
1: 3 1 1
2: 3 7 1
3: 3 1 1
4: 4 4 1
5: 5 5 4
6: 7 10 1In binary classification, the output of predict() is a vector of probabilities for the positive class:
pred <- predict(mod_lightrf, features(dat_test))head(pred)[1] 0.2261529 0.6323387 0.1499187 0.5185643 0.7358744 0.735956411.4 Train on multiple training/test resamples
To train on multiple resamples, we use the outer_resampling_config argument:
resmod_lightrf <- train(
dat_training,
algorithm = "LightRF",
outer_resampling_config = setup_Resampler(n_resamples = 10L, type = "KFold")
)<Resampled Classification Model>
LightRF (LightGBM Random Forest)
⟳ Tested using 10 independent folds.
<Resampled Classification Training Metrics>
Showing mean (sd) across resamples.
Sensitivity: 0.914 (0.019)
Specificity: 0.974 (4.4e-03)
Balanced_Accuracy: 0.944 (0.011)
PPV: 0.948 (0.009)
NPV: 0.956 (0.009)
F1: 0.930 (0.012)
Accuracy: 0.953 (0.008)
AUC: 0.989 (1.7e-03)
Brier_Score: 0.074 (1.4e-03)
<Resampled Classification Test Metrics>
Showing mean (sd) across resamples.
Sensitivity: 0.894 (0.041)
Specificity: 0.974 (0.040)
Balanced_Accuracy: 0.934 (0.023)
PPV: 0.953 (0.069)
NPV: 0.947 (0.019)
F1: 0.921 (0.035)
Accuracy: 0.946 (0.025)
AUC: 0.990 (0.014)
Brier_Score: 0.075 (0.008)Now, train() produced a ClassificationRes object:
class(resmod_lightrf)[1] "rtemis::ClassificationRes" "rtemis::SupervisedRes"
[3] "S7_object" 11.4.1 Describe
describe(resmod_lightrf)LightGBM Random Forest was used for classification. Mean balanced accuracy was 0.94 in the training set and 0.93 in the test set across 10 independent folds. 11.4.2 Plot
The plot() method for ClassificationRes objects plots boxplots of the training and test set metrics:
plot_true_pred(resmod_lightrf)11.4.3 Present
The present() method for ClassificationRes objects combines the describe() and plot() methods:
present(resmod_lightrf)LightGBM Random Forest was used for classification. Mean balanced accuracy was 0.94 in the training set and 0.93 in the test set across 10 independent folds. 