Class imbalance occurs when outcome levels have different frequencies. Accuracy alone can conceal low sensitivity for a minority class. Assessment therefore includes class counts, per-class metrics, and the intended costs of errors.
20.1 Weighting
Case weights are supplied through train(weights = ...). They contain one finite, nonnegative value per training case and have positive mean. Support and interpretation depend on the learner. An algorithm’s ifw setting requests inverse-frequency weighting where supported; it is an explicit modeling choice.
2026-10-08 02:53:52 Checking data is ready for training... ✔ [check_supervised]
2026-10-08 02:53:53 ▶ [train]2026-10-08 02:53:53 Training set: 150 cases x 4 features. [summarize_supervised]2026-10-08 02:53:53 //CPU | serial | 1 worker: algorithm 1 thread, tuning 1, outer resampling 1 [msg_resources]2026-10-08 02:53:53 Shared memory (auto): not sharing training data (7.2 kB) -- this run transfers nothing to workers. [report_shared_memory]2026-10-08 02:53:53 <> Training CART Classification using 3 independent folds... [train]⠋ Outer resamples 1/3⠙ Outer resamples 2/3ETA 0:01⠹ Outer resamples 3/32026-10-08 02:53:54 ✔ Outer resamples 3/3 done in 0:01
2026-10-08 02:53:54 </> Outer resampling done. [train]
<Resampled Classification Model>CART (CART decision tree (rpart))
⟳ Tested using 3 independent folds.
<Resampled Classification Training Metrics>Aggregate Confusion Matrix across resamples.
Predicted Referencevirginica other virginica 100 0
other 2 198
Showing mean (sd) across resamples.
Sensitivity: 1.000 (0.000) Specificity: 0.990 (0.009) Balanced Accuracy: 0.995 (4.3e-03) PPV: 0.981 (0.017) NPV: 1.000 (0.000) F1: 0.990 (0.008) Accuracy: 0.993 (0.006) AUC: 0.995 (4.1e-03) Brier Score: 0.006 (0.006)<Resampled Classification Test Metrics>Aggregate Confusion Matrix across resamples.
Predicted Referencevirginica other virginica 46 4
other 3 97
Showing mean (sd) across resamples.
Sensitivity: 0.920 (0.089) Specificity: 0.970 (0.029) Balanced Accuracy: 0.945 (0.030) PPV: 0.944 (0.053) NPV: 0.962 (0.042) F1: 0.928 (0.022) Accuracy: 0.953 (0.012) AUC: 0.946 (0.030) Brier Score: 0.047 (0.012)
2026-10-08 02:53:54 Done in 1.15 seconds. [train]
review(mod)
<SupervisedReview>
CART decision tree (rpart) was used for classification. Mean balanced accuracy was 1.00 in the training set and 0.95 in the test set across 3 independent folds.
3 resamples: 150 out-of-sample predictions of 150 cases, at least 99 training cases per resample; 4 predictors (8.2 minority-class training cases per predictor).
Performance, mean (SD) over 3 resamples
Training Test Training - test
Sensitivity 1.000 (0.000) 0.920 (0.089) 0.080
Specificity 0.990 (0.009) 0.970 (0.029) 0.020
Balanced Accuracy 0.995 (0.004) 0.945 (0.030) 0.050
PPV 0.981 (0.017) 0.944 (0.053) 0.037
NPV 1.000 (0.000) 0.962 (0.042) 0.038
F1 0.990 (0.008) 0.928 (0.022) 0.062
Accuracy 0.993 (0.006) 0.953 (0.012) 0.040
AUC 0.995 (0.004) 0.946 (0.030) 0.050
Brier Score 0.006 (0.006) 0.047 (0.012) -0.040
Baseline comparisons, mean over resamples
Accuracy 0.953 vs 0.667 (most common training class); better in 3 of 3 resamples compared
Balanced Accuracy 0.945 vs 0.500 (chance level); better in 3 of 3 resamples compared
AUC 0.946 vs 0.500 (chance level); better in 3 of 3 resamples compared
Brier Score 0.047 vs 0.222 (training proportion as probability); better in 3 of 3 resamples compared
Findings
* FOLD_VARIATION (note): Test balanced accuracy ranged from 0.912 to 0.971 across 3 resamples (mean 0.945, SD 0.030); 3 of 3 resamples in which the metric is defined outperformed their baseline. Resamples share training cases, so their results are not independent, and no confidence interval or test is computed from them.
* FEW_CASES_PER_PREDICTOR (note): There are 8.250 minority-class training cases per learner column (4 columns), fewer than 10. This is a rule of thumb from logistic regression (events per variable), not a validated sample-size requirement for every algorithm.
Suggestion: Judge the model by held-out performance. Consider a regularized or sparse algorithm, or a decomposition step in train(), which is fitted on training cases only.
* PRESELECTION_RISK (note): With few cases per predictor, predictors are often selected before modeling. Selection or filtering of predictors done on all the data before it was passed to train() biases evaluation; this review cannot determine whether that happened. Preprocessing, decomposition and tuning within train() use training cases only.
Suggestion: Avoid selecting predictors on all the data before train(). To reduce the predictors, use a sparse algorithm or a decomposition step in train(); any selection done outside rtemis must use only the training cases of each split.
Limitations
- Whether the model is useful depends on the question it addresses, the costs of different errors, and how its predictions will be used; performance metrics cannot establish this.
- Preprocessing, decomposition and tuning done by rtemis are fitted on the training cases of each split and only applied to its test cases. Steps taken before the data was passed to rtemis -- selecting or filtering predictors, transforming or removing cases using all the data, or choosing among models by their test performance -- can bias evaluation; this review cannot determine whether that happened.
- Performance estimates apply to cases from the population the data was drawn from; performance on a different population or setting may differ.
Ranger additionally accepts class weights through setup_Ranger(class_weights = ...); named weights are matched to outcome levels. Case weighting, class weighting, and changes to the sample represent distinct interventions and can affect probability calibration.
20.2 Assessment
Balanced accuracy is the mean of the class-specific recalls:
For binary classification, it is the mean of sensitivity and specificity. F1 combines precision and recall. AUC measures discrimination across thresholds; probability losses assess the probabilities themselves.
Use stratification to represent the outcome levels in resamples where sample size permits it. Evaluate the selected procedure at the target population’s class prevalence, and assess calibration separately.
