A supervised dataset contains predictors and an outcome. The last column is the outcome: a numeric column selects regression, and a factor selects classification.
3.1 Fit and assess a classifier
The iris data contain four numeric predictors and a three-level outcome. A stratified split allocates cases to training and test samples.
2026-10-08 02:53:02 Checking data is ready for training... ✔ [check_supervised]
2026-10-08 02:53:02 ▶ [train]2026-10-08 02:53:02 Training set: 120 cases x 4 features. [summarize_supervised]2026-10-08 02:53:02 Test set: 30 cases x 4 features. [summarize_supervised]2026-10-08 02:53:02 //CPU | serial | 1 worker: algorithm 1 thread, tuning 1, outer resampling 1 [msg_resources]2026-10-08 02:53:02 Shared memory (auto): not sharing training data (6.6 kB) -- this run transfers nothing to workers. [report_shared_memory]2026-10-08 02:53:02 Training CART Classification... [train]2026-10-08 02:53:02 Checking data is ready for training... ✔ [check_supervised]
<SupervisedReview>
CART decision tree (rpart) was used for classification. Balanced accuracy was 0.98 in the training set and 0.90 in the test set.
120 training and 30 test cases; 4 predictors (10.0 minority-class training cases per predictor).
Performance
Training Test Training - test
Balanced Accuracy 0.983 0.900 0.083
F1 0.983 0.898 0.086
Accuracy 0.983 0.900 0.083
Baseline comparisons
Accuracy 0.900 (95% CI 0.735 to 0.979) vs 0.333 (most common training class); McNemar p = 1.5e-05: better
Balanced Accuracy 0.900 (95% CI 0.773 to 1.000) vs 0.333 (chance level): better
Findings
* SINGLE_SPLIT (note): Performance was estimated on a single split, with 30 test cases. The test intervals reflect the number of test cases, not how much the estimate would change with a different split.
Suggestion: Use outer resampling to test on every case and to measure variation between splits; repeat it when cases are few.
* TEST_PRECISION (note): Test balanced accuracy is 0.900 (95% CI 0.773 to 1.000), from 30 test cases.
* BASELINE_ACCURACY (note): Test accuracy of 0.900 (17 cases correct only for the model, 0 only for the baseline; exact McNemar p = 1.5e-05) is better than the 0.333 of always predicting 'setosa', the most common training class.
* BASELINE_BALANCED_ACCURACY (note): Test balanced accuracy of 0.900 (95% CI 0.773 to 1.000) is better than the chance level of 0.333.
Limitations
- Whether the model is useful depends on the question it addresses, the costs of different errors, and how its predictions will be used; performance metrics cannot establish this.
- Preprocessing, decomposition and tuning done by rtemis are fitted on the training cases of each split and only applied to its test cases. Steps taken before the data was passed to rtemis -- selecting or filtering predictors, transforming or removing cases using all the data, or choosing among models by their test performance -- can bias evaluation; this review cannot determine whether that happened.
- Performance estimates apply to cases from the population the data was drawn from; performance on a different population or setting may differ.
3.2 Predict
features() selects every column except the last. Prediction input contains the same predictors, in the same order, as training input.
Classification probabilities form a matrix. Multiclass models return one column per class; binary models return one column for the positive class, which is the second outcome factor level.
3.3 Resample
An outer resampling configuration requests separate fits and assessments.
2026-10-08 02:53:03 Checking data is ready for training... ✔ [check_supervised]
2026-10-08 02:53:03 ▶ [train]2026-10-08 02:53:03 Training set: 150 cases x 4 features. [summarize_supervised]2026-10-08 02:53:03 //CPU | serial | 1 worker: algorithm 1 thread, tuning 1, outer resampling 1 [msg_resources]2026-10-08 02:53:03 Shared memory (auto): not sharing training data (7.3 kB) -- this run transfers nothing to workers. [report_shared_memory]2026-10-08 02:53:03 <> Training CART Classification using 3 independent folds... [train]⠋ Outer resamples 1/3⠙ Outer resamples 2/3ETA 0:00⠹ Outer resamples 3/32026-10-08 02:53:03 ✔ Outer resamples 3/3 done in 0:00
2026-10-08 02:53:03 </> Outer resampling done. [train]
<Resampled Classification Model>CART (CART decision tree (rpart))
⟳ Tested using 3 independent folds.
<Resampled Classification Training Metrics>Aggregate Confusion Matrix across resamples.
Predicted Referencesetosa versicolor virginica setosa 100 0 0
versicolor 0 98 2
virginica 0 1 99
Showing mean (sd) across resamples.
Balanced Accuracy: 0.990 (0.010) F1: 0.990 (0.010) Accuracy: 0.990 (0.010)<Resampled Classification Test Metrics>Aggregate Confusion Matrix across resamples.
Predicted Referencesetosa versicolor virginica setosa 50 0 0
versicolor 0 48 2
virginica 0 5 45
Showing mean (sd) across resamples.
Balanced Accuracy: 0.953 (0.030) F1: 0.953 (0.030) Accuracy: 0.953 (0.030)
2026-10-08 02:53:03 Done in 0.29 seconds. [train]
review(resampled)
<SupervisedReview>
CART decision tree (rpart) was used for classification. Mean balanced accuracy was 0.99 in the training set and 0.95 in the test set across 3 independent folds.
3 resamples: 150 out-of-sample predictions of 150 cases, at least 99 training cases per resample; 4 predictors (8.2 minority-class training cases per predictor).
Performance, mean (SD) over 3 resamples
Training Test Training - test
Balanced Accuracy 0.990 (0.010) 0.953 (0.030) 0.036
F1 0.990 (0.010) 0.953 (0.030) 0.037
Accuracy 0.990 (0.010) 0.953 (0.030) 0.036
Baseline comparisons, mean over resamples
Accuracy 0.953 vs 0.333 (most common training class); better in 3 of 3 resamples compared
Balanced Accuracy 0.953 vs 0.333 (chance level); better in 3 of 3 resamples compared
Findings
* FOLD_VARIATION (note): Test balanced accuracy ranged from 0.922 to 0.980 across 3 resamples (mean 0.953, SD 0.030); 3 of 3 resamples in which the metric is defined outperformed their baseline. Resamples share training cases, so their results are not independent, and no confidence interval or test is computed from them.
* FEW_CASES_PER_PREDICTOR (note): There are 8.250 minority-class training cases per learner column (4 columns), fewer than 10. This is a rule of thumb from logistic regression (events per variable), not a validated sample-size requirement for every algorithm.
Suggestion: Judge the model by held-out performance. Consider a regularized or sparse algorithm, or a decomposition step in train(), which is fitted on training cases only.
* PRESELECTION_RISK (note): With few cases per predictor, predictors are often selected before modeling. Selection or filtering of predictors done on all the data before it was passed to train() biases evaluation; this review cannot determine whether that happened. Preprocessing, decomposition and tuning within train() use training cases only.
Suggestion: Avoid selecting predictors on all the data before train(). To reduce the predictors, use a sparse algorithm or a decomposition step in train(); any selection done outside rtemis must use only the training cases of each split.
Limitations
- Whether the model is useful depends on the question it addresses, the costs of different errors, and how its predictions will be used; performance metrics cannot establish this.
- Preprocessing, decomposition and tuning done by rtemis are fitted on the training cases of each split and only applied to its test cases. Steps taken before the data was passed to rtemis -- selecting or filtering predictors, transforming or removing cases using all the data, or choosing among models by their test performance -- can bias evaluation; this review cannot determine whether that happened.
- Performance estimates apply to cases from the population the data was drawn from; performance on a different population or setting may differ.
