2026-10-08 02:53:20 Checking data is ready for training... ✔ [check_supervised]
2026-10-08 02:53:20 ▶ [train]2026-10-08 02:53:20 Training set: 120 cases x 4 features. [summarize_supervised]2026-10-08 02:53:20 Test set: 30 cases x 4 features. [summarize_supervised]2026-10-08 02:53:20 //CPU | serial | 1 worker: algorithm 1 thread, tuning 1, outer resampling 1 [msg_resources]2026-10-08 02:53:20 Shared memory (auto): not sharing training data (6.6 kB) -- this run transfers nothing to workers. [report_shared_memory]2026-10-08 02:53:20 Training CART Classification... [train]2026-10-08 02:53:20 Checking data is ready for training... ✔ [check_supervised]
<Classification>CART (CART decision tree (rpart))
<Training Classification Metrics> Predicted Referencevirginica other virginica 40 0
other 0 80
Overall Sensitivity 1.000
Specificity 1.000
Balanced Accuracy 1.000
PPV 1.000
NPV 1.000
F1 1.000
Accuracy 1.000
AUC 1.000
Brier Score 0.000
Positive Class virginica<Test Classification Metrics> Predicted Referencevirginica other virginica 9 1
other 2 18
Overall Sensitivity 0.900
Specificity 0.900
Balanced Accuracy 0.900
PPV 0.818
NPV 0.947
F1 0.857
Accuracy 0.900
AUC 0.900
Brier Score 0.100
Positive Class virginica
<SupervisedReview>
CART decision tree (rpart) was used for classification. Balanced accuracy was 1.00 in the training set and 0.90 in the test set.
120 training and 30 test cases; 4 predictors (10.0 minority-class training cases per predictor).
Performance
Training Test Training - test
Sensitivity 1.000 0.900 0.100
Specificity 1.000 0.900 0.100
Balanced Accuracy 1.000 0.900 0.100
PPV 1.000 0.818 0.182
NPV 1.000 0.947 0.053
F1 1.000 0.857 0.143
Accuracy 1.000 0.900 0.100
AUC 1.000 0.900 0.100
Brier Score 0.000 0.100 -0.100
Baseline comparisons
Accuracy 0.900 (95% CI 0.735 to 0.979) vs 0.667 (most common training class); McNemar p = 0.065: indistinguishable
Balanced Accuracy 0.900 (95% CI 0.762 to 1.000) vs 0.500 (chance level): better
AUC 0.900 (95% CI 0.781 to 1.000) vs 0.500 (chance level): better
Brier Score 0.100 vs 0.222 (training proportion as probability); loss reduction 0.122 (95% CI -0.006 to 0.251): indistinguishable
Findings
* SINGLE_SPLIT (note): Performance was estimated on a single split, with 30 test cases. The test intervals reflect the number of test cases, not how much the estimate would change with a different split.
Suggestion: Use outer resampling to test on every case and to measure variation between splits; repeat it when cases are few.
* TEST_PRECISION (note): Test balanced accuracy is 0.900 (95% CI 0.762 to 1.000), from 30 test cases.
* BASELINE_ACCURACY (warning): Test accuracy of 0.900 (9 cases correct only for the model, 2 only for the baseline; exact McNemar p = 0.065) does not provide clear evidence of a difference from the 0.667 of always predicting 'other', the most common training class; limited precision can cause this.
* BASELINE_BALANCED_ACCURACY (note): Test balanced accuracy of 0.900 (95% CI 0.762 to 1.000) is better than the chance level of 0.500.
* BASELINE_AUC (note): Test AUC of 0.900 (95% CI 0.781 to 1.000) is better than the chance level of 0.500.
* BASELINE_BRIER (warning): Test Brier score of 0.100 (mean reduction from the baseline's 0.222: 0.122, 95% CI -0.006 to 0.251) does not provide clear evidence of a difference from predicting the training proportion of 'virginica' for every case; limited precision can cause this.
Limitations
- Whether the model is useful depends on the question it addresses, the costs of different errors, and how its predictions will be used; performance metrics cannot establish this.
- Preprocessing, decomposition and tuning done by rtemis are fitted on the training cases of each split and only applied to its test cases. Steps taken before the data was passed to rtemis -- selecting or filtering predictors, transforming or removing cases using all the data, or choosing among models by their test performance -- can bias evaluation; this review cannot determine whether that happened.
- Performance estimates apply to cases from the population the data was drawn from; performance on a different population or setting may differ.
plot_true_pred(mod)
plot_roc(mod)
Binary probabilities form a one-column matrix for the positive class. Multiclass probabilities have one column per outcome level, in training level order. Assessment includes class-specific performance and aggregate metrics. Accuracy, balanced accuracy, probability loss, and AUC quantify different aspects of predictions.
2026-10-08 02:53:21 Checking data is ready for training... ✔ [check_supervised]
2026-10-08 02:53:21 ▶ [train]2026-10-08 02:53:21 Training set: 150 cases x 4 features. [summarize_supervised]2026-10-08 02:53:21 //CPU | serial | 1 worker: algorithm 1 thread, tuning 1, outer resampling 1 [msg_resources]2026-10-08 02:53:21 Shared memory (auto): not sharing training data (7.3 kB) -- this run transfers nothing to workers. [report_shared_memory]2026-10-08 02:53:21 <> Training CART Classification using 3 independent folds... [train]⠋ Outer resamples 1/3⠙ Outer resamples 3/3ETA 0:00⠹ Outer resamples 3/32026-10-08 02:53:22 ✔ Outer resamples 3/3 done in 0:00
2026-10-08 02:53:22 </> Outer resampling done. [train]
<Resampled Classification Model>CART (CART decision tree (rpart))
⟳ Tested using 3 independent folds.
<Resampled Classification Training Metrics>Aggregate Confusion Matrix across resamples.
Predicted Referencesetosa versicolor virginica setosa 100 0 0
versicolor 0 98 2
virginica 0 1 99
Showing mean (sd) across resamples.
Balanced Accuracy: 0.990 (0.010) F1: 0.990 (0.010) Accuracy: 0.990 (0.010)<Resampled Classification Test Metrics>Aggregate Confusion Matrix across resamples.
Predicted Referencesetosa versicolor virginica setosa 50 0 0
versicolor 0 48 2
virginica 0 5 45
Showing mean (sd) across resamples.
Balanced Accuracy: 0.953 (0.030) F1: 0.953 (0.030) Accuracy: 0.953 (0.030)
2026-10-08 02:53:22 Done in 0.29 seconds. [train]
review(resampled)
<SupervisedReview>
CART decision tree (rpart) was used for classification. Mean balanced accuracy was 0.99 in the training set and 0.95 in the test set across 3 independent folds.
3 resamples: 150 out-of-sample predictions of 150 cases, at least 99 training cases per resample; 4 predictors (8.2 minority-class training cases per predictor).
Performance, mean (SD) over 3 resamples
Training Test Training - test
Balanced Accuracy 0.990 (0.010) 0.953 (0.030) 0.036
F1 0.990 (0.010) 0.953 (0.030) 0.037
Accuracy 0.990 (0.010) 0.953 (0.030) 0.036
Baseline comparisons, mean over resamples
Accuracy 0.953 vs 0.333 (most common training class); better in 3 of 3 resamples compared
Balanced Accuracy 0.953 vs 0.333 (chance level); better in 3 of 3 resamples compared
Findings
* FOLD_VARIATION (note): Test balanced accuracy ranged from 0.922 to 0.980 across 3 resamples (mean 0.953, SD 0.030); 3 of 3 resamples in which the metric is defined outperformed their baseline. Resamples share training cases, so their results are not independent, and no confidence interval or test is computed from them.
* FEW_CASES_PER_PREDICTOR (note): There are 8.250 minority-class training cases per learner column (4 columns), fewer than 10. This is a rule of thumb from logistic regression (events per variable), not a validated sample-size requirement for every algorithm.
Suggestion: Judge the model by held-out performance. Consider a regularized or sparse algorithm, or a decomposition step in train(), which is fitted on training cases only.
* PRESELECTION_RISK (note): With few cases per predictor, predictors are often selected before modeling. Selection or filtering of predictors done on all the data before it was passed to train() biases evaluation; this review cannot determine whether that happened. Preprocessing, decomposition and tuning within train() use training cases only.
Suggestion: Avoid selecting predictors on all the data before train(). To reduce the predictors, use a sparse algorithm or a decomposition step in train(); any selection done outside rtemis must use only the training cases of each split.
Limitations
- Whether the model is useful depends on the question it addresses, the costs of different errors, and how its predictions will be used; performance metrics cannot establish this.
- Preprocessing, decomposition and tuning done by rtemis are fitted on the training cases of each split and only applied to its test cases. Steps taken before the data was passed to rtemis -- selecting or filtering predictors, transforming or removing cases using all the data, or choosing among models by their test performance -- can bias evaluation; this review cannot determine whether that happened.
- Performance estimates apply to cases from the population the data was drawn from; performance on a different population or setting may differ.
predict(resampled, features(iris)[1:3, ], type ="avg")
