Assess a trained model: whether its evaluation can be trusted given the sample size and the number of predictors, whether it performs better than a baseline that ignores the predictors, and whether it overfits. The review is deterministic: it draws no random numbers.
Arguments
- x
SupervisedorSupervisedResobject: A trained model, as returned by train.- confidence_level
Optional Numeric (0, 1): Confidence level of every interval. NULL uses 0.95.
- min_cases_per_predictor
Optional Numeric (0, Inf): Training cases per learner column – minority-class cases for classification – below which the review notes a shortfall. NULL uses 10, an events-per-variable rule of thumb from logistic regression (see References).
- ...
Not used.
Details
The review holds every value it rests on, so a reader can judge for
themselves: @sample (sample sizes, input predictors and the columns the
learner received), @class_counts, @performance (one row per metric:
training, test, training minus test, the spread over resamples, and the
test interval or pooled value), @baseline (one row per comparison with a
reference that ignores the predictors, naming the reference and the method)
and @tuning (tuned hyperparameters against their search range). Its
findings state what it observes. Printing shows a summary: one row per
metric – mean (SD) over resamples for a resampled model – the baseline
comparisons, the findings and the limitations.
Single split. The test cases are independent of the fitted model, so
the review computes confidence intervals and two-sided comparisons at
confidence_level. Accuracy is compared with always predicting the most
common training class by the exact McNemar test, which pairs the two
predictors' results case by case. Balanced accuracy and AUC are compared
with their chance levels (1/K, 0.5) through their intervals:
per-class binomial variances for balanced accuracy, the DeLong method for
AUC. The Brier score, MSE and MAE are compared with a constant baseline
(training proportion, training mean) through a paired t interval of the
per-case loss reduction; the skill score is reported as a point estimate.
An interval that cannot be computed – too few cases, a class absent from
the test set, an AUC of 0 or 1 – is left unset and no verdict is made from
it. A training value of the headline metric (balanced accuracy, or mean
squared error for regression) outside its test interval is reported as a
diagnostic sign of possible overfitting; it is not a test of the
train-test difference.
Resampled models. Resamples share training cases, so their test results are dependent, and the review makes no interval or test from them. Every metric is reported as its mean and standard deviation over the outer resamples, each baseline is fit to the training data of its resample, and the review counts the resamples in which the model beat its baseline. When every case is tested once, as with k-fold resampling, pooled out-of-sample values are reported as descriptions for metrics that average over cases; AUC is not pooled, since it would rank scores from different fitted models together.
A tuned hyperparameter selected at the edge of the values searched is reported, since a better value may lie beyond it.
The cases-per-predictor check counts the columns the learner received, after preprocessing and decomposition. Its threshold is a rule of thumb from logistic regression (events per variable; see References), reported for context.
Preprocessing, decomposition and tuning inside train() are fitted on the
training cases of each split and only applied to its test cases. Steps
taken before the data is passed to train() – selecting predictors or
transforming cases using all the data, or choosing among models by their
test performance – can bias evaluation, and the fitted model cannot show
whether they happened. Performance metrics also cannot establish whether a
model is useful. Every review states both.
Print the result to read it; write_text writes it as Markdown.
References
Clopper CJ, Pearson ES (1934). The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4), 404-413.
DeLong ER, DeLong DM, Clarke-Pearson DL (1988). Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics, 44(3), 837-845.
McNemar Q (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2), 153-157.
Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR (1996). A simulation study of the number of events per variable in logistic regression analysis. Journal of Clinical Epidemiology, 49(12), 1373-1379.
Examples
idx <- c(1:40, 51:90, 101:140)
mod <- train(
iris[idx, ],
dat_test = iris[-idx, ],
hyperparameters = setup_CART(),
verbosity = 0L
)
review(mod)#> <SupervisedReview>
#> CART decision tree (rpart) was used for classification. Balanced accuracy was 0.99 in the training set and 1.00 in the test set.
#> 120 training and 30 test cases; 4 predictors (10.0 minority-class training cases per predictor).
#>
#> Performance
#> Training Test Training - test
#> Balanced Accuracy 0.992 1.000 -0.008
#> F1 0.992 1.000 -0.008
#> Accuracy 0.992 1.000 -0.008
#>
#> Baseline comparisons
#> Accuracy 1.000 (95% CI 0.884 to 1.000) vs 0.333 (most common training class); McNemar p = 1.9e-06: better
#> Balanced Accuracy 1.000 (95% CI 0.901 to 1.000) vs 0.333 (chance level): better
#>
#> Findings
#> * SINGLE_SPLIT (note): Performance was estimated on a single split, with 30 test cases. The test intervals reflect the number of test cases, not how much the estimate would change with a different split.
#> Suggestion: Use outer resampling to test on every case and to measure variation between splits; repeat it when cases are few.
#> * TEST_PRECISION (note): Test balanced accuracy is 1.000 (95% CI 0.901 to 1.000), from 30 test cases.
#> * BASELINE_ACCURACY (note): Test accuracy of 1.000 (20 cases correct only for the model, 0 only for the baseline; exact McNemar p = 1.9e-06) is better than the 0.333 of always predicting 'setosa', the most common training class.
#> * BASELINE_BALANCED_ACCURACY (note): Test balanced accuracy of 1.000 (95% CI 0.901 to 1.000) is better than the chance level of 0.333.
#>
#> Limitations
#> - Whether the model is useful depends on the question it addresses, the costs of different errors, and how its predictions will be used; performance metrics cannot establish this.
#> - Preprocessing, decomposition and tuning done by rtemis are fitted on the training cases of each split and only applied to its test cases. Steps taken before the data was passed to rtemis -- selecting or filtering predictors, transforming or removing cases using all the data, or choosing among models by their test performance -- can bias evaluation; this review cannot determine whether that happened.
#> - Performance estimates apply to cases from the population the data was drawn from; performance on a different population or setting may differ.
# Resampled
mod_res <- train(
iris,
hyperparameters = setup_CART(),
outer_resampling_config = setup_KFold(5L),
verbosity = 0L
)
review(mod_res)#> <SupervisedReview>
#> CART decision tree (rpart) was used for classification. Mean balanced accuracy was 0.99 in the training set and 0.96 in the test set across 5 independent folds.
#> 5 resamples: 150 out-of-sample predictions of 150 cases, at least 120 training cases per resample; 4 predictors (10.0 minority-class training cases per predictor).
#>
#> Performance, mean (SD) over 5 resamples
#> Training Test Training - test
#> Balanced Accuracy 0.992 (0.000) 0.960 (0.028) 0.032
#> F1 0.992 (0.000) 0.960 (0.028) 0.032
#> Accuracy 0.992 (0.000) 0.960 (0.028) 0.032
#>
#> Baseline comparisons, mean over resamples
#> Accuracy 0.960 vs 0.333 (most common training class); better in 5 of 5 resamples compared
#> Balanced Accuracy 0.960 vs 0.333 (chance level); better in 5 of 5 resamples compared
#>
#> Findings
#> * FOLD_VARIATION (note): Test balanced accuracy ranged from 0.933 to 1.000 across 5 resamples (mean 0.960, SD 0.028); 5 of 5 resamples in which the metric is defined outperformed their baseline. Resamples share training cases, so their results are not independent, and no confidence interval or test is computed from them.
#>
#> Limitations
#> - Whether the model is useful depends on the question it addresses, the costs of different errors, and how its predictions will be used; performance metrics cannot establish this.
#> - Preprocessing, decomposition and tuning done by rtemis are fitted on the training cases of each split and only applied to its test cases. Steps taken before the data was passed to rtemis -- selecting or filtering predictors, transforming or removing cases using all the data, or choosing among models by their test performance -- can bias evaluation; this review cannot determine whether that happened.
#> - Performance estimates apply to cases from the population the data was drawn from; performance on a different population or setting may differ.