6  Fits and model diagnostics

library(rtemis.draw)

Compare predictions with observations, follow training progress, and summarize performance across resamples. These functions take data you already have and build on scatter, line, and box plots.

6.1 True and predicted values

draw_fit() puts true values on the horizontal axis and predictions on the vertical axis. This example fits a linear model of fuel economy:

model <- lm(mpg ~ wt + hp, data = mtcars)
actual <- mtcars[["mpg"]]
predicted <- unname(predict(model))
fit_chart <- draw_fit(actual, predicted, title = "Fuel economy predictions")
fit_chart

The dashed identity line marks perfect agreement. The solid line fits predictions against true values, with a shaded band for the fitted mean (plus or minus 1.96 standard errors). The legend’s R-squared describes this overlay fit. Here the points are in-sample predictions.

6.1.1 Compare training and test samples

Named lists draw separate samples with their own fitted trends. Fit a model on the training rows, then predict both samples:

set.seed(32)
training <- sort(sample(seq_len(nrow(mtcars)), 23))
held_out <- setdiff(seq_len(nrow(mtcars)), training)
model <- lm(mpg ~ wt + hp, data = mtcars[training, ])
truth <- list(Training = mtcars[["mpg"]][training],
              Test = mtcars[["mpg"]][held_out])
predictions <- list(Training = unname(predict(model, mtcars[training, ])),
                    Test = unname(predict(model, mtcars[held_out, ])))
draw_fit(truth, predictions, title = "Training and test predictions")

Click a sample’s legend entry to toggle its points, trend, and band together. Hover points to inspect individual predictions. Use se = FALSE to hide the band, or fit = NULL to show observations with the identity line alone.

6.1.2 Reveal nonlinear patterns

fit = "gam" adds a smooth trend using mgcv. These illustrative predictions contain a systematic nonlinear error that is easier to see with a smooth fit:

set.seed(33)
actual_curve <- seq(0, 10, length.out = 80)
predicted_curve <- actual_curve + 1.4 * sin(actual_curve / 1.5) +
  rnorm(length(actual_curve), sd = 0.35)
draw_fit(actual_curve, predicted_curve, fit = "gam",
         title = "A nonlinear prediction pattern")

6.2 Recorded learning curves

draw_learning_curve() plots recorded loss against training progress. Supply an iteration column and training or validation losses. In this illustrative training log, training loss keeps falling while validation loss eventually rises:

set.seed(34)
epoch <- 1:40
losses <- data.frame(
  iteration = epoch,
  loss_training = 0.2 + 1.8 * exp(-epoch / 9),
  loss_validation = 0.45 + 1.5 * exp(-epoch / 9) +
    0.001 * pmax(epoch - 20, 0)^2 + rnorm(length(epoch), sd = 0.015)
)
selected_epoch <- losses[["iteration"]][which.min(losses[["loss_validation"]])]
learning_chart <- draw_learning_curve(
  losses, unit = "epochs", selected = selected_epoch, zoom = TRUE,
  title = "Training and validation loss"
)
learning_chart

selected highlights a step you have chosen. The marker lies on the training curve when it is available. Use the slider to inspect part of the training history; double-click the plot to restore the full range.

6.3 Metric distributions across resamples

draw_metric() summarizes scores from repeated fits. Supply one row per fold and sample, with fold, split, metric, and value columns. These illustrative RMSE scores compare training and test performance:

set.seed(35)
scores <- data.frame(
  fold = rep(paste0("Fold", 1:12), 2),
  split = rep(c("Training", "Test"), each = 12),
  metric = "rmse",
  value = c(rnorm(12, mean = 2.3, sd = 0.2),
            rnorm(12, mean = 3.1, sd = 0.4))
)
metric_chart <- draw_metric(scores, ylab = "RMSE",
                            title = "Training and test error")
metric_chart

Each point is a fold score; hover to see its fold ID. Boxes summarize variation across folds using quartiles and whiskers. The default whiskers reach the most extreme observations within 1.5 interquartile ranges. Each fold contributes equally.

6.3.1 Change the orientation

A horizontal layout gives sample labels more room. Here full-range whiskers show the entire observed spread:

draw_metric(scores, horizontal = TRUE, xlab = "RMSE", whisker = 0,
            title = "Spread of resample errors")

Use boxpoints = "outliers" or "none" to adjust the point overlay, and metric = "rmse" to select a metric from a table containing several measures.

6.4 Save a figure

Save the chart you want to use in a report:

save_drawing(fit_chart, "predictions.svg", width = 700, height = 650)
save_drawing(learning_chart, "learning.svg", width = 800, height = 500)
save_drawing(metric_chart, "resample-errors.svg", width = 700, height = 500)

See Export for output options and Chart configs for reusable settings. For fitted rtemis objects, use the rtemis-owned plot_true_pred(), plot_learning(), and plot_metric() generics in Plotting model objects.