Train Supervised Learning Models

Preprocess, tune, train, and test supervised learning models using nested resampling in a single call.

Usage

train(
  x,
  dat_validation = NULL,
  dat_test = NULL,
  weights = NULL,
  preprocessor_config = NULL,
  decomposition_config = NULL,
  hyperparameters = NULL,
  tuner_config = NULL,
  outer_resampling_config = NULL,
  execution_config = setup_FutureExecution(),
  question = NULL,
  outdir = NULL,
  preflight = FALSE,
  verbosity = 1L,
  ...
)

Arguments

x: Tabular data, i.e. data.frame, data.table, or tbl_df (tibble): Training set data.
dat_validation: Tabular data: Validation set data.
dat_test: Tabular data: Test set data.
weights: Optional vector of case weights.
preprocessor_config: Optional SupervisedPreprocessorConfig object: Setup using setup_SupervisedPreprocessor.
decomposition_config: Optional DecompositionConfig object: Setup using a decomposition setup_* function.
hyperparameters: Hyperparameters object: Setup using one of setup_* functions.
tuner_config: TunerConfig object: Setup using setup_GridSearch.
outer_resampling_config: Optional ResamplerConfig object: Setup using one of the setup_* resampler functions, e.g. setup_KFold. This defines the outer resampling method, i.e. the splitting into training and test sets for the purpose of assessing model performance. If NULL, no outer resampling is performed, in which case you might want to use a dat_test dataset to assess model performance on a single test set.
execution_config: ExecutionConfig object: Setup using setup_SerialExecution, setup_FutureExecution or setup_MiraiExecution. The function you call selects the backend; each sets the number of workers per dispatch level, and setup_FutureExecution additionally sets the future plan.
question: Optional character string defining the question that the model is trying to answer.
outdir: Character, optional: String defining the output directory. The fitted model is saved there as train_<algorithm>.rds, alongside a run record, train_<algorithm>.record.json, stating what the run actually did: every value resolved, where each came from, and what produced it. See write_record.
preflight: Logical: If TRUE, check the configuration against x with validate_config before training and stop on any finding of severity "error". Findings of severity "warning" are reported and training continues. Defaults to FALSE, except for a SuperConfigTabular, where it defaults to TRUE: that type binds its data in memory and is what a server hands a run it accepted on a client's behalf, so the check is that type's default. Pass FALSE explicitly to skip it.
verbosity: Integer: Verbosity level.
...: Not used.

Value

Object of class Regression(Supervised), RegressionRes(SupervisedRes), Classification(Supervised), or ClassificationRes(SupervisedRes).

Details

Online book & documentation

See docs.rtemis.org/r for detailed documentation.

Preprocessing

There are many different stages at which preprocessing could be applied, when running a supervised learning pipeline with nested resampling. Some operations are best done before passing data to train():

Duplicate rows should be removed before resampling, so that duplicates don't end up in different resamples, e.g. one in training and one in test.
Constant columns should be removed before resampling. A column may appear constant in a small resample, even if it is not constant in the full dataset. Removing it inconsistently will throw an error during prediction.
All data-dependent preprocessing steps need to be performed on training data only and applied on validation and test data, e.g. scaling, centering, imputation.

User-defined preprocessing through preprocessor_config is applied on training set data, the learned parameters are stored in the returned Supervised or SupervisedRes object, and the preprocessing is applied on validation and test data.

For missing values, prefer an algorithm that accepts them, such as LightGBM, LightRF, Ranger or CART. For an algorithm that requires complete data, impute with setup_SupervisedPreprocessor(impute = TRUE), which uses mean and mode imputation by default. missRanger imputation is fitted on every training set, including every tuning resample, and multiplies the memory and computation of a run; see the Imputation section of setup_SupervisedPreprocessor.

Binary Classification

For binary classification, the outcome should be a factor where the 2nd level corresponds to the positive class.

Resampling

Note that you should not use an outer resampling method with replacement if you will also be using an inner resampling (for tuning). The duplicated cases from the outer resampling may appear both in the training and test sets of the inner resamples, leading to underestimated test error.

Reproducibility

If using outer resampling, you can set a seed when defining outer_resampling_config, e.g.

outer_resampling_config = setup_KFold(n_resamples = 10L, seed = 2026L)

If using tuning with inner resampling, you can set a seed when defining tuner_config, e.g.

tuner_config = setup_GridSearch(
  resampler_config = setup_KFold(n_resamples = 5L, seed = 2027L)
)

Parallelization

There are three levels of parallelization that may be used during training:

Algorithm training (e.g. a parallelized learner like LightGBM)
Tuning (where the full hyperparameter combination x inner resample cross product is processed in parallel, not just the resamples)
Outer resampling (where multiple outer resamples can be processed in parallel)

train() assigns all workers to exactly one of these levels, so that no two levels compete for the same cores. The level is chosen by the first of these that applies: the algorithm parallelizes itself, tuning is needed, outer resampling is set. A run that needs none of them is sequential.

The workers, backend, and future plan all come from execution_config; see setup_FutureExecution.

Reproducibility under parallelization

execution_config carries the master seed for the run's computation RNG, from which one independent substream per outer resample is derived and assigned by fold index. Results therefore do not depend on the backend, the number of workers, or the order folds happened to finish in: a parallel run reproduces its sequential counterpart exactly.

Author

EDG

Examples

# \donttest{
iris_c_lightRF <- train(
   iris,
   hyperparameters = setup_LightRF(),
   outer_resampling_config = setup_KFold(n_resamples = 3L)
)

#> Error in UseMethod("train"): no applicable method for 'train' applied to an object of class "data.frame"

# }