Skip to contents

Setup Preprocessor

Source: R/090_Preprocessor.R
setup_Preprocessor.Rd

Creates a PreprocessorConfig object, which can be used in preprocess.

Usage

setup_Preprocessor(
  complete_cases = FALSE,
  remove_features_thres = NULL,
  remove_cases_thres = NULL,
  missingness = FALSE,
  impute = FALSE,
  impute_type = c("missRanger", "micePMM", "meanMode"),
  impute_missRanger_params = list(pmm.k = 3L, maxiter = 10L, num.trees = 500L),
  impute_numeric = c("mean", "median"),
  impute_categorical = "mode",
  impute_numeric_values = NULL,
  impute_level_values = NULL,
  integer2factor = FALSE,
  integer2numeric = FALSE,
  logical2factor = FALSE,
  logical2numeric = FALSE,
  numeric2factor = FALSE,
  numeric2factor_levels = NULL,
  numeric_cut_n = 0L,
  numeric_cut_labels = FALSE,
  numeric_quant_n = 0L,
  numeric_quant_NAonly = FALSE,
  unique_len2factor = 0L,
  character2factor = FALSE,
  factorNA2missing = FALSE,
  factorNA2missing_level = "missing",
  factor2integer = FALSE,
  factor2integer_startat0 = TRUE,
  factor2integer_levels = NULL,
  scale = FALSE,
  center = scale,
  scale_centers = NULL,
  scale_coefficients = NULL,
  remove_constants = FALSE,
  remove_constants_skip_missing = TRUE,
  remove_features = NULL,
  remove_duplicates = FALSE,
  one_hot = FALSE,
  one_hot_levels = NULL,
  add_date_features = FALSE,
  date_features = c("weekday", "month", "year"),
  add_holidays = FALSE,
  holidays = c("USLaborDay", "NewYearsDay", "ChristmasDay"),
  exclude = NULL
)

Arguments

complete_cases

Logical: If TRUE, only retain complete cases (no missing data). Not available inside train: a fitted preprocessor is replayed at predict time, where dropping rows would return fewer predictions than rows.

remove_features_thres

Optional Numeric (0, 1]: Remove features with missing values in >= to this fraction of cases. 1 removes only fully-missing features. Not available inside train: each resample would learn its own set, so the folds would train on different features. Name the columns with remove_features instead, or drop them beforehand with preprocess.

remove_cases_thres

Optional Numeric (0, 1]: Remove cases with >= to this fraction of missing features. 1 removes only fully-missing cases. Not available inside train: a fitted preprocessor is replayed at predict time, where dropping rows would return fewer predictions than rows.

missingness

Logical: If TRUE, generate new boolean columns for each feature with missing values, indicating which cases were missing data.

impute

Logical: If TRUE, impute missing cases. See impute_type.

impute_type

Character {"missRanger", "micePMM", "meanMode"}: Imputation method. "missRanger" fits one random forest per feature with missing values, kept in the Preprocessor and applied to new data by apply_preprocessor. "meanMode" imputes the values that impute_numeric and impute_categorical compute on the training data. "micePMM" imputes by predictive mean matching with 'mice', which imputes only the data it is run on: a Preprocessor using it cannot be applied to new data. See the Imputation section.

impute_missRanger_params

List: Parameters passed to missRanger::missRanger, e.g. "pmm.k", "maxiter", "num.trees". pmm.k greater than 0 results in predictive mean matching. Reduce num.trees for faster imputation especially in large datasets. Set pmm.k = 0 to disable predictive mean matching.

impute_numeric

Character {"mean", "median"}: Statistic imputed into numeric and logical features for impute_type = "meanMode", and into such features with no missing training values for impute_type = "missRanger". Integer and bit64::integer64 features take the statistic rounded half to even, so they remain integer. Logical features take the statistic of their 0/1 values rounded half to even: the more frequent value, FALSE on a tie. Integer features that code categories belong in factors: see integer2factor.

impute_categorical

Character {"mode"}: Statistic imputed into factor and character features for impute_type = "meanMode", and into such features with no missing training values for impute_type = "missRanger". "mode" is the most frequent value. A tie goes to the first tied level of a factor and to the first tied value in Unicode code point order of a character feature.

impute_numeric_values

Optional Named vector: Value imputed into numeric or logical features, keyed by feature name; a logical feature takes 1 as TRUE and 0 as FALSE. Covers every such feature for impute_type = "meanMode" and features with no missing training values for impute_type = "missRanger". Used in place of the value impute_numeric computes. A value for an integer feature must be a whole number, and a value for a logical feature 0 or 1. preprocess stores the values it uses in the Preprocessor, and apply_preprocessor imputes new data with them.

impute_level_values

Optional Named vector: Value imputed into factor or character features, keyed by feature name. Covers every such feature for impute_type = "meanMode" and features with no missing training values for impute_type = "missRanger". Used in place of the value impute_categorical computes. preprocess stores the values it uses in the Preprocessor, and apply_preprocessor imputes new data with them.

integer2factor

Logical: If TRUE, convert all integers to factors. This includes bit64::integer64 columns.

integer2numeric

Logical: If TRUE, convert all integers to numeric (will only work if integer2factor = FALSE). This includes bit64::integer64 columns.

logical2factor

Logical: If TRUE, convert all logical variables to factors.

logical2numeric

Logical: If TRUE, convert all logical variables to numeric.

numeric2factor

Logical: If TRUE, convert all numeric variables to factors.

numeric2factor_levels

Optional Character vector: Will be passed to levels arg of factor() if numeric2factor = TRUE. For advanced/ specific use cases; need to know unique values of numeric vector(s) and given all numeric vars have same unique values.

numeric_cut_n

Integer [0, Inf): If > 0, convert all numeric variables to factors by binning using base::cut with breaks equal to this number.

numeric_cut_labels

Logical: The labels argument of base::cut.

numeric_quant_n

Integer [0, Inf): If > 0, convert all numeric variables to factors by binning using base::cut with breaks equal to this number of quantiles, produced using stats::quantile. 0 disables; 1 is a single break, which bounds no bin, and is rejected.

numeric_quant_NAonly

Logical: If TRUE, only bin numeric variables with missing values.

unique_len2factor

Integer [0, Inf): Convert all variables with less than or equal to this number of unique values to factors. 0 and 1 both disable it: a feature with one unique value is a constant, which remove_constants covers. For example, if binary variables are encoded with 1, 2, you could use unique_len2factor = 2 to convert them to factors.

character2factor

Logical: If TRUE, convert all character variables to factors.

factorNA2missing

Logical: If TRUE, make NA values in factors be of level factorNA2missing_level. In many cases this is the preferred way to handle missing data in categorical variables. Note that since this step is performed before imputation, you can use this option to handle missing data in categorical variables and impute numeric variables in the same preprocess call.

factorNA2missing_level

Character: Name of level if factorNA2missing = TRUE.

factor2integer

Logical: If TRUE, convert all factors to integers.

factor2integer_startat0

Logical: If TRUE, start integer coding at 0.

factor2integer_levels

Optional List: Named list of the form "feature_name" = "levels". Used to code factors identically to the training data when applying factor2integer to validation or test data using Preprocessor. Values whose level is absent take the single index above the known levels.

scale

Logical: If TRUE, scale columns of x.

center

Logical: If TRUE, center columns of x. If unset, follows scale.

scale_centers

Optional Named vector: Centering values for each feature.

scale_coefficients

Optional Named vector: Scaling values for each feature.

remove_constants

Logical: If TRUE, remove constant columns.

remove_constants_skip_missing

Logical: If TRUE, skip missing values, before checking if feature is constant.

remove_features

Optional Character vector: Features to remove.

remove_duplicates

Logical: If TRUE, remove duplicate cases. Not available inside train: a fitted preprocessor is replayed at predict time, where dropping rows would return fewer predictions than rows.

one_hot

Logical: If TRUE, convert all factors using one-hot encoding.

one_hot_levels

Optional List: Named list of the form "feature_name" = "levels". Used when applying one-hot encoding to validation or test data using Preprocessor.

add_date_features

Logical: If TRUE, extract date features from date columns.

date_features

Character {"weekday", "month", "year"} vector: Features to extract from dates.

add_holidays

Logical: If TRUE, extract holidays from date columns.

holidays

Character vector: Holidays to flag when add_holidays is TRUE, named as timeDate holiday functions, e.g. "USLaborDay", "ChristmasDay". See timeDate::listHolidays() for the valid names. Note that unprefixed names are not US holidays: "LaborDay" is May 1.

exclude

Optional Integer vector: Exclude these columns from preprocessing.

Value

PreprocessorConfig object.

Imputation

"meanMode" stores one value per feature, computed on the data preprocess() is given, and apply_preprocessor imputes new data with those values. Each value has its feature's type: an integer feature receives an integer, a logical feature a logical, and a factor one of its levels.

"missRanger" fits one random forest per feature with missing values and keeps the forests, and a copy of the data predict() draws starting values from, in the Preprocessor, which therefore grows with the data. The forests are an R object: a config written to JSON carries the method and its parameters, not the forests. A feature with no missing values in the data has no forest and takes its impute_numeric or impute_categorical value when it is missing in new data. predict() stops on a factor level of an imputed feature that is absent from the data the forests were fitted on.

"micePMM" imputes the data it is run on and fits no model for new data; apply_preprocessor refuses a Preprocessor that uses it.

For preprocessing inside train, see the Imputation section of setup_SupervisedPreprocessor.

Order of Operations

Steps run in this order, and the order is part of what each one does: remove_constants runs before the quantile cut because a constant has no quantiles, factorNA2missing before impute so that missing values in a categorical feature become a level, and one_hot last among the encoders so that it sees every factor the earlier steps created.

Feature creation precedes feature transformation: add_date_features and add_holidays run ahead of the conversions, so the weekday, month and holiday factors they derive are encoded and the year is scaled exactly as any column of the input would be. They run after the case and feature filters, which judge the data as it was given.

  • keep complete cases only

  • set aside excluded columns

  • remove named features

  • remove constants

  • remove duplicates

  • remove cases by missingness threshold

  • remove features by missingness threshold

  • add date features

  • add holidays

  • integer to factor

  • logical to factor

  • numeric to factor

  • character to factor

  • features with <= N unique values to factor

  • integer to numeric

  • logical to numeric

  • cut numeric to n bins

  • cut numeric to n quantiles

  • factor NA to named level

  • factor to integer

  • add missingness column

  • impute

  • scale and/or center

  • one-hot encoding

  • re-add excluded columns, after the preprocessed ones

Author

EDG

Examples

preproc_config <- setup_Preprocessor(factorNA2missing = TRUE)
preproc_config
#> <PreprocessorConfig>
#>                complete_cases: <lgc> FALSE
#>         remove_features_thres: <NUL> NULL
#>            remove_cases_thres: <NUL> NULL
#>                   missingness: <lgc> FALSE
#>                        impute: <lgc> FALSE
#>                   impute_type: <chr> missRanger
#>      impute_missRanger_params: 
#>                                    pmm.k: <int> 3
#>                                  maxiter: <int> 10
#>                                num.trees: <int> 500
#>                impute_numeric: <chr> mean
#>            impute_categorical: <chr> mode
#>         impute_numeric_values: <NUL> NULL
#>           impute_level_values: <NUL> NULL
#>                integer2factor: <lgc> FALSE
#>               integer2numeric: <lgc> FALSE
#>                logical2factor: <lgc> FALSE
#>               logical2numeric: <lgc> FALSE
#>                numeric2factor: <lgc> FALSE
#>         numeric2factor_levels: <NUL> NULL
#>                 numeric_cut_n: <int> 0
#>            numeric_cut_labels: <lgc> FALSE
#>               numeric_quant_n: <int> 0
#>          numeric_quant_NAonly: <lgc> FALSE
#>             unique_len2factor: <int> 0
#>              character2factor: <lgc> FALSE
#>              factorNA2missing: <lgc> TRUE
#>        factorNA2missing_level: <chr> missing
#>                factor2integer: <lgc> FALSE
#>       factor2integer_startat0: <lgc> TRUE
#>         factor2integer_levels: <NUL> NULL
#>                         scale: <lgc> FALSE
#>                        center: <lgc> FALSE
#>                 scale_centers: <NUL> NULL
#>            scale_coefficients: <NUL> NULL
#>              remove_constants: <lgc> FALSE
#> remove_constants_skip_missing: <lgc> TRUE
#>             remove_duplicates: <lgc> FALSE
#>               remove_features: <NUL> NULL
#>                       one_hot: <lgc> FALSE
#>                one_hot_levels: <NUL> NULL
#>             add_date_features: <lgc> FALSE
#>                 date_features: <chr> weekday, month, year
#>                  add_holidays: <lgc> FALSE
#>                      holidays: <chr> USLaborDay, NewYearsDay, ChristmasDay
#>                       exclude: <NUL> NULL