Skip to contents

Setup LINADForest Hyperparameters

Source: R/070_Hyperparameters.R
setup_LINADForest.Rd

Setup hyperparameters for LINADForest: a bagged ensemble of Linear Additive Trees. Each tree is fitted on a bootstrap sample of the cases, restricted to mtry_tree of the features, searching mtry_split of them at each split, and chooses its own number of leaves on the cases its bootstrap left out.

Usage

setup_LINADForest(
  n_trees = 50L,
  mtry_split = NULL,
  mtry_tree = NULL,
  max_leaves = 20L,
  min_cases_split = 2L,
  min_cases_leaf = 1L,
  min_cases_node_model = NULL,
  nvmax = NULL,
  forward_stop = NULL,
  lambda = NULL,
  alpha = NULL,
  learning_rate = 1,
  root_nvmax = NULL,
  root_lambda = NULL,
  root_alpha = NULL,
  root_learning_rate = 1,
  n_cuts = NULL,
  gamma = 0.1,
  split_binning = NULL,
  split_bin_type = "frequency",
  node_test = NULL,
  split_criterion = NULL,
  line_search = "expansion",
  constant_rule = "closed_form",
  ifw = FALSE,
  patience = NULL,
  force_max_leaves = FALSE,
  smooth_validation_curve = NULL,
  node_model = "forward",
  root_model = NULL,
  split_search = "stump",
  split_features = NULL,
  linear_features = NULL,
  global_features = NULL,
  line_search_max = 1000,
  root_forward_stop = NULL,
  root_node_test = NULL
)

Arguments

n_trees

(Tunable) Integer [1, Inf): Number of trees in the forest.

mtry_split

(Tunable) Optional Integer [1, Inf): Features sampled at each split search. NULL scans every feature the tree holds.

mtry_tree

(Tunable) Optional Integer [1, Inf): Features available to a whole tree. NULL gives every tree every feature.

max_leaves

(Tunable) Integer [1, Inf): Largest number of terminal nodes to grow in each tree.

min_cases_split

(Tunable) Integer [2, Inf): Fewest cases a node may hold and still be considered for splitting.

min_cases_leaf

(Tunable) Integer [1, Inf): Fewest cases a split must leave on each side.

min_cases_node_model

(Tunable) Optional Integer [1, Inf): Fewest cases needed to fit a linear model at a node. Applies only when node_model fits one. NULL uses 10.

nvmax

(Tunable) Optional Integer [1, Inf): Most terms forward selection may add beside the intercept. Applies only when node_model is "forward". NULL uses 3.

forward_stop

(Tunable) Optional Character {"bic", "aic", "none"}: Cost a term must earn to be added by forward selection. Applies only when node_model is "forward". NULL uses "bic".

lambda

(Tunable) Optional Numeric [0, Inf): L2 penalty on the node models. Applies only when node_model is "forward", "ridge" or "elasticnet". NULL uses 0.05.

alpha

(Tunable) Optional Numeric [0, 1]: Elastic-net mixing, 0 ridge to 1 lasso. Applies only when node_model is "elasticnet". NULL uses 1.

learning_rate

(Tunable) Numeric (0, 1]: Shrinkage applied to every functional update.

root_nvmax

(Tunable) Optional Integer [1, Inf): nvmax for the root model. NULL uses nvmax.

root_lambda

(Tunable) Optional Numeric [0, Inf): lambda for the root model. NULL uses lambda.

root_alpha

(Tunable) Optional Numeric [0, 1]: alpha for the root model. NULL uses alpha.

root_learning_rate

(Tunable) Numeric [0, 1]: Shrinkage applied to the root model's slopes.

n_cuts

(Tunable) Optional Integer [2, Inf): Cut points tried per feature. Applies only when split_search is "exhaustive". NULL uses 20.

gamma

(Tunable) Numeric [0, 1]: Weight a case retains in the branch it does not belong to. 0 is a hard partition.

split_binning

(Tunable) Optional Integer [2, Inf): Discretize each numeric feature into this many bins and consider only bin boundaries as splits.

split_bin_type

(Tunable) Character {"frequency", "width"}: How bin edges are placed.

node_test

(Tunable) Optional Character {"none", "aic", "bic"}: Cost a node's slopes must earn over the constant alone. Applies only when node_model is "ridge" or "elasticnet"; forward selection uses forward_stop. NULL uses "none".

split_criterion

(Tunable) Optional Character {"mean", "linear"}: What the stump search scores a candidate side by. Applies only when split_search is "stump". NULL uses "mean".

(Tunable) Character {"expansion", "child", "none"}: Scope of the Newton step for each update.

constant_rule

(Tunable) Character {"closed_form", "least_squares"}: How a node's constant is computed.

ifw

(Tunable) Logical: If TRUE, use Inverse Frequency Weighting in classification.

patience

(Tunable) Optional Integer [1, Inf): Expansions without validation improvement before growth stops. Requires a validation set. NULL disables patience-based early stopping.

force_max_leaves

Logical: If TRUE, grow every tree to max_leaves instead of selecting a size on its out-of-bag cases.

smooth_validation_curve

Optional Logical: If TRUE, smooth the out-of-bag curve before reading its minimum. Applies only when force_max_leaves is FALSE. NULL does not smooth.

node_model

Character {"forward", "ridge", "elasticnet", "constant"}: Model fitted at each node.

root_model

Optional Character {"forward", "ridge", "elasticnet", "constant"}: Model fitted at each tree's root. NULL uses node_model.

Character {"stump", "exhaustive"}: How a split is chosen.

split_features

Optional Character vector: Features that may define a split. NULL imposes no constraint, so every feature may. mtry_split samples within this set.

linear_features

Optional Character vector: Features that get a slope in the node models. NULL imposes no constraint, so every feature does.

global_features

Optional Character vector: Linear features whose slope is shared by every leaf of a tree. Must be a subset of linear_features where that is set. NULL imposes no constraint.

line_search_max

Numeric (0, Inf): Largest absolute step the line search may take.

root_forward_stop

(Tunable) Optional Character {"bic", "aic", "none"}: forward_stop for the root model. NULL uses forward_stop, or "bic" where that is NULL. Applies only when the root model is "forward".

root_node_test

(Tunable) Optional Character {"none", "aic", "bic"}: node_test for the root model. NULL uses node_test, or "none" where that is NULL. Applies only when the root model is "ridge" or "elasticnet".

Value

LINADForestHyperparameters object.

Details

LINADForest is designed to be used untuned. Every tree-level hyperparameter of setup_LINAD is available here and tunable, but the defaults are the configuration the ensemble was first evaluated with, and the one structural choice that usually needs tuning – how large each tree should be – is made per tree from data.

How the trees differ

Three sources of variation, each acting at a different level:

The bootstrap

Each tree is fitted on n cases drawn with replacement, so roughly 37% of cases are out-of-bag for any given tree.

mtry_tree

Each tree is restricted to a random subset of the features, for its splits and its node models alike. A tree's coefficients then cover only the features it held.

mtry_split

Each split search sees a fresh sample of the features the tree holds, which decorrelates the trees further. If that sample finds no split worth making, the node is searched again over every feature, so mtry_split changes which split is made and never whether a node can split at all.

Both mtry_* default to NULL, meaning every feature, which is the configuration the ensemble was first evaluated with.

Choosing each tree's size

A LINAD tree picks its number of leaves on held-out data, as gradient boosting picks its number of trees. A bootstrap produces held-out data for free, so each tree selects its own size on its own out-of-bag cases and no validation set has to be supplied or set aside. force_max_leaves = TRUE turns the selection off and grows every tree to max_leaves, which is what the ensemble's first evaluation did.

The same out-of-bag predictions give the fitted forest an error estimate with no resampling, reported by print().

Defaults that differ from a single tree

learning_rate defaults to 1, where a single tree uses 0.1. A lone LINAD tree shrinks each node update because nothing else controls its variance; in a forest the averaging does that, so each tree should be a strong learner – the same reasoning that leaves a random forest's trees unpruned.

Workers

LINADForest parallelizes across trees, by process rather than by thread. n_workers_algorithm therefore has an effect only when the run is not already dispatching: inside a parallel outer fold the forest runs its trees sequentially, since a fold may not dispatch again. For a threaded backend such as Ranger the same setting means threads within each fold.

References

Wager S, Hastie T, Efron B (2014). Confidence intervals for random forests: the jackknife and the infinitesimal jackknife. Journal of Machine Learning Research, 15, 1625-1651.

Author

EDG

Examples

linadforest_hyperparams <- setup_LINADForest(n_trees = 25L)
linadforest_hyperparams
#> <LINADForestHyperparameters>
#>         hyperparameters: 
#>                                     n_trees: <int> 25
#>                                  max_leaves: <int> 20
#>                            force_max_leaves: <lgc> FALSE
#>                             min_cases_split: <int> 2
#>                              min_cases_leaf: <int> 1
#>                                  node_model: <chr> forward
#>                               learning_rate: <nmr> 1.00
#>                          root_learning_rate: <nmr> 1.00
#>                                split_search: <chr> stump
#>                              split_bin_type: <chr> frequency
#>                                       gamma: <nmr> 0.10
#>                                 line_search: <chr> expansion
#>                             line_search_max: <nmr> 1000.00
#>                               constant_rule: <chr> closed_form
#>                                         ifw: <lgc> FALSE
#>                   unset: <chr> mtry_split, mtry_tree, patience, smooth_validation_curve, min_cases_node_model, nvmax, (+16 more)
#> tunable_hyperparameters: <int> 28
#>   fixed_hyperparameters: <int> 9
#>                   tuned: <int> -1
#>               resampled: <int> 0
#>               n_workers: <int> 1
#> 
#>   No search values defined for tunable hyperparameters.