Setup hyperparameters for MLP (Multilayer Perceptron) training.
Usage
setup_MLP(
hidden_units = NULL,
shape = NULL,
shape_layers = NULL,
shape_max_units = NULL,
activation = "relu",
norm = NULL,
residual = FALSE,
dropout = 0,
input_dropout = 0,
weight_decay = 0,
l1_penalty = 0,
embedding_dim = NULL,
embedding_dropout = 0,
optimizer = "adamw",
lr = 0.001,
batch_size = 256L,
max_epochs = 100L,
max_grad_norm = NULL,
ifw = FALSE,
norm_first = FALSE,
bias = TRUE,
embeddings = TRUE,
beta1 = NULL,
beta2 = NULL,
eps = NULL,
momentum = NULL,
lr_scheduler = NULL,
patience = 10L,
loss = NULL,
seed = NULL,
num_workers = 0L,
drop_last = FALSE
)Arguments
(Tunable) Optional Integer [1, Inf) vector: Units in each hidden layer, one value per layer. NULL generates the widths from the shape settings.
- shape
(Tunable) Optional Character {"funnel", "constant", "triangle", "long_funnel", "diamond", "hexagon", "stairs"}: Profile of the generated hidden layer widths. Ignored when
hidden_unitsis set. NULL uses "funnel" wherehidden_unitsis NULL.- shape_layers
(Tunable) Optional Integer [1, Inf): Number of hidden layers to generate. Ignored when
hidden_unitsis set. NULL uses 3 wherehidden_unitsis NULL.- shape_max_units
(Tunable) Optional Integer [1, Inf): Widest generated hidden layer. Ignored when
hidden_unitsis set. NULL derives it from the encoded input width wherehidden_unitsis NULL.- activation
(Tunable) Character {"relu", "gelu", "silu", "elu", "selu", "leaky_relu", "tanh"}: Activation applied after every hidden layer.
- norm
(Tunable) Optional Character {"batch_norm", "layer_norm"}: Normalization applied in every hidden layer. NULL applies none.
- residual
(Tunable) Logical: If TRUE, add a residual connection around every hidden layer, projected when the layer changes width.
- dropout
(Tunable) Numeric [0, 1): Dropout probability applied after every hidden layer.
- input_dropout
(Tunable) Numeric [0, 1): Dropout probability applied to the encoded input.
- weight_decay
(Tunable) Numeric [0, Inf): L2 penalty, decoupled from the gradient under the adamw optimizer.
- l1_penalty
(Tunable) Numeric [0, Inf): L1 penalty on the linear weights, added to the loss.
- embedding_dim
(Tunable) Optional Integer [1, Inf): Width of every embedding. NULL sizes each from its feature's cardinality.
- embedding_dropout
(Tunable) Numeric [0, 1): Dropout probability applied to the concatenated embeddings.
- optimizer
(Tunable) Character {"adamw", "adam", "sgd", "rmsprop"}: Optimization algorithm.
- lr
(Tunable) Numeric (0, Inf): Learning rate.
- batch_size
(Tunable) Integer [1, Inf): Cases per optimization step.
- max_epochs
(Tunable) Integer [1, Inf): Largest number of passes over the training set.
- max_grad_norm
(Tunable) Optional Numeric (0, Inf): Clip the gradient norm to this value before each step. NULL does not clip.
- ifw
(Tunable) Logical: If TRUE, use Inverse Frequency Weighting in classification.
- norm_first
Logical: If TRUE, normalize before the activation rather than after it.
- bias
Logical: If TRUE, give every hidden layer and the output layer a bias term.
- embeddings
Logical: If TRUE, represent each categorical feature by a learned embedding; if FALSE, one-hot encode them.
- beta1
Optional Numeric [0, 1): Exponential decay rate of the first moment estimate. Applies to the adam and adamw optimizers.
- beta2
Optional Numeric [0, 1): Exponential decay rate of the second moment estimate. Applies to the adam and adamw optimizers.
- eps
Optional Numeric (0, Inf): Term added to the denominator for numerical stability. Applies to the adam, adamw and rmsprop optimizers.
- momentum
Optional Numeric [0, Inf): Momentum factor. Applies to the sgd and rmsprop optimizers.
- lr_scheduler
Optional Character {"step", "cosine_annealing", "one_cycle", "reduce_on_plateau"}: Learning-rate schedule. NULL holds the learning rate fixed.
- patience
Integer [1, Inf): Epochs without validation improvement before stopping early.
- loss
Optional Character {"mse", "l1", "smooth_l1", "cross_entropy"}: Training objective. NULL sets it from the outcome type.
- seed
Optional Integer: Random seed for weight initialization, dropout and batch shuffling.
- num_workers
Integer [0, Inf): Subprocesses used to load batches.
- drop_last
Logical: If TRUE, drop the last incomplete batch of each training epoch.
Details
A fully connected feedforward network built and trained with torch, for
regression, binary and multiclass classification.
Architecture. Give the hidden layers directly with hidden_units –
c(256L, 128L, 64L) is three layers of those widths – or leave it NULL and
let shape, shape_layers and shape_max_units generate them. Setting both
is an error. The generated profiles are
funnel (a linear taper from the widest layer down to a third of it),
constant, triangle (a linear rise from the input width), long_funnel,
diamond, hexagon and stairs; the vocabulary is Talos's, by way of
AutoPyTorch. shape_max_units defaults to four times the encoded input
width, clamped to [64, 512] and never below that width.
The tabular deep-learning benchmarks tune and publish constant-width MLPs,
so shape = "constant" is what to compare against even though funnel is
the better default before tuning on the small-n, wide-p data rtemis is usually
pointed at.
Tuning. hidden_units is tunable like any other hyperparameter, with one
architecture per candidate:
setup_MLP(hidden_units = tune_over(c(64L, 32L), c(128L, 64L, 32L))). A
single bare vector there is one architecture, not a set of candidates, and is
rejected as such.
Categorical features are represented by learned embeddings, each sized
min(600, round(1.6 * cardinality^0.56)) unless embedding_dim fixes them
all. embeddings = FALSE one-hot encodes them. Numeric features are always
centered and scaled, and the fitted encoder is re-applied at predict time.
Device and reproducibility. The device is the execution config's: see the
device argument of setup_FutureExecution. Unset picks cuda where
available and cpu otherwise, and train names the one it resolved. "mps",
the Apple silicon GPU, runs only when named. In rtemis benchmarks on an Apple M5
the CPU was faster for every network narrower than about 1024 units or
trained in batches smaller than about 2048 cases, often by 2 to 5 times; with
both, and thousands of features, "mps" was faster by up to about 25%. It
also slows down across many fits in one R session. Try it for a wide network
on wide data. A seed governs weight initialization and batch shuffling on
mps but not dropout, so a seeded mps fit reproduces exactly until a
dropout rate is non-zero. That combination warns.
Early stopping needs a validation set: pass dat_validation to train,
and the fit keeps the weights of the best validation epoch rather than the
last. Without one it runs the full max_epochs and patience has no effect.
l1_penalty is not weight_decay. weight_decay is the L2 term the
torch optimizer applies, decoupled from the gradient under adamw;
l1_penalty has no torch equivalent and is accumulated over the linear
weights and added to the loss.
Batch normalization and dropout interact badly when combined; both are off by default.
The scheduler takes no settings of its own – each configures itself from
the run's budget: step decays by 0.1 every max_epochs / 3 epochs,
cosine_annealing anneals over max_epochs, one_cycle peaks at lr over
the run's real step count, and reduce_on_plateau decays by 0.1 after half
the early-stopping patience.
get_varimp() returns NULL: a torch MLP has no native importance measure.
Examples
mlp_hyperparams <- setup_MLP(hidden_units = c(64L, 32L), max_epochs = 20L)
mlp_hyperparams#> <MLPHyperparameters>
#> hyperparameters:
#> hidden_units: <int> 64, 32
#> activation: <chr> relu
#> norm_first: <lgc> FALSE
#> bias: <lgc> TRUE
#> residual: <lgc> FALSE
#> dropout: <nmr> 0.00
#> input_dropout: <nmr> 0.00
#> weight_decay: <nmr> 0.00
#> l1_penalty: <nmr> 0.00
#> embeddings: <lgc> TRUE
#> embedding_dropout: <nmr> 0.00
#> optimizer: <chr> adamw
#> lr: <nmr> 1e-03
#> batch_size: <int> 256
#> max_epochs: <int> 20
#> patience: <int> 10
#> num_workers: <int> 0
#> drop_last: <lgc> FALSE
#> ifw: <lgc> FALSE
#> unset: <chr> shape, shape_layers, shape_max_units, norm, embedding_dim, beta1, (+7 more)
#> tunable_hyperparameters: <int> 19
#> fixed_hyperparameters: <int> 13
#> tuned: <int> -1
#> resampled: <int> 0
#> n_workers: <int> 1
#>
#> No search values defined for tunable hyperparameters.