Setup hyperparameters for MLP (Multilayer Perceptron) training.
Usage
setup_MLP(
hidden_units = NULL,
shape = NULL,
shape_layers = NULL,
shape_max_units = NULL,
activation = "relu",
norm = NULL,
residual = FALSE,
dropout = 0,
input_dropout = 0,
weight_decay = 0,
l1_penalty = 0,
embedding_dim = NULL,
embedding_dropout = 0,
optimizer = "adamw",
lr = 0.001,
batch_size = 256L,
max_epochs = 100L,
max_grad_norm = NULL,
ifw = FALSE,
norm_first = FALSE,
bias = TRUE,
embeddings = TRUE,
beta1 = NULL,
beta2 = NULL,
eps = NULL,
momentum = NULL,
lr_scheduler = NULL,
patience = 10L,
loss = NULL,
device = NULL,
seed = NULL,
num_workers = 0L,
drop_last = FALSE
)Arguments
(Tunable) Optional Integer [1, Inf) vector: Units in each hidden layer, one value per layer. NULL generates the widths from the shape settings.
- shape
(Tunable) Optional Character {"funnel", "constant", "triangle", "long_funnel", "diamond", "hexagon", "stairs"}: Profile of the generated hidden layer widths. Ignored when hidden_units is set.
- shape_layers
(Tunable) Optional Integer [1, Inf): Number of hidden layers to generate. Ignored when hidden_units is set.
- shape_max_units
(Tunable) Optional Integer [1, Inf): Widest generated hidden layer. NULL derives it from the encoded input width.
- activation
(Tunable) Character {"relu", "gelu", "silu", "elu", "selu", "leaky_relu", "tanh"}: Activation applied after every hidden layer.
- norm
(Tunable) Optional Character {"batch_norm", "layer_norm"}: Normalization applied in every hidden layer. NULL applies none.
- residual
(Tunable) Logical: If TRUE, add a residual connection around every hidden layer, projected when the layer changes width.
- dropout
(Tunable) Numeric [0, 1): Dropout probability applied after every hidden layer.
- input_dropout
(Tunable) Numeric [0, 1): Dropout probability applied to the encoded input.
- weight_decay
(Tunable) Numeric [0, Inf): L2 penalty, decoupled from the gradient under the adamw optimizer.
- l1_penalty
(Tunable) Numeric [0, Inf): L1 penalty on the linear weights, added to the loss.
- embedding_dim
(Tunable) Optional Integer [1, Inf): Width of every embedding. NULL sizes each from its feature's cardinality.
- embedding_dropout
(Tunable) Numeric [0, 1): Dropout probability applied to the concatenated embeddings.
- optimizer
(Tunable) Character {"adamw", "adam", "sgd", "rmsprop"}: Optimization algorithm.
- lr
(Tunable) Numeric (0, Inf): Learning rate.
- batch_size
(Tunable) Integer [1, Inf): Cases per optimization step.
- max_epochs
(Tunable) Integer [1, Inf): Largest number of passes over the training set.
- max_grad_norm
(Tunable) Optional Numeric (0, Inf): Clip the gradient norm to this value before each step. NULL does not clip.
- ifw
(Tunable) Logical: If TRUE, use Inverse Frequency Weighting in classification.
- norm_first
Logical: If TRUE, normalize before the activation rather than after it.
- bias
Logical: If TRUE, give every hidden layer and the output layer a bias term.
- embeddings
Logical: If TRUE, represent each categorical feature by a learned embedding; if FALSE, one-hot encode them.
- beta1
Optional Numeric [0, 1): Exponential decay rate of the first moment estimate. Applies to the adam and adamw optimizers.
- beta2
Optional Numeric [0, 1): Exponential decay rate of the second moment estimate. Applies to the adam and adamw optimizers.
- eps
Optional Numeric (0, Inf): Term added to the denominator for numerical stability. Applies to the adam, adamw and rmsprop optimizers.
- momentum
Optional Numeric [0, Inf): Momentum factor. Applies to the sgd and rmsprop optimizers.
- lr_scheduler
Optional Character {"step", "cosine_annealing", "one_cycle", "reduce_on_plateau"}: Learning-rate schedule. NULL holds the learning rate fixed.
- patience
Integer [1, Inf): Epochs without validation improvement before stopping early.
- loss
Optional Character {"mse", "l1", "smooth_l1", "cross_entropy"}: Training objective. NULL sets it from the outcome type.
- device
Optional Character {"cpu", "cuda", "mps"}: Compute device.
- seed
Optional Integer: Random seed for weight initialization, dropout and batch shuffling.
- num_workers
Integer [0, Inf): Subprocesses used to load batches.
- drop_last
Logical: If TRUE, drop the last incomplete batch of each training epoch.
Details
A fully connected feedforward network built and trained with torch, for
regression, binary and multiclass classification.
Architecture. Give the hidden layers directly with hidden_units –
c(256L, 128L, 64L) is three layers of those widths – or leave it NULL and
let shape, shape_layers and shape_max_units generate them. Setting both
is an error rather than a silent override. The generated profiles are
funnel (a linear taper from the widest layer down to a third of it),
constant, triangle (a linear rise from the input width), long_funnel,
diamond, hexagon and stairs; the vocabulary is Talos's, by way of
AutoPyTorch. shape_max_units defaults to four times the encoded input
width, clamped to [64, 512] and never below that width.
The tabular deep-learning benchmarks tune and publish constant-width MLPs,
so shape = "constant" is what to compare against even though funnel is
the better default before tuning on the small-n, wide-p data rtemis is usually
pointed at.
Tuning. hidden_units is tunable like any other hyperparameter, with one
architecture per candidate:
setup_MLP(hidden_units = tune_over(c(64L, 32L), c(128L, 64L, 32L))). A
single bare vector there is one architecture, not a set of candidates, and is
rejected as such.
Categorical features are represented by learned embeddings, each sized
min(600, round(1.6 * cardinality^0.56)) unless embedding_dim fixes them
all. embeddings = FALSE one-hot encodes instead. Numeric features are
always centered and scaled – an unscaled network fails quietly rather than
loudly – and the fitted encoder is re-applied at predict time.
Device and reproducibility. device = NULL picks cuda where available
and cpu otherwise, and train names the one it resolved. "mps" is
supported but never chosen automatically: on Apple silicon it is slower than
the CPU for networks of the size tabular data calls for – the matrices are
small enough that dispatch dominates – and a seed governs weight
initialization and batch shuffling there but not dropout, so a seeded
mps fit reproduces exactly until a dropout rate is non-zero. That
combination warns.
Early stopping needs a validation set: pass dat_validation to train,
and the fit keeps the weights of the best validation epoch rather than the
last. Without one it runs the full max_epochs and patience has no effect.
l1_penalty is not weight_decay. weight_decay is the L2 term the
torch optimizer applies, decoupled from the gradient under adamw;
l1_penalty has no torch equivalent and is accumulated over the linear
weights and added to the loss.
Batch normalization and dropout interact badly when combined; both are off by default.
The scheduler takes no settings of its own – each configures itself from
the run's budget: step decays by 0.1 every max_epochs / 3 epochs,
cosine_annealing anneals over max_epochs, one_cycle peaks at lr over
the run's real step count, and reduce_on_plateau decays by 0.1 after half
the early-stopping patience.
get_varimp() returns NULL: a torch MLP has no native importance measure.
Examples
mlp_hyperparams <- setup_MLP(hidden_units = c(64L, 32L), max_epochs = 20L)
mlp_hyperparams
#> <MLPHyperparameters>
#> hyperparameters:
#> hidden_units: <int> 64, 32
#> shape: <NUL> NULL
#> shape_layers: <NUL> NULL
#> shape_max_units: <NUL> NULL
#> activation: <chr> relu
#> norm: <NUL> NULL
#> norm_first: <lgc> FALSE
#> bias: <lgc> TRUE
#> residual: <lgc> FALSE
#> dropout: <nmr> 0.00
#> input_dropout: <nmr> 0.00
#> weight_decay: <nmr> 0.00
#> l1_penalty: <nmr> 0.00
#> embeddings: <lgc> TRUE
#> embedding_dim: <NUL> NULL
#> embedding_dropout: <nmr> 0.00
#> optimizer: <chr> adamw
#> lr: <nmr> 1e-03
#> beta1: <NUL> NULL
#> beta2: <NUL> NULL
#> eps: <NUL> NULL
#> momentum: <NUL> NULL
#> lr_scheduler: <NUL> NULL
#> batch_size: <int> 256
#> max_epochs: <int> 20
#> patience: <int> 10
#> max_grad_norm: <NUL> NULL
#> loss: <NUL> NULL
#> device: <NUL> NULL
#> seed: <NUL> NULL
#> num_workers: <int> 0
#> drop_last: <lgc> FALSE
#> ifw: <lgc> FALSE
#> tunable_hyperparameters: <chr> hidden_units, shape, shape_layers, shape_max_units, activation, norm, residual, dropout, input_dropout, weight_decay, l1_penalty, embedding_dim, embedding_dropout, optimizer, lr, batch_size, max_epochs, max_grad_norm, ifw
#> fixed_hyperparameters: <chr> norm_first, bias, embeddings, beta1, beta2, eps, momentum, lr_scheduler, patience, loss, device, seed, num_workers, drop_last
#> tuned: <int> -1
#> resampled: <int> 0
#> n_workers: <int> 1
#>
#> No search values defined for tunable hyperparameters.