A fully connected autoencoder built and trained with torch: an encoder maps
the features to k components through the hidden layers, and a decoder
mirroring it maps the components back.
Usage
setup_Autoencoder(
k = 2L,
hidden_units = NULL,
activation = "relu",
norm = NULL,
dropout = 0,
input_dropout = 0,
weight_decay = 0,
input_noise = 0,
loss = "mse",
optimizer = "adamw",
lr = 0.001,
beta1 = NULL,
beta2 = NULL,
eps = NULL,
momentum = NULL,
lr_scheduler = NULL,
batch_size = NULL,
max_epochs = 100L,
patience = 10L,
max_grad_norm = NULL,
validation_fraction = 0.1,
features = NULL
)Arguments
- k
Integer [1, Inf): Number of components, the width of the latent layer.
Optional Integer [1, Inf) vector: Units in each hidden layer of the encoder, input side first.
- activation
Character {"relu", "gelu", "silu", "elu", "selu", "leaky_relu", "tanh"}: Activation applied after every hidden layer.
- norm
Optional Character {"batch_norm", "layer_norm"}: Normalization applied in every hidden layer. NULL applies none.
- dropout
Numeric [0, 1): Dropout probability applied after every hidden layer.
- input_dropout
Numeric [0, 1): Probability of zeroing each standardized input during training.
- weight_decay
Numeric [0, Inf): L2 penalty, decoupled from the gradient under the adamw optimizer.
- input_noise
Numeric [0, Inf): Standard deviation of the Gaussian noise added to the standardized inputs during training.
- loss
Character {"mse", "l1", "smooth_l1"}: Reconstruction loss.
- optimizer
Character {"adamw", "adam", "sgd", "rmsprop"}: Optimization algorithm.
- lr
Numeric (0, Inf): Learning rate.
- beta1
Optional Numeric [0, 1): Exponential decay rate of the first moment estimate. Applies to the adam and adamw optimizers.
- beta2
Optional Numeric [0, 1): Exponential decay rate of the second moment estimate. Applies to the adam and adamw optimizers.
- eps
Optional Numeric (0, Inf): Term added to the denominator for numerical stability. Applies to the adam, adamw and rmsprop optimizers.
- momentum
Optional Numeric [0, Inf): Momentum factor. Applies to the sgd and rmsprop optimizers.
- lr_scheduler
Optional Character {"step", "cosine_annealing", "one_cycle", "reduce_on_plateau"}: Learning-rate schedule. NULL holds the learning rate fixed.
- batch_size
Optional Integer [1, Inf): Cases per optimization step.
- max_epochs
Integer [1, Inf): Largest number of passes over the training cases.
- patience
Integer [1, Inf): Epochs without improvement of the held-out loss before stopping early.
- max_grad_norm
Optional Numeric (0, Inf): Clip the gradient norm to this value before each step. NULL does not clip.
- validation_fraction
Numeric [0, 1): Fraction of the cases held out for early stopping.
- features
Optional Character vector: Names of at least 2 distinct feature columns to decompose.
NULLdecomposes all numeric features.
Details
Architecture. hidden_units gives the encoder's hidden widths from the
input side, e.g. c(64L, 16L); the decoder uses them in reverse, and both
the latent layer and the output layer are linear. NULL uses one hidden
layer of width round(sqrt(p * k)) for p features, the geometric mean of
the input and latent widths, held between 32 and 512 and never below k.
batch_size = NULL uses a tenth of the training cases, held between 16 and
256, so small data still gets several optimization steps per epoch. The fit
reports and records both values it used.
Inputs are always centered and scaled with the training cases' statistics (a constant feature is only centered). reconstruct undoes this, so reconstructions and the reconstruction metrics are in the units of the data, comparable with any other decomposition's.
Denoising. input_noise adds Gaussian noise of that standard deviation to
the standardized inputs, and input_dropout zeroes each input with that
probability, during training only. The model learns to reconstruct the
uncorrupted input, which regularizes the components; apply_decomp and
reconstruct never corrupt.
Early stopping. validation_fraction of the cases, drawn at random, are
held out; training stops after patience epochs without improvement of
their reconstruction loss and keeps the weights of the best epoch. The
components are then computed for every case, held-out ones included.
validation_fraction = 0 trains for max_epochs.
Reproducibility, threads and device come from the execution config
passed to decomp: its seed seeds torch as well as R, its algorithm
threads are libtorch's thread count, and its device is the compute device.
Unset, the device is cuda where available and cpu otherwise. "mps",
the Apple silicon GPU, runs only when named. In rtemis benchmarks on an Apple
M5 the CPU fitted faster than "mps" in every configuration measured, up to
50,000 cases, 5,000 features, 1,024 hidden units and batches of 2,048, by
1.1 to 3.7 times; "mps" also slows down across many fits in one R
session. A seed does not reach dropout on "mps", so a seeded fit there
reproduces only while dropout and input_dropout are 0, and warns
otherwise.
The optimizer, schedule and early-stopping settings mean what they do in setup_MLP.
Examples
ae_config <- setup_Autoencoder(k = 2L, hidden_units = 8L, input_noise = 0.1)
ae_config#> <Autoencoder DecompositionConfig>
#> k: <int> 2
#> hidden_units: <int> 8
#> activation: <chr> relu
#> norm: <NUL> NULL
#> dropout: <nmr> 0.00
#> input_dropout: <nmr> 0.00
#> weight_decay: <nmr> 0.00
#> input_noise: <nmr> 0.10
#> loss: <chr> mse
#> optimizer: <chr> adamw
#> lr: <nmr> 1e-03
#> beta1: <NUL> NULL
#> beta2: <NUL> NULL
#> eps: <NUL> NULL
#> momentum: <NUL> NULL
#> lr_scheduler: <NUL> NULL
#> batch_size: <NUL> NULL
#> max_epochs: <int> 100
#> patience: <int> 10
#> max_grad_norm: <NUL> NULL
#> validation_fraction: <nmr> 0.10