Setup Variational Autoencoder config. — setup_VariationalAutoencoder • rtemis

A fully connected variational autoencoder built and trained with torch: the encoder maps each case to a normal distribution over k latent dimensions, and the decoder maps a draw from it back to the features.

Usage

setup_VariationalAutoencoder(
  k = 2L,
  hidden_units = NULL,
  activation = "relu",
  norm = NULL,
  dropout = 0,
  input_dropout = 0,
  weight_decay = 0,
  input_noise = 0,
  loss = "mse",
  optimizer = "adamw",
  lr = 0.001,
  beta1 = NULL,
  beta2 = NULL,
  eps = NULL,
  momentum = NULL,
  lr_scheduler = NULL,
  batch_size = NULL,
  max_epochs = 100L,
  patience = 10L,
  max_grad_norm = NULL,
  validation_fraction = 0.1,
  beta = 1,
  features = NULL
)

Arguments

k: Integer [1, Inf): Number of components, the width of the latent layer.
hidden_units: Optional Integer [1, Inf) vector: Units in each hidden layer of the encoder, input side first.
activation: Character {"relu", "gelu", "silu", "elu", "selu", "leaky_relu", "tanh"}: Activation applied after every hidden layer.
norm: Optional Character {"batch_norm", "layer_norm"}: Normalization applied in every hidden layer. NULL applies none.
dropout: Numeric [0, 1): Dropout probability applied after every hidden layer.
input_dropout: Numeric [0, 1): Probability of zeroing each standardized input during training.
weight_decay: Numeric [0, Inf): L2 penalty, decoupled from the gradient under the adamw optimizer.
input_noise: Numeric [0, Inf): Standard deviation of the Gaussian noise added to the standardized inputs during training.
loss: Character {"mse", "l1", "smooth_l1"}: Reconstruction loss.
optimizer: Character {"adamw", "adam", "sgd", "rmsprop"}: Optimization algorithm.
lr: Numeric (0, Inf): Learning rate.
beta1: Optional Numeric [0, 1): Exponential decay rate of the first moment estimate. Applies to the adam and adamw optimizers.
beta2: Optional Numeric [0, 1): Exponential decay rate of the second moment estimate. Applies to the adam and adamw optimizers.
eps: Optional Numeric (0, Inf): Term added to the denominator for numerical stability. Applies to the adam, adamw and rmsprop optimizers.
momentum: Optional Numeric [0, Inf): Momentum factor. Applies to the sgd and rmsprop optimizers.
lr_scheduler: Optional Character {"step", "cosine_annealing", "one_cycle", "reduce_on_plateau"}: Learning-rate schedule. NULL holds the learning rate fixed.
batch_size: Optional Integer [1, Inf): Cases per optimization step.
max_epochs: Integer [1, Inf): Largest number of passes over the training cases.
patience: Integer [1, Inf): Epochs without improvement of the held-out loss before stopping early.
max_grad_norm: Optional Numeric (0, Inf): Clip the gradient norm to this value before each step. NULL does not clip.
validation_fraction: Numeric [0, 1): Fraction of the cases held out for early stopping.
beta: Numeric [0, Inf): Weight of the KL divergence from the prior.
features: Optional Character vector: Names of at least 2 distinct feature columns to decompose. NULL decomposes all numeric features.

Value

VariationalAutoencoderConfig object.

Details

Objective. For each case, the reconstruction loss summed over features plus beta times the KL divergence of the case's latent distribution from the standard normal, summed over the k dimensions; the mean over cases is minimized. beta = 1 is the standard variational autoencoder (with loss = "mse", up to constants, the negative evidence lower bound of a unit-variance Gaussian decoder); beta > 1 is a beta-VAE, trading reconstruction for components closer to independent; beta = 0 drops the prior and leaves a noisy autoencoder.

Components are the means of the latent distributions, so decomp, apply_decomp and reconstruct are deterministic: sampling happens only in training. reconstruct decodes the means.

Architecture, input standardization, denoising, early stopping, and the execution config's seed, threads and device work as in setup_Autoencoder. The loss reported during training and for early stopping is the full objective, so it is not on the scale of setup_Autoencoder's, which averages the reconstruction loss over features.

Author

EDG

Examples

vae_config <- setup_VariationalAutoencoder(k = 2L, beta = 4)
vae_config

#> <VariationalAutoencoder DecompositionConfig>
#>                   k: <int> 2
#>        hidden_units: <NUL> NULL
#>          activation: <chr> relu
#>                norm: <NUL> NULL
#>             dropout: <nmr> 0.00
#>       input_dropout: <nmr> 0.00
#>        weight_decay: <nmr> 0.00
#>         input_noise: <nmr> 0.00
#>                loss: <chr> mse
#>           optimizer: <chr> adamw
#>                  lr: <nmr> 1e-03
#>               beta1: <NUL> NULL
#>               beta2: <NUL> NULL
#>                 eps: <NUL> NULL
#>            momentum: <NUL> NULL
#>        lr_scheduler: <NUL> NULL
#>          batch_size: <NUL> NULL
#>          max_epochs: <int> 100
#>            patience: <int> 10
#>       max_grad_norm: <NUL> NULL
#> validation_fraction: <nmr> 0.10
#>                beta: <nmr> 4.00