Set up a FutureExecutionConfig

Parallel execution through future. A plan is required, and is filled in from getOption("future.plan", "mirai_multisession") when not given.

Usage

setup_FutureExecution(
  n_workers = NULL,
  n_workers_outer = NULL,
  n_workers_tuning = NULL,
  n_workers_algorithm = NULL,
  device = NULL,
  future_plan = getOption("future.plan", "mirai_multisession"),
  on_error = c("continue", "stop", "stop_outer"),
  seed = NULL,
  shared_memory = c("auto", "none", "always")
)

Arguments

n_workers

Integer [1, Inf): Number of parallel workers. Left NULL, it follows the named dispatch levels, or default_n_workers() if none is named.

n_workers_outer

Optional Integer [1, Inf): Workers for outer resampling, overriding the automatic assignment.

n_workers_tuning

Optional Integer [1, Inf): Workers for tuning, overriding the automatic assignment.

n_workers_algorithm

Optional Integer [1, Inf): Threads for a self-parallelizing algorithm, overriding the automatic assignment.

device

Optional DeviceConfig: Compute device, as a type name – "cpu", "cuda", "mps" (the Apple silicon GPU) or "opencl" (a LightGBM GPU build) – or setup_CUDA to name which GPUs. NULL selects one per algorithm: "cuda" where the machine has it and the algorithm can use it, else the CPU. An algorithm that cannot use the requested device runs on the CPU, and the run's resources line says so; a device the machine lacks is an error when an algorithm tries to use it. "mps" is never selected automatically: in rtemis benchmarks on an Apple M5 the CPU was faster for every network narrower than about 1024 units or trained in batches smaller than about 2048 cases, often by 2 to 5 times, and "mps" was faster by up to about 25% only with both and thousands of features. It also slows down across many fits in one R session, and a seed does not reach dropout on it.

future_plan

Character: Future plan to use. Defaults to getOption("future.plan", "mirai_multisession").

on_error

Character {"continue", "stop", "stop_outer"}: Failure policy. "continue" makes grid cells and unscorable hyperparameter combinations non-fatal (recorded, warned, and excluded), failing only when nothing is scorable or the final model fails; "stop" aborts on any error; "stop_outer" tolerates grid-cell failures but aborts on an outer-fold failure.

seed

Optional Integer [0, Inf): Master seed for the run's computation RNG, from which one independent substream per parallel task is derived. Left NULL, a seed is drawn from the current RNG stream and recorded on the returned object, so every run is reproducible and the seed it used is auditable. Distinct from a ResamplerConfig seed, which governs how the data is split.

shared_memory

Character {"none", "auto", "always"}: Whether to hand workers the training data through OS shared memory instead of serializing a copy to each. "auto" is the default: it shares whenever it can – workers parallel and on this machine, mori installed – and quietly does not when it cannot, falling back to the ordinary transport. "none" disables it. "always" is a demand: it shares even when the run is sequential, so a run can be compared against its own shared counterpart, and raises an error when the request cannot be met, for a caller relying on sharing to stay inside a memory budget.

Value

FutureExecutionConfig object.

Details

Worker levels

There are three levels that can absorb workers: a self-parallelizing algorithm, tuning, and outer resampling. Left alone, n_workers is assigned to exactly one of them, in that order of priority, so no two levels compete for the same cores.

Naming any of n_workers_outer, n_workers_tuning or n_workers_algorithm takes over that assignment: the automatic one is switched off entirely and any level not named gets 1, so a run never mixes a hand-picked level with an inferred one.

Outer resampling and tuning dispatch to worker processes, and an outer fold runs inside one of those processes, so only one of the two can be parallel – setting both above 1 is an error. n_workers_algorithm is threads within a worker, so it combines with either.

Reproducibility

Substreams are assigned by task index, so a run gives the same answer under every backend and worker count, and the parallel result matches the sequential one exactly.

Shared memory needs mori, whose minimum R version (4.3) is above rtemis's own (4.1), so "auto" is the portable setting for a script that may run anywhere. It is local RAM: workers on another machine cannot map it.

Author

EDG

Examples

setup_FutureExecution(n_workers = 4L, future_plan = "multisession")

#> <FutureExecutionConfig>
#>       backend: <chr> future
#>        device: <NUL> NULL
#>      on_error: <chr> continue
#>          seed: <int> 230262545
#> shared_memory: <chr> auto
#>     n_workers: <int> 4
#>   future_plan: <chr> multisession
#>