Validate an rtemis config, against the schema and optionally against data

Checks a config document – as parsed from JSON, or already resolved – and reports what is wrong with it. With no data, only the schema is checked: whether the document reconstructs into a valid rtemis config object. With data, the config is additionally checked against the dataset it would run on, catching the errors a schema cannot see: an outcome column that is not there, more folds than the rarer class has cases, predictors that never vary.

Usage

validate_config(config, data = NULL, outcome = NULL, step = NULL)

Arguments

config

Named list parsed from a config document (carrying its $schema), or a resolved rtemis config object. A resolved object is valid by construction, so the schema check is skipped for it.

data

Optional tabular data: The dataset the config would run on. NULL checks the schema only.

outcome

Optional Character: Name of the outcome column in data. NULL takes rtemis's convention, the last column – but only for a config that designates an outcome at all; a decomposition or clustering config has no outcome and every column is a feature.

step

Optional Integer [1, Inf): Position of this config in the plan it came from, recorded on every finding. NULL when the config stands alone.

Value

Diagnostics object. Empty when nothing is wrong.

Details

Nothing is thrown. Every finding is a Diagnostic carrying a stable code, a severity, a technical message, a plain-language plain text, the evidence behind it, and – where a deterministic one exists – a fix, an RFC 6902 JSON Patch that resolves it. An empty result means the config is clean.

Each data check runs only when the config carries the part it is about: the resampling checks need a resampler, the outcome checks need an outcome. A config carrying neither is checked for what it does declare and reports nothing about what it does not, so a preprocessing step and a training step each get the checks that apply to them.

Author

EDG

Examples

# A config that does not reconstruct: one finding, and no data checks.
bad <- list(
  `$schema` = "https://schema.rtemis.org/supervised/r/v1/schema.json",
  n_foldz = 10L
)
validate_config(bad)

#> <Diagnostics>
#>   * SCHEMA_INVALID (error): Unknown supervised config key: `n_foldz`.

# A resolved config with nothing wrong: an empty Diagnostics.
validate_config(setup_SuperConfig(hyperparameters = setup_LightRF()))

#> <Diagnostics>
#>   No problems found.