Read a data file and normalize it to Parquet

The first step of any pipeline. A delimited file or a spreadsheet carries no usable type information; Parquet does. Ingesting once means nothing downstream infers a type a second time, and the decisions that were made are in the config.

The config's format is checked against the file, and a mismatch is an error: a config written for a delimited file sets options a Parquet reader would ignore.

Returns a manifest describing what happened: the file in, the file out, a DataFingerprint of the data, the config, and which engine ran it. The engine is named from the start because it is unrecoverable later – once a second implementation exists, nothing else could say which produced a given Parquet.

Usage

ingest(path, outfile, config = NULL, overwrite = FALSE, verbosity = 1L)

Arguments

path: Character: Path to the input file.
outfile: Character: Path to write the Parquet file to.
config: Optional IngestConfig: How to read it. If NULL, the defaults for the file's own format.
overwrite: Logical: If TRUE, overwrite outfile if it exists.
verbosity: Integer: Verbosity level.

Value

Named list: the ingest manifest.

Author

EDG

Examples

if (FALSE) { # \dontrun{
ingest("data.csv", "data.parquet")
ingest("data.csv", "data.parquet", setup_DelimitedIngest(sep = ";"))
} # }