Read the records of a record set

Reads the files a record set draws from, and returns its records as a table with one column per field, named by field @id. Subfields are columns of their own, named by their @id.

Usage

read_record_set(
  x,
  record_set,
  output = c("data.table", "tibble", "data.frame", "arrow"),
  base = NULL,
  cache_dir = getOption("croissant.cache_dir", tools::R_user_dir("croissant", "cache")),
  n_max = Inf,
  token = NULL,
  verbosity = 1L
)

Arguments

x

Croissant object, or Character: Path or URL of a Croissant JSON-LD file.

record_set

Character: @id or name of the record set.

output

Character {"data.table", "tibble", "data.frame", "arrow"}: Class of the result. "tibble" requires the 'tibble' package and "arrow" the 'arrow' package.

base

Optional Character: Directory or URL that relative contentUrls and FileSet globs resolve against. When x is a path or URL, this defaults to its directory.

cache_dir

Character: Directory for downloaded and extracted files.

n_max

Numeric [1, Inf]: Most records to return. Files of a FileSet are read in order until the first origin holds n_max records, so later files are not downloaded.

token

Optional Character: Hugging Face token, for files of gated or private Hub datasets. NULL reads HF_TOKEN.

verbosity

Integer: Verbosity level; 1L reports downloads.

Value

A data.table, tibble, data.frame, or arrow Table, by output.

Details

Files with an http(s) contentUrl are downloaded into cache_dir, and their sha256 or md5 is verified. Files of a Hugging Face dataset repository (a git+https FileObject with a Hub contentUrl, as in the metadata read_croissant_hf() reads, or an hf://datasets/ path) are downloaded from the resolved commit and verified against the Hub's listing, as hf_download() does. Zip and tar archives are extracted into cache_dir. Relative contentUrls and FileSet globs resolve against base. Columns are read from CSV, TSV, JSON, JSON Lines, and Parquet files; Parquet requires the 'arrow' package. A field extracting the content of an image, audio, or video file holds the file's local path; with a text data type it holds the file's text.

Values are cast by the field's first data type with an R counterpart: sc:Integer to integer, sc:Float and sc:Number to double, sc:Boolean to logical, sc:Date to Date, sc:DateTime to POSIXct (UTC), and sc:Text and sc:URL to character. A repeated field, or one split by a separator transform, is a list column.

Author

EDG

Examples

dir <- tempfile()
dir.create(dir)
writeLines(c("name,age", "Alice,22", "Bob,23"), file.path(dir, "people.csv"))
x <- Croissant(
  name = "people",
  distribution = list(
    FileObject(id = "people.csv", content_url = "people.csv", encoding_format = "text/csv")
  ),
  record_set = list(RecordSet(
    id = "people",
    field = list(
      Field(
        id = "people/name",
        data_type = "sc:Text",
        source = Source(file_object = "people.csv", extract = Extract(column = "name"))
      ),
      Field(
        id = "people/age",
        data_type = "sc:Integer",
        source = Source(file_object = "people.csv", extract = Extract(column = "age"))
      )
    )
  ))
)
read_record_set(x, "people", base = dir)

#>    people/name people/age
#>         <char>      <int>
#> 1:       Alice         22
#> 2:         Bob         23