5 Reading records

library(croissant)
options(croissant.cache_dir = file.path(tempdir(), "croissant-cache"))

read_record_set() reads the files that a record set draws from and returns its records as a table. The procedure follows the record loading of mlcroissant 1.1.1.

read_record_set(
  x,
  record_set,
  output = c("data.table", "tibble", "data.frame", "arrow"),
  base = NULL,
  cache_dir = getOption("croissant.cache_dir", tools::R_user_dir("croissant", "cache")),
  n_max = Inf,
  token = NULL,
  verbosity = 1L
)

Argument	Meaning
`x`	A `Croissant` object, or the path or URL of a Croissant document
`record_set`	The `@id` or name of the record set
`output`	Class of the result; `"tibble"` requires `tibble` and `"arrow"` requires `arrow`
`base`	Directory or URL that relative `contentUrl`s resolve against; defaults to the directory of `x` when `x` is a path or URL
`cache_dir`	Directory for downloaded and extracted files
`n_max`	Largest number of records to return
`token`	Hugging Face token for gated and private datasets; `NULL` reads `HF_TOKEN`
`verbosity`	`1L` reports downloads; `0L` reports nothing

The result has one column per leaf field, in depth-first order, named by field @id. A subfield is a column of its own, named by its @id.

5.1 Data for the examples

The examples on this page use two files: a CSV file of orders and a JSON Lines file of customers, written to a temporary directory.

dir <- file.path(tempdir(), "shop")
dir.create(dir, showWarnings = FALSE)
writeLines(
  c(
    "order_id,customer_id,items,placed",
    "1,c1,apple;pear,2026-03-01",
    "2,c2,fig,2026-03-02",
    "3,c1,kiwi;plum;lime,2026-03-04"
  ),
  file.path(dir, "orders.csv")
)
writeLines(
  c(
    '{"id": "c1", "profile": {"city": "Athens"}}',
    '{"id": "c2", "profile": {"city": "Oslo"}}'
  ),
  file.path(dir, "customers.jsonl")
)
md5 <- function(file) unname(tools::md5sum(file.path(dir, file)))

The metadata defines three record sets. customers reads the JSON Lines file, with a column extract and a JSONPath extract. orders reads the CSV file, joins customers through references, and splits items with a separator. sizes holds its records inline.

x <- Croissant(
  name = "shop",
  license = "CC0-1.0",
  cite_as = "Shop data (2026).",
  date_published = "2026-10-27",
  dataset_version = "1.0.0",
  distribution = list(
    FileObject(
      id = "orders.csv",
      content_url = "orders.csv",
      encoding_format = "text/csv",
      md5 = md5("orders.csv")
    ),
    FileObject(
      id = "customers.jsonl",
      content_url = "customers.jsonl",
      encoding_format = "application/jsonl",
      md5 = md5("customers.jsonl")
    )
  ),
  record_set = list(
    RecordSet(
      id = "customers",
      key = "customers/id",
      field = list(
        Field(
          id = "customers/id",
          data_type = "sc:Text",
          source = Source(file_object = "customers.jsonl", extract = Extract(column = "id"))
        ),
        Field(
          id = "customers/city",
          data_type = "sc:Text",
          source = Source(
            file_object = "customers.jsonl",
            extract = Extract(json_path = "$.profile.city")
          )
        )
      )
    ),
    RecordSet(
      id = "orders",
      field = list(
        Field(
          id = "orders/order_id",
          data_type = "sc:Integer",
          source = Source(file_object = "orders.csv", extract = Extract(column = "order_id"))
        ),
        Field(
          id = "orders/customer_id",
          data_type = "sc:Text",
          source = Source(file_object = "orders.csv", extract = Extract(column = "customer_id")),
          references = Source(field = "customers/id")
        ),
        Field(
          id = "orders/city",
          data_type = "sc:Text",
          source = Source(field = "customers/city")
        ),
        Field(
          id = "orders/items",
          data_type = "sc:Text",
          source = Source(
            file_object = "orders.csv",
            extract = Extract(column = "items"),
            transform = list(Transform(separator = ";"))
          )
        ),
        Field(
          id = "orders/placed",
          data_type = "sc:Date",
          source = Source(file_object = "orders.csv", extract = Extract(column = "placed"))
        )
      )
    ),
    RecordSet(
      id = "sizes",
      data_type = "sc:Enumeration",
      field = list(
        Field(id = "sizes/code", data_type = "sc:Text"),
        Field(id = "sizes/label", data_type = "sc:Text")
      ),
      data = list(
        list(`sizes/code` = "S", `sizes/label` = "Small"),
        list(`sizes/code` = "L", `sizes/label` = "Large")
      )
    )
  )
)
x

<Croissant>
  name: shop
  version: Croissant 1.1
  distribution: 2 FileObject
  record sets: 3
    customers: 2 fields
    orders: 5 fields
    sizes: 2 fields, 2 inline records

validate_croissant(x)

<CroissantIssues>
  0 errors, 0 warnings

metadata <- file.path(dir, "croissant.json")
write_croissant(x, metadata)

5.2 Reading a record set

read_record_set(metadata, "customers")

   customers/id customers/city
         <char>         <char>
1:           c1         Athens
2:           c2           Oslo

base defaults to the directory of metadata, where the relative contentUrls resolve. With a Croissant object, base names the directory:

read_record_set(x, "customers", base = dir)

   customers/id customers/city
         <char>         <char>
1:           c1         Athens
2:           c2           Oslo

output selects the class of the result:

read_record_set(metadata, "sizes", output = "data.frame")

  sizes/code sizes/label
1          S       Small
2          L       Large

class(read_record_set(metadata, "sizes", output = "tibble"))

[1] "tbl_df"     "tbl"        "data.frame"

5.3 Files

5.3.1 Location and download

A relative contentUrl resolves against base. When base is a URL, the file is downloaded from it. A file with an http(s) contentUrl is downloaded into <cache_dir>/downloads/, verified against its sha256 or md5, and reused on later reads while its checksum matches. A checksum mismatch removes the file and raises an error of class croissant_checksum_error. A git+https contentUrl raises an error of class croissant_file_error, except for Hugging Face repositories, described in Hugging Face datasets.

cache_dir defaults to the option croissant.cache_dir and otherwise to tools::R_user_dir("croissant", "cache").

5.3.2 Archives and file sets

A zip or tar archive is extracted into <cache_dir>/extracted/. A FileObject contained in an archive is the file at its contentUrl inside the extracted archive. A FileSet lists the files under each of its containers whose path, relative to the container, matches an includes pattern and no excludes pattern. Files are sorted within each container. Patterns match as Python’s fnmatch: * matches any sequence of characters, including /.

The following dataset is a zip archive of two CSV files and a text file. The FileSet selects the CSV files; the sensor field extracts each file’s name and keeps the part before .csv with a regex transform.

src <- file.path(tempdir(), "sensors-src")
dir.create(file.path(src, "readings"), recursive = TRUE, showWarnings = FALSE)
writeLines(c("time,value", "0,1.5", "1,1.7"), file.path(src, "readings", "north.csv"))
writeLines(c("time,value", "0,2.1", "1,2.4"), file.path(src, "readings", "south.csv"))
writeLines("Notes on the sensors.", file.path(src, "readings", "README.txt"))

sensors_dir <- file.path(tempdir(), "sensors")
dir.create(sensors_dir, showWarnings = FALSE)
zip_path <- file.path(sensors_dir, "readings.zip")
withr::with_dir(src, utils::zip(zip_path, "readings", flags = "-rq"))

sensors <- Croissant(
  name = "sensors",
  distribution = list(
    FileObject(
      id = "archive",
      content_url = "readings.zip",
      encoding_format = "application/zip",
      md5 = unname(tools::md5sum(zip_path))
    ),
    FileSet(
      id = "csv-files",
      contained_in = "archive",
      encoding_format = "text/csv",
      includes = "readings/*.csv"
    )
  ),
  record_set = list(
    RecordSet(
      id = "readings",
      field = list(
        Field(
          id = "readings/sensor",
          data_type = "sc:Text",
          source = Source(
            file_set = "csv-files",
            extract = Extract(file_property = "filename"),
            transform = list(Transform(regex = "^(.*)\\.csv$"))
          )
        ),
        Field(
          id = "readings/time",
          data_type = "sc:Integer",
          source = Source(file_set = "csv-files", extract = Extract(column = "time"))
        ),
        Field(
          id = "readings/value",
          data_type = "sc:Float",
          source = Source(file_set = "csv-files", extract = Extract(column = "value"))
        )
      )
    )
  )
)
read_record_set(sensors, "readings", base = sensors_dir)

   readings/sensor readings/time readings/value
            <char>         <int>          <num>
1:           north             0            1.5
2:           north             1            1.7
3:           south             0            2.1
4:           south             1            2.4

5.3.3 File formats

The encoding format of a file determines how it is read.

Encoding format	Reading
`text/csv`, `text/plain`	`data.table::fread()`, every column as character
`text/tab-separated-values`	`data.table::fread()` with a tab separator
`application/json`	Parsed whole
`application/jsonl`, `application/jsonlines`, `application/x-jsonlines`, `application/x-ndjson`	Parsed line by line
`application/x-parquet`, `application/parquet`, `application/vnd.apache.parquet`	`arrow::read_parquet()`

In CSV and TSV files, the strings that pandas reads as missing, such as the empty string, NA, NaN, N/A, NULL, null, and None, read as NA. Columns are cast afterwards by the field’s data type. A file in any other format can be read through a file property.

5.4 Extraction

An Extract takes values from a file in one of three ways.

Extract	Takes
`column`	A column of a table, or a key of each JSON record in an array or JSON Lines file
`json_path`	The matches of a JSONPath in a JSON document, or the first match in each JSON Lines record
`file_property`	A property of the file itself

5.4.1 File properties

`file_property`	Value
`filename`	The file name
`filepath`	The local path of the file
`fullpath`	The path of the file relative to its container
`content`	The content of the file
`lines`	One record per line, holding the line
`lineNumbers`	One record per line, holding the line number, from 0

A field that extracts content with a text data type holds the file’s text, or its path when the file contains a NUL byte. With an image, audio, or video data type, it holds the file’s local path. A record set whose fields extract only filename or fullpath lists files without downloading them.

5.4.2 JSONPath

croissant evaluates the JSONPath operators $, .key, ['key'], [n], [*], .*, and ..key. Any other syntax raises an error of class croissant_jsonpath_error.

5.5 Transforms

A source’s transforms apply in order to each extracted value.

Transform	Effect
`regex`	Keeps the first non-empty capture group of a match anchored at the start of the value; keeps the value when there is no match or no group
`json_path`	Keeps the first match
`format`	Formats a date-time
`separator`	Splits the value into a list

orders <- read_record_set(metadata, "orders")
orders

   orders/order_id orders/customer_id orders/city   orders/items orders/placed
             <int>             <char>      <char>         <list>        <Date>
1:               1                 c1      Athens     apple,pear    2026-03-01
2:               2                 c2        Oslo            fig    2026-03-02
3:               3                 c1      Athens kiwi,plum,lime    2026-03-04

orders[["orders/items"]]

[[1]]
[1] "apple" "pear" 

[[2]]
[1] "fig"

[[3]]
[1] "kiwi" "plum" "lime"

5.6 Joins

Each leaf field’s source names an origin: a file, a file set, or the record set of a source field. The first origin that the fields name gives the rows. Each further origin, including one that a field only references, is left-joined through a field that reads from an origin already joined and references a field of the new one.

In the orders record set above, orders/customer_id reads from orders.csv and references customers/id, so the rows of customers are joined on that key, and orders/city takes its value from customers/city. The rules of the join are:

the transforms of the referencing field apply to the left key;
keys compare as strings, with whole numbers written without decimals;
a left row with several matches is repeated;
a left row without a match keeps NA in the joined columns.

A record set whose fields read from two origins without such a field raises an error of class croissant_record_error; validate_croissant() reports it as undeclared_join.

5.7 Inline data and constant values

A record set with inline data reads its records from it, with one record per element and keys equal to the field @ids. A field with a value and no source repeats the value in every record.

5.8 Data types

The first data type of the field that has an R counterpart determines the cast. Data types are matched by local name, so sc:Integer and https://schema.org/Integer cast alike. A field without such a data type keeps the values as read.

Data type	R type
`sc:Integer`, `Int8`, `Int16`, `Int32`, `UInt8`, `UInt16`	integer; double for values beyond the integer range
`Int64`, `UInt32`, `UInt64`	double
`sc:Float`, `Float16`, `Float32`, `Float64`, `sc:Number`	double
`sc:Boolean`	logical
`sc:Date`	Date when every value is a date without time; otherwise POSIXct (UTC)
`sc:DateTime`	POSIXct (UTC)
`sc:Text`, `sc:URL`, `sc:Time`	character
`cr:BoundingBox`	list of numeric vectors
`sc:ImageObject`, `sc:AudioObject`, `sc:VideoObject`	the path of the file

A repeated field, an array field, and a field split by a separator are list columns.

5.9 Limiting the number of records

n_max sets the largest number of records returned. The files of the first origin are read in order until it holds n_max records; later files are neither read nor downloaded. Other origins are read whole.

read_record_set(sensors, "readings", base = sensors_dir, n_max = 2)

   readings/sensor readings/time readings/value
            <char>         <int>          <num>
1:           north             0            1.5
2:           north             1            1.7