7 Hugging Face datasets

The Hugging Face Hub generates Croissant metadata for each dataset it hosts. croissant reads this metadata, reads the records it describes, and downloads individual files of a dataset. Every download is pinned to a commit and verified against the Hub’s listing.

The examples on this page require network access and are not executed when this book is built.

library(croissant)

7.1 Paths

A Hugging Face dataset or file is named by a path of the form

hf://datasets/<owner>/<name>[@<revision>][/<file>]

The revision is a branch, tag, or commit, and defaults to main. A revision that contains / is URL-encoded: @refs%2Fconvert%2Fparquet names the branch refs/convert/parquet, where the Hub stores the Parquet conversion of a dataset.

7.2 Reading metadata

read_croissant_hf(repo, token = NULL) downloads the metadata at <endpoint>/api/datasets/<owner>/<name>/croissant and reads it with read_croissant(). repo is <owner>/<name> or an hf://datasets/ path.

x <- read_croissant_hf("ylecun/mnist")
x
validate_croissant(x)

read_croissant() accepts the same paths. A path that names the dataset only, hf://datasets/<owner>/<name>, reads the generated metadata, as read_croissant_hf() does; a path that names a file downloads and reads that file.

x <- read_croissant("hf://datasets/ylecun/mnist")

7.3 Reading records

The generated metadata describes the dataset repository as a FileObject with a git+https contentUrl, and its data files as a FileSet contained in it, with the repository URL naming the refs/convert/parquet branch. read_record_set() resolves the revision to a commit, lists the repository’s files at that commit, selects those that match the FileSet patterns, and downloads each file when a field reads it. A FileObject whose contentUrl is an hf://datasets/ path is read the same way.

digits <- read_record_set(x, "mnist", n_max = 5)
digits[["mnist/label"]]

With n_max = 5, only the first Parquet file of the record set is downloaded. On 2026-10-27, this call downloaded mnist/test/0000.parquet (2.6 MB) and returned the labels 7, 2, 1, 0, and 4; reading the record set without n_max returned 70,000 records (60,000 train and 10,000 test).

Parquet struct columns read as one named list per record, so a json_path transform can select from them. Image, audio, and video values stored as raw bytes are kept as raw vectors.

7.4 Downloading files

hf_download(path, cache_dir, token, verbosity) downloads one file and returns its local path. The attribute hf holds the path pinned to the commit.

path <- hf_download(
  "hf://datasets/ylecun/mnist@refs%2Fconvert%2Fparquet/mnist/test/0000.parquet"
)
attr(path, "hf")

A path that names the repository only downloads its one data file, with extension .parquet, .csv, .tsv, .csv.gz, .tsv.gz, .rds, .xlsx, .dta, .arff, or .fasta. A repository with no such file, or with more than one, raises an error of class rtemis_value_error.

7.5 Download and verification

Each download follows the same procedure:

The revision is resolved to a commit through <endpoint>/api/datasets/<owner>/<name>/revision/<revision>?blobs=true, which also lists each file’s size, LFS SHA-256, and git blob SHA-1.
The file is downloaded from <endpoint>/datasets/<owner>/<name>/resolve/<commit>/<file> into <cache_dir>/hf/datasets/<owner>/<name>/<commit>/<file>.
A new download is checked against the listing: its size, then its LFS SHA-256, or, for a file stored outside LFS, its git blob SHA-1. A mismatch removes the file and raises an error of class croissant_checksum_error.

A cached file of the listed size is reused. The files of a commit do not change, so a pinned path always returns the same content.

A failed request raises an error of class croissant_download_error. The download timeout is at least 3,600 seconds.

7.6 Gated and private datasets

A gated or private dataset requires a token with access to it. The token is the token argument of read_croissant_hf(), read_record_set(), or hf_download(), and otherwise the environment variable HF_TOKEN. It is sent as an Authorization: Bearer header and does not appear in any message or condition. A gated dataset read without a token raises an error of class rtemis_unsupported_error.

Sys.setenv(HF_TOKEN = "<token>")
x <- read_croissant_hf("<owner>/<gated-dataset>")

7.7 Endpoint

The Hub endpoint is the environment variable HF_ENDPOINT, without a trailing slash, and otherwise https://huggingface.co. An endpoint that is not an http(s) URL raises an error of class rtemis_value_error.