Download a file from a Hugging Face dataset — hf

Resolves an hf://datasets/ path to a commit, downloads the file from that commit into cache_dir, and verifies its size and its LFS SHA-256 or git blob SHA-1 against the Hub's listing. A cached file of the listed size is reused; a commit's files do not change. A path that names the repository only reads its one data file (.parquet, .csv, .tsv, .csv.gz, .tsv.gz, .rds, .xlsx, .dta, .arff, or .fasta).

Usage

hf_download(
  path,
  cache_dir = getOption("croissant.cache_dir", tools::R_user_dir("croissant", "cache")),
  token = NULL,
  verbosity = 1L
)

Arguments

path: Character: hf://datasets/<owner>/<name>[@<revision>][/<file>]. A revision that contains / is URL-encoded, as refs%2Fconvert%2Fparquet.
cache_dir: Character: Cache directory.
token: Optional Character: Hugging Face token. NULL reads HF_TOKEN.
verbosity: Integer: Verbosity level; 1L reports downloads.

Value

Character: Local path of the file, with attribute hf, the path pinned to the commit (hf://datasets/<owner>/<name>@<commit>/<file>).

Details

A gated or private dataset requires a token with access, read from HF_TOKEN by default. The Hub is at HF_ENDPOINT, or https://huggingface.co.

Author

EDG

Examples

if (FALSE) { # interactive()
path <- hf_download("hf://datasets/ylecun/mnist@refs%2Fconvert%2Fparquet/mnist/test/0000.parquet")
attr(path, "hf")
}