2  Metadata objects

library(croissant)

croissant represents each node of a Croissant document as an S7 object. read_croissant() returns a tree of these objects, write_croissant() writes one, and each class has a constructor of the same name for creating metadata in R.

2.1 Classes

Class	Croissant node	Holds
Croissant	sc:Dataset	Dataset metadata, distribution, and record_set
FileObject	cr:FileObject	One file or archive
FileSet	cr:FileSet	A set of files matched by glob patterns, usually inside an archive or repository
RecordSet	cr:RecordSet	The fields of one table of records, and optional inline data
Field	cr:Field	One column: its data type, source, join, and subfields
Source	source, references	The file, file set, record set, or field a value comes from, with its extract and transforms
Extract	extract	A column, a JSONPath, or a file property
Transform	transform	One transformation of an extracted value
ParentField	parentField	The source and join of a field’s parent
Person, Organization	sc:Person, sc:Organization	A creator or publisher
CreativeWork	sc:CreativeWork	A license given as an object

2.2 Properties

Each class types the properties that mlcroissant 1.1.1 types on the same node. Property names are the JSON-LD keys in snake case: contentUrl is content_url, encodingFormat is encoding_format, and @id is id. Two names differ from that rule: the dataset’s schema.org version is dataset_version, and Croissant@version holds the Croissant version of the document ("1.0" or "1.1").

Class	Properties
Croissant	version, id, name, description, conforms_to, dataset_version, sd_version, license, sd_license, url, same_as, cite_as, creator, publisher, date_created, date_modified, date_published, keywords, in_language, is_live_dataset, distribution, record_set, 19 Responsible AI properties
FileObject	id, name, description, content_url, content_size, encoding_format, sha256, md5, contained_in, same_as, source
FileSet	id, name, description, encoding_format, includes, excludes, contained_in
RecordSet	id, name, description, data_type, key, field, annotation, data, examples
Field	id, name, description, data_type, source, references, sub_field, parent_field, annotation, repeated, is_array, array_shape, equivalent_property, value
Source	file_object, file_set, record_set, field, extract, transform, format, sampling_rate
Extract	column, file_property, json_path
Transform	format, json_path, read_lines, regex, replace, sampling_rate, separator, un_archive
ParentField	source, references
Person, Organization	id, name, description, email, url
CreativeWork	id, name, description, url

Every class also has other_properties, described below.

The Responsible AI properties of Croissant correspond to the rai: namespace: data_collection, data_collection_type, data_collection_missing_data, data_collection_raw_data, data_imputation_protocol, data_preprocessing_protocol, data_manipulation_protocol, data_annotation_protocol, data_annotation_platform, data_annotation_analysis, annotations_per_item, annotator_demographics, machine_annotation_tools, data_biases, data_use_cases, data_limitations, data_social_impact, personal_sensitive_information, and data_release_maintenance_plan. Each is a character vector.

2.2.1 Value types

Croissant value	R value
Text (name, description)	Character vector; a value with a language tag is named by the tag
String, URL, date	Character scalar; dates are kept as the strings the document gives
Array of strings (keywords, encoding_format, data_type)	Character vector
Boolean (is_live_dataset, repeated, is_array)	Logical scalar
Reference ({"@id": "x"})	The character string "x"
Nested node	Object of the corresponding class
Array of nodes (distribution, record_set, field, transform)	List of objects
license, sd_license	Character vector, or a list of strings and CreativeWork objects
creator, publisher	List of Person and Organization objects
data, examples, Field value	The parsed JSON value

A property that holds one value or a one-element array reads to the same R value. Constructors check the type of each argument; a value of another type raises an error.

Croissant(name = 1)

Error:
! <croissant::Croissant> object properties are invalid:
- @name must be <NULL> or <character>, not <double>

2.3 Creating metadata

The constructors take properties as arguments. A reference to another node is the @id of that node as a string. The following object describes one CSV file and one record set with two fields.

x <- Croissant(
  name = "people",
  description = "Names and ages.",
  license = "CC0-1.0",
  creator = list(Person(name = "A. Author", email = "[email protected]")),
  distribution = list(
    FileObject(
      id = "people.csv",
      content_url = "people.csv",
      encoding_format = "text/csv"
    )
  ),
  record_set = list(
    RecordSet(
      id = "people",
      field = list(
        Field(
          id = "people/name",
          data_type = "sc:Text",
          source = Source(file_object = "people.csv", extract = Extract(column = "name"))
        ),
        Field(
          id = "people/age",
          data_type = "sc:Integer",
          source = Source(file_object = "people.csv", extract = Extract(column = "age"))
        )
      )
    )
  )
)
x

<Croissant>
  name: people
  version: Croissant 1.1
  distribution: 1 FileObject
  record sets: 1
    people: 2 fields

print() shows the dataset name, the Croissant version, the distribution by class, and each record set with its number of fields. Properties are read and set with @:

x@record_set[[1]]@field[[2]]@data_type

[1] "sc:Integer"

x@keywords <- c("demography", "example")
x@keywords

[1] "demography" "example"   

The Croissant version of a new object defaults to "1.1"; version = "1.0" creates 1.0 metadata.

2.4 Text with language tags

name and description accept a character vector named by language tag. write_croissant() writes it as a language map in Croissant 1.1 and as an array of value objects in 1.0.

birds <- Croissant(version = "1.0", name = c(en = "Birds", fr = "Oiseaux"))
birds@name

       en        fr 
  "Birds" "Oiseaux" 

2.5 Other properties

A property that the class does not type is kept in the node’s other_properties list under its canonical JSON-LD key, with its parsed JSON value, and is written back as a key of the same node. Reading and then writing a document therefore keeps every property. Typical contents are the prov: properties, schema.org properties such as temporalCoverage and alternateName, and a creator given as a plain name rather than a Person or Organization.

doc <- '{
  "@context": {
    "@vocab": "https://schema.org/",
    "sc": "https://schema.org/",
    "dct": "http://purl.org/dc/terms/",
    "conformsTo": "dct:conformsTo",
    "prov": "http://www.w3.org/ns/prov#"
  },
  "@type": "sc:Dataset",
  "name": "survey",
  "conformsTo": "http://mlcommons.org/croissant/1.1",
  "temporalCoverage": "2020/2024",
  "prov:wasDerivedFrom": {"@id": "https://example.org/raw"}
}'
survey <- read_croissant(doc)
str(survey@other_properties)

List of 2
 $ temporalCoverage   : chr "2020/2024"
 $ prov:wasDerivedFrom:List of 1
  ..$ @id: chr "https://example.org/raw"

other_properties is a field of the R object; it is not a Croissant property and never appears as a key in a written document.