4 Reading Assessments

Assessment documents can be converted into grouped item text for mapping. read_document() extracts document text, and extract_assessment() uses a language model to identify assessment items. The extracted items are drafts that require comparison with the source document before analysis.

4.1 Document reading

library(curriculum)

text <- read_document("assignment_1.docx")
cat(substr(text, 1, 1500))

File type	Reading method
`.docx`, `.html`, `.odt`, `.rtf`, `.epub`, `.tex`	Conversion to Markdown through Pandoc
`.pdf`	Text extraction through `pdftools::pdf_text()`
`.R`, `.qmd`, `.Rmd`, `.md`, `.txt`	Line-based reading
`.csv`	Cells joined within rows, with one line per row

Markdown conversion retains structural cues such as headings, lists, and tables. Without Pandoc, .docx uses a fallback reader with a warning; other supported markup formats require Pandoc. Legacy .doc files require conversion to .docx.

Inspect the extracted text for reading order, mathematical notation, table contents, and missing material. Image-only PDFs require a separate text recognition step. extract_assessment() rejects empty document text.

4.2 Item extraction

Operation	Method	Result
Document reading	Format-specific extraction	Document text
Segmentation	Structured language-model response	Assessment title and item texts
Metadata extraction	Pattern matching	Point values and bonus indicators

The segmentation prompt requests separately answerable items with sufficient context to be interpreted individually. An array schema represents the item boundaries. Schema compliance establishes response structure; it does not establish that segmentation or transcription is correct.

Point values are extracted from patterns such as (5 points), [10 pts], and @grade 1/1. Bonus indicators are derived from expressions such as “bonus” and “extra credit.” Inline markup is cleaned during extraction.

4.2.1 One document

library(rtemis.llm)

extraction_config <- config_Ollama(
  model_name = "your-llm-model",
  temperature = 0
)

assessment <- extract_assessment(
  "assignment_1.docx",
  llmconfig = extraction_config,
  name = "assignment_1"
)
assessment
as.data.frame(assessment)

Replace the model name with an available structured-output model. Extraction uses an LLMConfig; decision-model configurations are used for alignment scoring, not document extraction. The document must fit the selected model’s input limits.

The result is an Assessment object with name, items, source, and run properties. Its item table contains index, text, points, and bonus. An explicit name determines the assessment key. Otherwise, a sufficiently specific extracted title is used, with the file name as a fallback.

Use kind to describe the document when it differs from a standard question set:

assessment <- extract_assessment(
  "lab_03.R",
  llmconfig = extraction_config,
  name = "lab_03",
  kind = "an R script containing exercises and their solutions"
)

4.2.2 Several documents

documents <- c("assignment_1.docx", "assignment_2.docx", "final_exam.pdf")
assessments <- extract_assessments(
  documents,
  llmconfig = extraction_config,
  names = c("assignment_1", "assignment_2", "final_exam")
)

attr(assessments, "failed")
lapply(assessments, as.data.frame)

The batch reader accepts an explicit vector of paths, including mixed file types. Failed documents are reported in the "failed" attribute while successful extractions are retained. Duplicate assessment names receive suffixes and generate a warning. Explicit names provide stable keys across extraction runs.

4.3 Reviewing extracted items

Compare the draft with the original assessment before writing the mapping input:

Check that all intended documents and questions are represented.
Remove administrative instructions that were classified as assessment items.
Review the division of multipart questions into separately answerable tasks.
Check that each item retains its required stem, tables, and contextual details.
Verify point values, bonus indicators, and assessment names.

4.3.1 Context within each item

An item such as “Was matching on age appropriate?” requires the study context to support an interpretable alignment score. The extraction prompt requests that shared context be included in each dependent item. Review whether the returned text meets that requirement.

vapply(assessments, function(x) nrow(x@items), integer(1L))
saveRDS(assessments, "assessments_extracted.rds")

The saved objects preserve extraction metadata for later inspection.

4.4 Writing assessment files

write_assessments(assessments, "assessments.yaml")

The mapping input stores assessment text in named groups:

assessments:
  assignment_1:
    - Calculate the median of the values 2, 4, 5, 7, and 12.
  assignment_2:
    - Construct a histogram of the age variable in the supplied dataset.

write_assessments() excludes bonus items by default. Set drop_bonus = FALSE to include them. State which policy is used when reporting assessment coverage. The YAML contains the item text used for mapping; retain the extracted R objects for point values, bonus indicators, and extraction provenance. Omitting file returns the YAML as a character string for inspection.

4.5 Attaching assessments to a course

items <- read_assessments("assessments.yaml")
lengths(items)

course <- read_course("objectives.yaml", "assessments.yaml")

The assessment reader requires a nonempty, correctly structured assessments block. A course file containing its own assessment block is read with read_course("course.yaml"). Supplying both an embedded nonempty block and a separate assessment file raises an error.

Assessment items support the source-record syntax described in Reading objectives. The separate assessment reader keeps resolved records in the returned list’s "sources" attribute. Mapping uses the item text.