Extract assessment items from a document — extract

Turns one Word, PDF, R, Quarto or CSV document into a list of individual assessment items, ready to be written to YAML by write_assessments() and mapped by map_sessions_to_assessments().

Usage

extract_assessment(
  file,
  llmconfig,
  name = NULL,
  kind = "a course assessment: a homework set, problem set or exam",
  verbosity = 1L,
  ...
)

Arguments

file: Character: Path to the document. .docx, .html, .odt, .rtf, .epub and .tex (each requiring the pandoc package), .pdf (requiring pdftools), .csv, or .R, .qmd, .Rmd, .md and .txt read as they are. See read_document() for how each is read, and why the markup formats are converted to markdown rather than to plain text.
llmconfig: rtemis.llm::LLMConfig: Model configuration.
name: Character: Assessment name, used as the YAML key and in generated ids. Defaults to the document's title where it states one that identifies the document, and to the file name without its extension otherwise. A title made of nothing but category words – "Assessment Items", "Course Assessment Items", "Homework Questions" – is treated as no title, since every document in a course answers with the same one.
kind: Character: What the document is, in a few words, placed in the prompt. The default suits question sets; say so if it is something else, e.g. "an R script containing exercises and their solutions".
verbosity: Integer [0, Inf): Verbosity level.
...: Additional arguments forwarded to rtemis.llm::generate().

Value

Assessment object.

This produces a draft

Extraction is the least reliable step in this package, and it is the only one whose input is arbitrary. A PDF exam with two-column layout, a table-heavy Word file, or a solutions script whose questions live in comments will all produce something – and it will sometimes be wrong. Read the result before mapping it. print() shows the counts; as.data.frame() shows the items.

The model is asked only to segment, and the schema settles where one item ends: items is an array, so a boundary is structural rather than a delimiter the model has to remember to emit, and a backend with constrained decoding cannot return anything else. What can be decided by pattern is decided by pattern: points is read from @grade 1/1, (5 points) or [10 pts], bonus from the words "bonus" and "extra credit". The inline markdown a Word document arrives wrapped in is removed before the model reads it and again from the items it returns, so no span or emphasis mark reaches the text an item is scored on.

Every item stands alone

Each item is scored on its own against a session objective, so an item that needs a neighbor to be understood cannot be scored. A multi-part question is split into its separately-answerable parts, and each part repeats what it needs: the stem the parts share, the study they concern, the table they read from. A label with nothing after it – a table row, a blank left for the student – is not an item, and is dropped where one arrives anyway.

Author

EDG

Examples

if (FALSE) { # \dontrun{
  a <- extract_assessment(
    "data/epi207/4_5_DAGs_Homework_Questions_POST.docx",
    llmconfig = config_Ollama("gemma4:12b", temperature = 0, think = FALSE)
  )
  as.data.frame(a)
} # }