Map a course, end to end — map_curriculum • curriculum

Runs the whole pipeline on one course: classifies every objective against a taxonomy, scores every (course objective, session objective) pair, compares the two, and collects the objectives whose wording undermines the result.

Usage

map_curriculum(
  course,
  llmconfig,
  course_id = NULL,
  scoring_system = "ioc",
  taxonomy = "blooms_revised",
  block = c("pairwise", "joint"),
  repeats = 1L,
  rationale = TRUE,
  threshold = 0.5,
  tolerance = 0L,
  retry = TRUE,
  check = TRUE,
  verbosity = 1L,
  outdir = NULL,
  ...
)

Arguments

course: Character or List: The course to map – the list read_course() returns, or a path to a YAML file of objectives, which is read for you. Pass the list where the assessments live in their own file, since the path form reads one file only.
llmconfig: rtemis.llm::LLMConfig: Model configuration, used for both the classification and the mapping.
course_id: Character: Course identifier, used to build stable ids. Defaults to the file's course_name, lowercased with spaces removed.
scoring_system: Character or ScoringSystem: Scoring system. See list_scoring_systems().
taxonomy: Character or Taxonomy: Classification scheme. See list_taxonomies().
block: Character {"pairwise", "joint"}: Prompt block structure for the mapping.
repeats: Integer [1, Inf): Independent raters per judgement, for both steps. Only buys something if the raters can differ – at temperature = 0 they cannot.
rationale: Logical: Whether to ask for justifications. Required for the rewrite suggestions in issues, so the report is thinner without it.
threshold: Numeric [0, 1]: Minimum strength for an edge to count as aligned, in the consistency check and the report.
tolerance: Integer [0, Inf): Rank difference treated as consistent.
retry: Logical: Whether to re-score pairs that failed.
check: Logical: Whether to run check_objectives() first and stop on a finding of severity "error". The findings are recorded in run$checks either way.
verbosity: Integer [0, Inf): Verbosity level.
outdir: Character: Directory to write the run to, or NULL for a run that keeps nothing until it finishes. Each step is saved as it completes and each criterion's scores as they are produced, so a run that ends early – a dropped connection, a sleeping laptop – is resumed by calling this again with the same arguments. See the "Resuming a run" section.
...: Additional per-call arguments forwarded to rtemis.llm::generate(), e.g. temperature, seed.

Value

CurriculumMapping object.

What runs, in what order

check_objectives() runs first, before any call is made. It is deterministic and free, so it is the only step that can refuse the file cheaply: a finding of severity "error" – an empty objective, two sessions sharing a key – stops the run, and check = FALSE maps the file as it stands. Everything it found is kept in run$checks whether or not it stopped anything, so a result always records the state of the file it was built from.

Classify every course and session objective. This is deliberately first. It costs one call per objective against n_criteria x n_candidates for the map, and it is the step that reveals whether the objectives are written as outcomes at all – which decides how much the rest can be trusted. Finding that out after 30 calls beats finding it out after 162.
Map every (course objective, session objective) pair with the chosen scoring system, then re-score whatever came back NA if retry = TRUE.
Map assessments, if the course carries an assessments entry: every (session objective, assessment item) pair, same engine and scoring system, with the relation changed. Skipped entirely when there are none, and the result says so rather than reporting an empty map.
Compare the levels across the surviving edges of each direction, and roll up – per course objective for the first, per session objective for the second.

Both model steps use the same llmconfig, so a run is one model unless you say otherwise. The deterministic tier stops at data problems: whether an objective is written as an outcome is the model's judgement here, and plan/12_objective_review.md records why the string-inspection version of that was measured and parked.

Resuming a run

outdir makes a run restartable. Each step is written as it finishes – <course_id>_levels.rds, <course_id>_course_to_session.rds, <course_id>_session_to_assessment.rds, <course_id>_curriculum.rds – and within the two scoring steps each criterion's rows are written as they are produced, under <course_id>_<relation>_parts/. Calling map_curriculum() again with the same arguments reads all of that back and scores only what is missing.

A part is reused only where the run that wrote it is the run being resumed: the same scoring system, block, raters, model, objectives and objective text. Change any of them and the run stops rather than filling half a matrix under one configuration and half under another. Editing an objective is enough – its id survives the edit, and a score against the old wording is not a score against the new one.

The parts directory is a cache, so deleting it costs only the calls to rebuild it.

Reading the result

print() gives the verdict; report_curriculum() writes the document. The components are the ordinary objects – x@mapping is a Mapping, x@levels a Classification – so as.matrix(), as_sankey(), objective_issues() and the rest apply unchanged.

Author

EDG

Examples

if (FALSE) { # \dontrun{
  course <- read_course(
    "data/epi207/objectives.yaml",
    "data/epi207/assessments.yaml"
  )
  cm <- map_curriculum(
    course,
    llmconfig = config_Ollama("gemma4:e4b", temperature = 0, think = FALSE)
  )
  cm
  report_curriculum(cm, "epi207_report.html")
} # }