Skip to contents

Map course objectives to session objectives

Source: R/05_map.R
map_course_to_sessions.Rd

Scores every (course objective, session objective) pair with a pluggable scoring system and returns the long form plus everything needed to reproduce the run.

Usage

map_course_to_sessions(
  course_objectives,
  session_objectives,
  course_id,
  llmconfig,
  scoring_system = "ioc",
  block = c("pairwise", "joint"),
  repeats = 1L,
  rationale = TRUE,
  verbosity = 1L,
  outdir = NULL,
  ...
)

Arguments

course_objectives

Character: Course objectives.

session_objectives

List of character vectors: Session objectives, named by session.

course_id

Character: Course identifier, used to build stable ids.

llmconfig

rtemis.llm::LLMConfig: Model configuration, from config_Ollama(), config_OpenAI() or config_Anthropic(). One object carries the model, temperature, endpoint and key handling, so an OpenAI-compatible gateway needs no special support here – see the OpenRouter example. Only the model name, backend, base URL and temperature are recorded in run; an API key held on the config is never persisted.

scoring_system

Character or ScoringSystem: Scoring system. See list_scoring_systems().

block

Character {"pairwise", "joint"}: Prompt block structure.

repeats

Integer [1, Inf): Independent raters per pair. IOC and CVI are panel instruments and are defined over several raters; with repeats = 1 they still work but their thresholds do not mean what they were calibrated to mean.

rationale

Logical: Whether to ask for a one-sentence justification per cell. Rationales are what make a mapping reviewable, so this defaults to TRUE – but they dominate generation time under constrained decoding, since the free-text field is far longer than the label it explains. Set FALSE for a fast pass when only the matrix is wanted.

verbosity

Integer [0, Inf): Verbosity level.

outdir

Character: Directory to write resumable parts to, or NULL for a run that keeps nothing. Each criterion's scores are written as they are produced, and a later run over the same objectives with the same settings reads them back instead of scoring them again.

...

Additional per-call arguments forwarded to rtemis.llm::generate(), e.g. temperature, seed.

Value

Mapping object.

Block structure

block decides how many criteria the model sees at once, and it is not a free choice.

  • "pairwise" (default) asks one isolated question per pair. Each call is short – one label, one rationale – so despite issuing n_criteria times as many calls it is competitive on wall time, and it buys three things joint scoring cannot. Judgements are independent, which makes repeats a genuine panel rather than correlated draws within a row – the thing IOC and CVI are actually defined over. The per-pair cache invalidates one cell at a time. And because the criterion sits in the system prompt while the run is criterion-major, the cached prefix is reused across every candidate.

  • "joint" shows all course objectives in one call and rates the session objective against each. The comparative context can improve discrimination: an isolated judgement has nothing to be relatively stingy against, so pairwise runs are worth checking for a dense, undifferentiated matrix.

The distinction matters most across courses. Joint scoring induces an implicit normalization within a call: shown a list, a model spreads its ratings across it, so a course with 20 objectives will produce systematically different marginals from one with 6. That is harmless within a course and misleading between them, which is the other reason "pairwise" is the default. Never pool joint-mode matrices with different n_criteria; block and n_criteria are both recorded in run so the comparison can be checked rather than assumed.

Duplicate candidates

A candidate text that appears more than once – a module taught over several sessions, a column copied in the source sheet, an objective two sessions share – is scored once and the answer written to every id that carries it. The map keeps its full width, and the calls made are n_criterion_texts x n_candidate_texts in pairwise mode and n_candidate_texts in joint; run records the distinct-text counts alongside n_criteria and n_candidates, so what was asked and what was returned stay separable. Criteria are deduplicated in pairwise mode only: joint mode shows every criterion in one call, where a copy costs no extra call and removing it would change the list the model normalizes across.

With repeats > 1 the copies share the panel drawn for their text: it is the same question about the same words, so the raters are not asked twice.

Author

EDG

Examples

# Requires a running Ollama server and the named model
if (FALSE) { # \dontrun{
obj <- read_objectives("objectives.yaml")
m <- map_course_to_sessions(
  obj[["objectives"]],
  obj[["sessions"]],
  course_id = "datasci213",
  llmconfig = config_Ollama("granite4.2:8b", temperature = 0, think = FALSE),
  scoring_system = "ioc"
)
as.matrix(m)

# Via OpenRouter with zero data retention
m <- map_course_to_sessions(
  obj[["objectives"]],
  obj[["sessions"]],
  course_id = "datasci213",
  llmconfig = config_OpenAI(
    "openai/gpt-4o-mini",
    temperature = 0,
    base_url = "https://openrouter.ai/api/v1",
    api_key_env = "OPENROUTER_API_KEY",
    zero_data_retention = TRUE
  ),
  scoring_system = "ioc"
)
} # }