5 Scoring Alignment

A scoring system defines the judgment made for each criterion–candidate pair. Its rubric, labels, native values, and alignment strengths determine how results are interpreted. curriculum applies the same scoring interface to language models and categorical decision models.

5.1 Available scoring systems

library(curriculum)

list_scoring_systems()
scoring_system("ioc")

Identifier	Construct	Native scale	Interpretation of strength
`ioc`	Congruence	Congruent: 1; unclear: 0; incongruent: −1	Support on a nonnegative scale; unclear and incongruent ratings each contribute zero support
`cvi`	Relevance	Four categories with values 1–4	Proportion of available ratings at 3 or 4
`binary`	Congruence	Aligned: 1; not aligned: 0	Support derived from binary ratings
`support_ladder`	Contribution toward an objective	Five categories with values 0, 0.25, 0.5, 0.8, and 1	Anchored contribution score
`continuous`	Congruence	A directly generated number from 0 to 1	The generated numerical score

ioc is the default. Its rubric asks whether the candidate serves the criterion. support_ladder also represents prerequisite contributions; it is an unvalidated scale defined by the package. The continuous scale is an unvalidated numerical baseline. A scale’s historical use with human raters does not establish the validity of model-generated ratings for a particular course.

5.1.1 Values, strengths, and thresholds

value retains the scoring system’s native value. strength is the support measure in [0, 1] used for thresholding and visualization. For IOC, these quantities differ:

Single rating	`value`	`strength`
Congruent	1	1
Unclear	0	0
Incongruent	−1	0

For repeated ratings, native values are combined by the scoring system’s panel rule. Interpolated strength uses the mean of available native ratings. CVI instead uses the proportion of ratings meeting its relevance cut. cvi_threshold(n) returns the package’s recommended CVI threshold for a panel of size n; repeated calls to one model are not equivalent to independent content experts.

An edge is treated as aligned when strength >= threshold. With a single IOC rating, any positive threshold at most 1 selects the congruent category. Changing the threshold within that interval does not refine the three-category judgment.

5.2 Language-model scoring

library(rtemis.llm)

scoring_config <- config_Ollama(
  model_name = "your-llm-model",
  temperature = 0
)

mapping <- map_course_to_sessions(
  course[["objectives"]],
  course[["sessions"]],
  course_id = "demo",
  scoring_config = scoring_config,
  scoring_system = "ioc",
  block = "pairwise",
  repeats = 1L,
  rationale = TRUE
)

The example uses the course prepared in Reading objectives. Replace the model name with an available model. A structured response supplies each requested score and, when enabled, its rationale.

repeats requests multiple ratings. The scoring system combines their values, and the individual ratings remain available in the result. Repeated calls can describe variation under the specified settings; agreement among them does not establish accuracy. A temperature of zero can reduce sampling variation without guaranteeing identical outputs across all providers and execution conditions.

5.3 Decision-model scoring

Decision models select among the scoring system’s categories. Configure the scoring stage separately from the language model used for classification:

classification_config <- config_Ollama("your-llm-model", temperature = 0)
scoring_config <- config_OllamaDecision("your-decision-model")

cm <- map_curriculum(
  course,
  classification_config = classification_config,
  scoring_config = scoring_config,
  course_id = "demo",
  scoring_system = "ioc",
  outdir = "runs/demo_decision"
)

Use a model supported by the decision interface described in the rtemis.llm decision-model guide. Classification and assessment extraction continue to use an LLMConfig.

Decision scoring requires:

A categorical scoring system with 2–26 levels.
One rating (scoring_repeats = 1L in map_curriculum()).
No scoring rationale (scoring_rationale = FALSE).
An empty scoring_options list.

These are the defaults when scoring_config is a DecisionConfig. Continuous scoring is incompatible with the categorical decision interface. Classification can independently request repeated ratings and rationales.

5.3.1 Probability fields

Decision results retain the selected label, its probability p, and the full named probabilities vector. value and strength come from the selected category; they are not probability-weighted averages of category values.

edges <- as.data.frame(cm@mapping)
edges[, c("from_id", "to_id", "label", "value", "strength", "p")]
edges[["probabilities"]][[1L]]

confidence retains its panel-agreement meaning and is missing for a single rating. Selected-label probability and entropy concentration describe the model’s categorical distribution. They do not establish calibration or the probability that the curriculum judgment is correct. Responses with incomplete or invalid distributions are retained as failed scores.

5.4 Pairwise and joint requests

Setting	Request organization
`block = "pairwise"`	One criterion–candidate pair per request
`block = "joint"` with an LLM	One candidate evaluated against all criteria in a structured response
`block = "joint"` with a decision model	One candidate evaluated against all criteria, in requests containing at most 64 questions

Pairwise scoring is the default. Both modes deduplicate candidate text. Pairwise mode also deduplicates criterion text. Results are expanded to all original item identifiers, so repeated text remains represented in the output.

Joint requests change the context in which each judgment is made and can change the scores. Fewer calls alone do not establish better speed or equivalent results. Compare the modes on the same course using mapping comparisons.

5.5 Custom scoring systems

Define a scoring system by supplying its rubric and aligned vectors of labels, descriptions, values, and strengths:

coverage_scale <- create_scoring_system(
  id = "three_level_coverage",
  label = "Three-level coverage",
  citation = "Unvalidated example scale.",
  construct = "coverage",
  scale_type = "ordinal",
  rubric = "How much of the criterion does this candidate address?",
  range = c(0, 1),
  levels = c("full", "partial", "none"),
  level_descriptions = c(
    "Addresses the criterion in full.",
    "Addresses part of the criterion.",
    "Does not address the criterion."
  ),
  values = c(1, 0.5, 0),
  strengths = c(1, 0.5, 0)
)

Pass the returned object as scoring_system = coverage_scale. The constructor does not add it to the built-in registry. Preserve its definitions with the analysis. score_prompt(), score_schema(), score_parse(), score_panel(), and score_strength() expose the scoring operations for inspection; their signatures are documented in the API reference.