8 Classification and Consistency

Classification assigns taxonomy levels to the supplied text. Consistency checks then compare those levels among aligned items. The classification is a model judgment; the subsequent rank comparisons are deterministic calculations on the recorded classifications and alignment scores.

8.1 Classifying a course

library(curriculum)
library(rtemis.llm)

course <- read_course("objectives.yaml", "assessments.yaml")
classification_config <- config_Ollama("your-llm-model", temperature = 0)

levels <- classify_objectives(
  course_objectives = course[["objectives"]],
  session_objectives = course[["sessions"]],
  assessments = course[["assessments"]],
  course_id = "demo",
  llmconfig = classification_config,
  taxonomy = "blooms_revised",
  repeats = 1L,
  rationale = TRUE
)

items <- as.data.frame(levels)
items[, c("id", "kind", "text", "level", "rank", "verb", "issue")]

These standalone functions use the argument name llmconfig. classify_objectives() generates identifiers compatible with the mapping functions when the input groups, item order, and course_id match. It includes assessment items when supplied.

classify_levels() provides the general interface for a character vector:

levels <- classify_levels(
  c("List the assumptions of a linear model.",
    "Critique a model against its stated assumptions."),
  llmconfig = classification_config,
  kind = "course objective",
  course_id = "demo"
)

Supply ids explicitly when these classifications will be joined to an existing mapping. Automatically generated item identifiers from classify_levels() are not the course/session identifiers generated by classify_objectives().

8.2 Taxonomies and repeated ratings

list_taxonomies()
taxonomy("blooms_revised")
taxonomy("solo")

The default taxonomy is the cognitive process dimension of revised Bloom’s taxonomy. SOLO is also available. The conceptual distinctions are described in Bloom’s taxonomy.

Repeated classifications are combined using the taxonomy’s panel rule, such as "median_rank" for an ordered scheme. The ratings list column retains individual rank judgments. confidence summarizes agreement among ratings; it is not an estimated probability of a correct classification. Inspect missing levels and the underlying statements when interpreting summaries.

8.2.1 Defining a taxonomy

two_level <- create_taxonomy(
  id = "recall_or_application",
  label = "Recall or application",
  citation = "Unpublished example taxonomy.",
  levels = c("Recall", "Application"),
  definitions = c(
    "Retrieve a stated fact.",
    "Carry out a procedure using supplied information."
  ),
  verbs = list(c("list", "name"), c("compute", "execute")),
  rubric = "Classify the performance required by this objective."
)

Pass the returned object as taxonomy = two_level. Definitions, ranks, and verb examples correspond by position. The example is illustrative and is not a validated replacement for a published taxonomy. Use ordered = FALSE for categories without a meaningful rank order; check_level_consistency() requires an ordered taxonomy and rejects unordered schemes.

8.3 Reviewing objective wording

list_issues()
objective_issues(levels)

The classifier reports no_verb, category_label, unobservable, compound, and vague_object, or none. These labels describe the package’s review conventions. They support editorial review and do not independently establish the quality of an objective.

With rationale = TRUE, responses can include a rationale and a suggested revision. Review suggestions against the intended learning activity and source material. The original text remains in the classification. Several requested performances can receive a highest-required-level classification while also being flagged as compound.

8.4 Comparing levels across aligned pairs

For a complete run, the comparisons are already in cm@consistency and cm@assessment_consistency. They can also be recomputed from the components:

consistency <- check_level_consistency(
  cm@mapping,
  cm@levels,
  threshold = 0.5,
  tolerance = 0L
)
gaps <- level_gaps(consistency)

The comparison uses delta = to_rank - from_rank for edges meeting the strength threshold:

Condition	`flag`	Interpretation
`delta < -tolerance`	`under`	Candidate rank is below the criterion’s permitted range
`abs(delta) <= tolerance`	`ok`	Rank difference is within the selected tolerance
`delta > tolerance`	`over`	Candidate rank is above the criterion’s permitted range
Either rank is missing	`NA`	Available classifications do not support the comparison

tolerance = 1L treats adjacent ranks as consistent. Its use is an analytical choice that should be reported. Ordinal ranks indicate order; their numerical differences are not calibrated distances in cognitive difficulty.

Lower-level session objectives can provide preparation for a higher-level course objective. An assessment above an objective’s rank may indicate broader task demands, incomplete objective wording, or classification uncertainty. Interpret flags in relation to the full curriculum and the task’s conditions.

8.4.1 Criterion-level summaries

gaps[, c(
  "from_id", "from_level", "n_aligned", "max_to_level", "reached",
  "n_under", "n_ok", "n_over"
)]

level_gaps() retains criteria with no aligned candidate and reports n_aligned = 0. Its reached condition is a direct comparison: max_to_rank >= from_rank, with both ranks known. It does not incorporate tolerance. Increasing tolerance can change under, ok, and over counts without changing reached.

Missing classifications can produce reached = FALSE even when aligned text is present. Distinguish that case from a classified candidate set entirely below the criterion and from a criterion with no aligned items.

For assessments, apply the same functions to cm@assessment_mapping and cm@levels. These calculations describe the documented alignment chain; they do not establish that students have attained an objective.