10  Comparing Mappings

compare_mappings() compares two completed Mapping or CurriculumMapping objects. It summarizes agreement, categorical probabilities where available, and recorded computation time without making model calls.

10.1 Comparing two runs

Save runs from different model configurations or request groupings in separate directories. Preserve the course input, identifiers, and scoring definitions when the purpose is to compare model behavior.

run_a <- readRDS("mapping_a.rds")
run_b <- readRDS("mapping_b.rds")

comparison <- compare_mappings(run_a, run_b, threshold = 0.5)
comparison$summary
comparison$labels
comparison$probabilities
comparison$timing

Pairs are matched by course, relation, and item identifiers. The function checks matched text and scoring definitions. Duplicate pair identifiers, changed text under a matched identifier, and incompatible scoring definitions raise errors. Different input coverage is retained in unmatched-pair counts.

Whole-course comparisons report each mapping direction separately. Individual mapping components can also be compared:

compare_mappings(run_a@mapping, run_b@mapping)

10.2 Agreement summaries

Output	Interpretation
matched, unmatched_x, unmatched_y	Pair matching and input coverage
failed_x, failed_y, both_scored	Missing scores and the comparison denominator
label_agreement	Proportion of matched, labeled pairs with identical labels
kappa	Cohen’s kappa for the paired categorical labels
kendall_tau_b	Rank association between the two strength vectors, accounting for ties
jaccard	Intersection divided by union of thresholded edge sets among pairs scored in both runs

Agreement statistics use the eligible matched pairs. Undefined statistics are NA, for example when a rank correlation lacks variation or both thresholded edge sets are empty. Review matching and failure counts alongside the metrics.

Separate thresholds can be supplied:

compare_mappings(run_a, run_b, threshold = 0.5, threshold_y = 0.7)

This changes the edge-set comparison, not the underlying scores or categorical label agreement. A threshold comparison remains subject to the scoring scale’s resolution; see Scoring alignment.

10.3 Probability summaries

The probability table summarizes selected-label probabilities and entropy concentration for runs carrying complete categorical distributions. A concentrated distribution indicates that probability mass is concentrated on fewer categories. It does not establish agreement with a content expert.

The labels table reports the distribution of assigned labels in each run. Consider it alongside agreement: two runs can agree frequently while assigning nearly every pair to the same category.

10.4 Timing and checkpoint reuse

The timing table distinguishes recorded computation, retry time, current-run elapsed time, and reused parts. A resumed part retains its original computation duration. compute_seconds can therefore exceed the elapsed time of the resuming call. Historical objects without timing records have missing values.

Timing describes each run’s complete recorded scoring work, including work on unmatched pairs. It is not a speed estimate restricted to the pairs used in the agreement calculation. Interpret differences in light of model, hardware, request grouping, input length, failures, and checkpoint reuse.

10.5 Interpretation

These comparisons describe concordance and operational behavior. Agreement between models does not establish accuracy, and probability summaries do not estimate calibration. Evaluating either requires appropriate reference judgments and a separate evaluation design. Human-labeled evaluation is optional work and is not required to run or compare mappings.