Define a scoring system — create_scoring

The functional constructor for a scoring system. Use it to add a scoring system the package does not ship; the registered ones are fetched with scoring_system() and listed by list_scoring_systems().

Usage

create_scoring_system(
  id,
  label,
  citation,
  scale_type,
  rubric,
  range,
  levels = NULL,
  level_descriptions = NULL,
  values = NULL,
  strengths = NULL,
  construct = "congruence",
  panel = "mean",
  strength_rule = "interpolate",
  strength_threshold = NULL,
  recommended_for = "course_to_session"
)

Arguments

id: Character: Short identifier, unique among scoring systems.
label: Character: Human-readable name.
citation: Character: Short citation, or a statement that the scale is unvalidated. Say which – an unvalidated construct that borrows a citation is worse than one that admits it has none.
scale_type: Character {"binary", "trichotomous", "ordinal", "continuous", "probability"}: Shape of the emitted value. "continuous" asks the model for a number and needs no levels.
rubric: Character: Framing sentence placed above the level list in the system prompt.
range: Numeric: Length-2 range of the native value.
levels: Character: Permitted labels, ordered most to least support. NULL for a continuous scale.
level_descriptions: Character: One rubric line per level.
values: Numeric: Native value per level.
strengths: Numeric [0, 1]: Support per level.
construct: Character {"congruence", "coverage", "relevance"}: What the scale claims to measure. Congruence asks whether the candidate matches the criterion, coverage whether it contributes toward it; these are different questions and suit different mapping directions.
panel: Character {"mean", "median"}: How repeated native ratings combine into the reported value.
strength_rule: Character {"interpolate", "proportion_at_or_above"}: How support is derived from the raters' ratings.
strength_threshold: Numeric: Native-scale cut, required when strength_rule = "proportion_at_or_above".
recommended_for: Character: Relations this scale suits.

Value

ScoringSystem object.

Details

A scoring system is data. Its behavior – building the prompt and schema, parsing the response, combining raters – is supplied by the package, so defining one means describing a scale rather than writing functions.

levels, level_descriptions, values and strengths are four positionally-aligned vectors describing one level each. values is the scoring system's own scale and strengths is support in [0, 1]; they are separate because a native scale is not always a support scale. IOC's 0 (unclear) and -1 (incongruent) are different ratings but both mean no support, and the long form keeps the signed value while visualizations use the strength.

Author

EDG

Examples

create_scoring_system(
  id = "thirds",
  label = "Three-level coverage",
  citation = "Unvalidated: example only.",
  scale_type = "ordinal",
  rubric = "How much does this candidate advance the criterion?",
  range = c(0, 1),
  levels = c("full", "partial", "none"),
  level_descriptions = c("Fully.", "Partly.", "Not at all."),
  values = c(1, 0.5, 0),
  strengths = c(1, 0.5, 0)
)

#> <curriculum::ScoringSystem>
#>  @ id                : chr "thirds"
#>  @ label             : chr "Three-level coverage"
#>  @ citation          : chr "Unvalidated: example only."
#>  @ construct         : chr "congruence"
#>  @ scale_type        : chr "ordinal"
#>  @ levels            : chr [1:3] "full" "partial" "none"
#>  @ level_descriptions: chr [1:3] "Fully." "Partly." "Not at all."
#>  @ values            : num [1:3] 1 0.5 0
#>  @ strengths         : num [1:3] 1 0.5 0
#>  @ range             : num [1:2] 0 1
#>  @ rubric            : chr "How much does this candidate advance the criterion?"
#>  @ panel             : chr "mean"
#>  @ strength_rule     : chr "interpolate"
#>  @ strength_threshold: NULL
#>  @ recommended_for   : chr "course_to_session"