library(rtemis.llm)5 Decision Models
A decision model answers typed questions about a passage with a probability for every option, and writes no text. There are two kinds of question:
- a choice among 2 to 26 named options, each with what it means;
- a noul, which asks how true a statement is. The answer is a probability.
The model cannot answer outside the options it was given, so its answers need no repair. Every answer carries a probability, which you can rank by, threshold, or average.
Ollama serves decision models locally, for example clef-flash. OpenRouter serves others, for example typesafe/jev-1.13, with an API key.
ollama_list_decision_models()[1] "clef-flash:latest"5.1 Create a DecisionModel
A configuration says where the model runs:
config_OllamaDecision()for a local model on Ollama;config_OpenRouterDecision()for a model on OpenRouter. The key is read fromOPENROUTER_API_KEY,api_key, or a Keychain item.
dm <- create_DecisionModel(config_OllamaDecision("clef-flash"))
dm<DecisionModel>
Model: clef-flash
Provider: Ollama
Base URL: http://localhost:11434
A decision model has no system prompt and no sampling options. Text that applies to every passage, such as the task or a rubric, goes in context, which is placed before each passage.
5.2 Ask questions
Questions are a decision model’s own interface. You write each question, and for a choice, what each option means. The model judges the passage against those meanings.
The examples below screen study abstracts. They ask for the study design and for whether the primary result is statistically significant.
abstracts <- c(
trial = paste(
"In this multicenter randomized controlled trial, 412 adults with type 2 diabetes were",
"assigned to semaglutide or placebo for 52 weeks. HbA1c fell by 1.4 percentage points with",
"semaglutide and by 0.2 with placebo (difference -1.2; 95% CI -1.5 to -0.9; p < 0.001)."
),
cohort = paste(
"We followed 12,840 nurses for 20 years to examine the association between rotating",
"night-shift work and incident breast cancer. Night-shift work was associated with a slightly",
"higher risk that was not statistically significant (HR 1.12; 95% CI 0.98 to 1.28)."
),
model = paste(
"We trained a convolutional neural network on 38,000 chest radiographs to detect",
"pneumothorax and evaluated it on an external test set of 2,100 radiographs read by three",
"thoracic radiologists. The model reached an AUC of 0.94 (95% CI 0.92 to 0.96)."
)
)qs <- list(
design = choice(
"Which study design does the abstract describe?",
c(
randomized_trial = "Participants are randomly assigned to the interventions being compared",
cohort = "A group is followed over time to relate exposures to later outcomes",
case_control = "People with and without an outcome are compared on past exposures",
cross_sectional = "Exposures and outcomes are measured at a single point in time",
diagnostic_accuracy = "A test or model is evaluated against a reference standard",
systematic_review = "Existing studies are identified and summarized or pooled"
)
),
significant = noul("Does the abstract report a statistically significant primary result?")
)
d <- decide(dm, abstracts[["cohort"]], qs)d<Decision>
Model: clef-flash
design: cohort (p = 0.957, confidence 0.863)
significant: noul 0.135
probabilities() returns every option’s probability:
probabilities(d) question option p chosen unsure
<char> <char> <num> <lgcl> <lgcl>
1: design cohort 0.956659546 TRUE FALSE
2: design systematic_review 0.012267256 FALSE FALSE
3: design case_control 0.011475782 FALSE FALSE
4: design cross_sectional 0.008184194 FALSE FALSE
5: design diagnostic_accuracy 0.005850910 FALSE FALSE
6: design randomized_trial 0.005562312 FALSE FALSE
7: significant true 0.135298251 FALSE FALSE
8: significant false 0.864701749 TRUE FALSEunsure flags an answer whose probabilities are spread out: a choice whose confidence is below 0.5, or a noul within 0.15 of 0.5. The provider’s confidence measures how concentrated the probabilities are, not how often the pick is right.
5.3 Ask the same questions of many passages
dmapply() asks the same questions about each element. It returns one table with an index column, and the wall time of each call in seconds.
res <- dmapply(abstracts, dm, questions = qs)res[chosen == TRUE] index question option p chosen unsure
<int> <char> <char> <num> <lgcl> <lgcl>
1: 1 design randomized_trial 0.9712882 TRUE FALSE
2: 1 significant true 0.9463476 TRUE FALSE
3: 2 design cohort 0.9566595 TRUE FALSE
4: 2 significant false 0.8647017 TRUE FALSE
5: 3 design diagnostic_accuracy 0.9492614 TRUE FALSE
6: 3 significant true 0.8376366 TRUE FALSEattr(res, "elapsed") trial cohort model
2.184 2.092 2.355 5.4 Fill a schema
generate() and dmapply() can also fill a closed schema. The result is a validated JSON document in the same shape as an LLM’s structured output. Use this when a decision model takes an LLM’s place in code that expects a document.
A schema is closed when every field is one of:
- a field with an
enum; - a boolean;
- an array whose
itemsis a field with anenum.
is_decidable() says whether a schema is closed, and lists each open field with the reason:
is_decidable(schema(
"StudySummary",
field("design", "Study design", enum = c("randomized_trial", "cohort", "case_control")),
field("n", "Number of participants", type = "integer"),
field("conclusion", "The authors' main conclusion")
))[1] FALSE
attr(,"open")
path reason
1 n an integer with no bounds, or more than 100 values
2 conclusion free text
attr(,"questions")
[1] NAThis schema is closed:
study <- schema(
"Study",
field(
"design",
"Study design",
enum = c(
"randomized_trial", "cohort", "case_control",
"cross_sectional", "diagnostic_accuracy", "systematic_review"
)
),
field(
"significant",
"Whether the primary result is statistically significant",
type = "boolean"
),
field(
"data",
"Kinds of data analyzed",
type = "array",
items = field("data_type", enum = c("imaging", "laboratory", "questionnaire", "genomic"))
)
)
dm_study <- create_DecisionModel(config_OllamaDecision("clef-flash"), output_schema = study)
msg <- generate(dm_study, abstracts[["model"]])msg[Decision] {"design":"diagnostic_accuracy","significant":true,"data":["imaging"]}
design p = 0.893
significant p = 0.883
data imaging 0.96, laboratory 0.00, questionnaire 0.00, genomic 0.00
probabilities(msg) path option p chosen unsure
<char> <char> <num> <lgcl> <lgcl>
1: design diagnostic_accuracy 0.892655230 TRUE FALSE
2: design cross_sectional 0.042228636 FALSE FALSE
3: design cohort 0.025346867 FALSE FALSE
4: design case_control 0.017007728 FALSE FALSE
5: design systematic_review 0.011680064 FALSE FALSE
6: design randomized_trial 0.011081477 FALSE FALSE
7: significant true 0.883470758 TRUE FALSE
8: significant false 0.116529242 FALSE FALSE
9: data imaging 0.960803722 TRUE FALSE
10: data laboratory 0.003694112 FALSE FALSE
11: data questionnaire 0.003503407 FALSE FALSE
12: data genomic 0.003322608 FALSE FALSEA batch returns one document per abstract:
dmapply(abstracts, dm_study) trial
"{\"design\":\"randomized_trial\",\"significant\":true,\"data\":[\"laboratory\"]}"
cohort
"{\"design\":\"cohort\",\"significant\":false,\"data\":[]}"
model
"{\"design\":\"diagnostic_accuracy\",\"significant\":true,\"data\":[\"imaging\"]}"
attr(,"validation")
Schema validation: valid 3; invalid 0; unavailable 0; not_validated 0
attr(,"errors")
[1] index message
<0 rows> (or 0-length row.names)
attr(,"elapsed")
trial cohort model
4.712 5.161 4.254 5.4.1 How a schema becomes questions
The schema is converted into questions before anything is sent:
- A field with an
enumbecomes a choice named after the field: “Which value of the field<name>(<description>) does the passage support?”. Each option means only its own label. - A boolean field becomes a noul.
- An array of
enumvalues becomes one noul per value, named<field>[<value>]. The values at or above 0.5 are kept. - A field that is not required adds a noul named
<field>?, which asks whether the passage gives the field at all. Below 0.5, the field is left out of the document.
as_questions() returns exactly those questions:
as_questions(study)$design
<Choice>
Which value of the field `design` (Study design) does the passage support?
randomized_trial
cohort
case_control
cross_sectional
diagnostic_accuracy
systematic_review
$significant
<Noul>
Is the field `significant` (Whether the primary result is statistically significant) true, according to the passage?
$`data[imaging]`
<Noul>
Does the field `data` (Kinds of data analyzed) include `imaging`, according to the passage?
$`data[laboratory]`
<Noul>
Does the field `data` (Kinds of data analyzed) include `laboratory`, according to the passage?
$`data[questionnaire]`
<Noul>
Does the field `data` (Kinds of data analyzed) include `questionnaire`, according to the passage?
$`data[genomic]`
<Noul>
Does the field `data` (Kinds of data analyzed) include `genomic`, according to the passage?
An enum value has no meaning beyond its label: case_control is all the model is told about that option. To give the options their meanings, edit the questions and ask them with decide():
qs_study <- as_questions(study)
qs_study[["design"]] <- qs[["design"]]
res_study <- dmapply(abstracts, dm, questions = qs_study)res_study[chosen == TRUE] index question option p chosen unsure
<int> <char> <char> <num> <lgcl> <lgcl>
1: 1 design randomized_trial 0.9323397 TRUE FALSE
2: 1 significant true 0.9455053 TRUE FALSE
3: 1 data[imaging] false 0.9933773 TRUE FALSE
4: 1 data[laboratory] true 0.8904418 TRUE FALSE
5: 1 data[questionnaire] false 0.9736278 TRUE FALSE
6: 1 data[genomic] false 0.9942642 TRUE FALSE
7: 2 design cohort 0.9617279 TRUE FALSE
8: 2 significant false 0.9010801 TRUE FALSE
9: 2 data[imaging] false 0.9929581 TRUE FALSE
10: 2 data[laboratory] false 0.9904390 TRUE FALSE
11: 2 data[questionnaire] false 0.6998288 TRUE FALSE
12: 2 data[genomic] false 0.9949054 TRUE FALSE
13: 3 design diagnostic_accuracy 0.9328380 TRUE FALSE
14: 3 significant true 0.8677437 TRUE FALSE
15: 3 data[imaging] true 0.9604992 TRUE FALSE
16: 3 data[laboratory] false 0.9957891 TRUE FALSE
17: 3 data[questionnaire] false 0.9964573 TRUE FALSE
18: 3 data[genomic] false 0.9965528 TRUE FALSE5.5 Which to use
Questions (decide()) |
Schema (generate()) |
|
|---|---|---|
| You get | probabilities per option | a validated JSON document, and the probabilities |
| Wording | yours | made from field names and descriptions |
| Option meanings | yes | no: an option means its label |
| Fits | judgments, scores, rankings, thresholds | code written for an LLM’s structured output |
Start with questions. Use a schema when the result must be a document.
5.6 Compare with an LLM
llmapply(), agentapply() and dmapply() all record each call’s wall time as attr(result, "elapsed"). A decision model and an LLM can then be compared on the same passages, each in its best setup.
llm <- create_Ollama("qwen3.5:9b", output_schema = study, temperature = 0)
llm_res <- llmapply(abstracts, llm)
dm_res <- dmapply(abstracts, dm, questions = qs)
summary(attr(llm_res, "elapsed"))
summary(attr(dm_res, "elapsed"))
