HealthTasks Research · v0.1 · 26 Sep 2026
HealthTasks Agents Benchmark
This benchmark asks a practical question: does a model have to lead a general frontier benchmark to be the right model inside the HealthTasks Agents harness? Every model here gets the same prompt, the same dataset, and the same tool loop. Accuracy is checked against the source evaluations. Cost is what it takes to run the job again.
Accuracy and cost decide how much intelligence an institution actually gets. GPT-6 Luna stays with the other frontier models on this Agents task, finishes faster, and costs a fraction as much. OpenAI reports lower error rates on GPT-6 Luna than on the prior Luna generation. That is why HealthTasks now ships GPT-6 Luna as the default agent, and why more usage, not a more expensive frontier score, is the path to more intelligence.
- 1
- task
- 119
- evaluations
- 5
- models
- 5
- runs
Leaderboard
sorted by task score · gateway cost summed
| # | Model | Task | Cost | Speed | Depth | Out tok | Steps |
|---|---|---|---|---|---|---|---|
| 1 | gpt-6-luna OpenAI | 100 | 100 $0.009 | 77 34.3s | 0 | 3.1k | 6 |
| 2 | gemini-3.8-flash Google | 100 | 4 $0.253 | 43 61.6s | 30 | 3.7k | 9 |
| 3 | grok-4.6 xAI | 100 | 4 $0.251 | 35 75.0s | 0 | 5.8k | 6 |
| 4 | gpt-6-sol OpenAI | 40 | 4 $0.231 | 41 65.0s | 0 | 4.6k | 4 |
| 5 | grok-4.7 xAI | 0 | 6 $0.150 | 100 26.4s | 0 | 1.7k | 4 |
Task, cost, and speed are each 100 points. Depth is 30 extra points and does not change the rank. Cost is 100 × cheapest / this run. Speed is 100 × fastest / this run. Input tokens: Gemini 319,053 · Grok 4.6 193,188 · Luna 6 185,763 · Sol 100,172 · Grok 4.7 115,319.
Task
The agent receives a fixed institution in the HealthTasks development environment and the prompt below. It may query submitted evaluations, write a short answer, and attach a canvas.
evaluation trends over time
A full task score is a correct monthly trend: scores and counts by month, with inactive months left as gaps. The prompt does not ask for an overall average, so a missing or rounded headline is not scored. Cost and speed are scored separately: cheaper and faster runs get more points because they support follow-up questions and more agent usage. Classroom and template splits add depth. They do not raise the task score.
Method continues the July educational-analytics bakeoff and the June clinical-analytics bakeoff. Those runs compared two models. This v0.1 bakeoff holds the harness fixed and swaps the model.
Ground truth
| Submissions | 119 |
|---|---|
| Overall average | 94.11% |
| April 2025 | 43 · 96.28% |
| June 2025 | 14 · 84.02% |
| September 2026 | 9 · 97.04% |
| ICU | 78 · 95.72% |
| Med Surg | 32 · 90.42% |
| Unassigned | 8 · 92.50% |
| Classroom “1” | 1 · 100% |
| Daily Eval | 43 · 94.26% |
| Preceptor | 34 · 92.94% |
| Location | 32 · 98.13% |
Scoring
| Score | Criterion | Pts | Rule |
|---|---|---|---|
| Task | Monthly trend | 100 | The prompt is “evaluation trends over time.” Month-by-month scores and counts match the source, with inactive months omitted = 100. Correct quarter totals only = 40. Any stated month with the wrong volume = 0. An overall average is not required and is not scored. |
| Cost | Gateway spend | 100 | 100 × cheapest run in this bakeoff / this run. Cheaper inference scores higher because the same answer can be repeated more often. |
| Speed | Persisted duration | 100 | 100 × fastest run in this bakeoff / this run. Faster answers score higher because they support follow-up questions in the same session. |
| Depth | Classroom split | 15 | ICU, Med Surg, Unassigned, and classroom “1” match the source. Not required by the prompt. |
| Depth | Template split | 15 | Daily Eval, Preceptor, and Location match the source. Not required by the prompt. |
Notes
gpt-6-luna
task 100 · cost 100 · speed 77 · $0.009
The monthly score series and the submissions-by-month chart both match the source. Inactive months are omitted. No classroom or template split.
gemini-3.8-flash
task 100 · cost 4 · speed 43 · $0.253
The monthly series matches the source. Only run with a correct classroom split and a correct template split.
grok-4.6
task 100 · cost 4 · speed 35 · $0.251
The monthly chart and counts match the source. No classroom or template split.
gpt-6-sol
task 40 · cost 4 · speed 41 · $0.231
Quarterly means and counts match. The prompt asked for trends over time, and month grain is omitted.
grok-4.7
task 0 · cost 6 · speed 100 · $0.150
The monthly volumes are wrong, including April 2025 at 41 instead of 43. Classroom counts do not match.
Limitations
- v0.1 is one task. A model that wins here can lose on a different Agents prompt.
- Each model has one finished run. That is a point estimate, not a confidence interval.
- The prompt is scored as written. An overall average is not required.
- Cost and latency include the tool loop, not a single completion.
Conclusions
Three models answered the question correctly: GPT-6 Luna, Gemini 3.8 Flash, and Grok 4.6. Each recovered the monthly trend in the source data. For HealthTasks Agents, that is the bar that matters. A public coding leaderboard is not.
GPT-6 Luna is now the default. It finished in 34 seconds at about a cent. Gemini and Grok 4.6 reached the same trend at about twenty-five cents and took longer. Cheaper, faster runs are the ones faculty will actually use again: a follow-up question, a second classroom, a weekly check. More usage is more intelligence.
Use Gemini when you want the extra classroom and template breakdown. Skip Grok 4.6: it costs Gemini’s price without Gemini’s extra cuts. Skip GPT-6 Sol: it summarized by quarter instead of by month. Skip Grok 4.7: it was the fastest run, and the month counts were wrong.
| Model | Recommend for | Against the other options |
|---|---|---|
| gpt-6-luna | Default | Top recommendation and the default HealthTasks agent. Task 100, cost 100, speed 77. $0.009 in 34.3s, with a matching monthly series and a volume chart. That is the run you can afford to ask again. OpenAI reports lower error rates on GPT-6 Luna than on the prior Luna generation. |
| gemini-3.8-flash | Extra depth | Task 100, depth 30, cost 4, speed 43. The extra cuts are real. They are not required by the prompt, and they cost many times the Luna default. Use Gemini when those cuts are the product. |
| grok-4.6 | Neither | Task 100, cost 4, speed 35. The monthly series is correct at $0.251, in Gemini’s cost band, without Gemini’s depth. Luna is the better value choice. |
| gpt-6-sol | Neither | Task 40, cost 4, speed 41. Same family as Luna, coarser answer, much higher cost. It answered in quarters. The prompt asked for trends over time. |
| grok-4.7 | Neither | Speed 100, cost 6, task 0. The fastest run, and the monthly volumes are wrong, including April 2025 at 41 instead of 43. Speed without a correct trend is not usable. |