Doximity Benchmark: LLMs Get 2.3% of Doses Wrong

The result measures a specific risk of clinical AI. It does not prove that all medical use of LLMs is unsafe.

In this article you will see:

• How Doximity set up the dosing test.
• Which errors appeared most frequently.
• What DoseCheck checks without another LLM.
• What the benchmark allows us to conclude, and what it does not allow.

What Doximity evaluated in the dose benchmark

The evaluation brought together nine LLMs released in 2026 and dosing questions inspired by physician consultations. The publication reports 8,530 questions answered per model, covering 2,204 medications.

In total, Doximity analyzed 261,508 dosages. The company says the error rate of at least 2.3% is a conservative lower bound, not a complete count of all possible failures.

The questions included factors that change a recommendation, such as age, weight, renal or hepatic function and drug interactions. The test checked whether the models considered this information when it was in the prompt.

This focus matters because a dose does not depend only on the drug name. Formulation, route of administration, frequency and the person's conditions can also change what would be appropriate.

The test evaluated responses without external tools and with reasoning budget at the lowest level, as described in the chart. Therefore, it measures a specific configuration, not all ways to use the same models.

What Doximity's numbers reveal about dose errors

Doximity reports that at least 2.3% of dosages were wrong. The rate is presented as conservative because some failures may not be identified by the classification method used in the study.

Almost 20 responses per thousand contained a serious error, according to the EQUIP scale cited by the company. In a sensitivity analysis with the NCC MERP scale, about 30.5 responses per thousand fell into categories E to H.

The publication classifies 43.8% of detected errors as serious and 3.4% as potentially lethal.

These percentages describe the errors identified in the study, not the proportion of all responses given in real world use.

The most common pattern was failing to adjust the dose for patient characteristics, even when the question provided age, weight or clinical conditions. Doximity also observed higher risk with medications approved in 2024 or later.

In the analysis released, the most recent medications had 5.1 times the odds of error, after adjustments for model, question type and route of administration. The company reports a confidence interval of 3.9 to 6.5 for this estimate.

How DoseCheck verifies a dose without consulting another LLM

Alongside the study, Doximity introduced DoseCheck, a dosage verification layer for Doximity Ask. Instead of asking another model to review the answer, the system compares claims against a structured drug database.

The company describes a flow that extracts the drug, dose, route, frequency and relevant indications. Then it compares those fields with structured references and flags when it finds a match.

According to Doximity, the RxIndex database combines FDA labeling, guidelines and primary literature. The company reports coverage of 1,900 monographs and more than 36,000 dosing regimens, broken down by indication, population and route.

When the dose matches the records, the clinician sees a green seal and can open the full reference.

The company says the check returns in under 100 milliseconds. That speed is vendor reported, not an independent audit.

The publication itself stresses that DoseCheck does not replace clinical judgment.

The study and the product are different things. The results describe model responses, while the tool applies a deterministic check after generation.

What Doximity's benchmark does not prove about clinical AI

The announcement is a publication from Doximity itself, which also develops DoseCheck.

That context does not invalidate the data. Still, it matters when interpreting a test published by a company that offers a related solution.

The sample does not represent all models, versions, medical questions or flows with search and tools.

The result does not predict the error rate of a specific product. For that, the evaluation would need to be repeated in that system's configuration and cases.

This caution applies to benchmarks in general. In the SnowPR test with real world coding tasks, the team defines what can be generalized.

To compare test types, see the analysis of the Artificial Analysis Index v5.

For those building applications, the lesson is to separate probabilistic tasks from exact checks.

A model can summarize evidence. Units, limits, conversions and dosing rules require reliable sources, explicit validation and review by qualified professionals.

The benchmark shows why fluency is not enough to decide if a system is safe.

In high risk areas, tests should cover edge cases, version changes, missing data and extraction failures. They also need to record the source of each rule.