How we grade evidence

Every grade on this site is attached to an intervention–outcome pair, never to a molecule. Creatine can be well-evidenced for strength and unevidenced for cognition, and collapsing that into one verdict is how health misinformation gets made. So the unit of grading is the pair, and each pair has its own score, its own study list and its own reviewer.

The score is built in four steps. First, every study in the base is appraised for internal validity, which produces a per-study rigor and a per-study mass. Second, four sub-scores are computed from those masses. Third, the sub-scores are combined by a weighted geometric mean. Fourth, named penalties are subtracted and two hard ceilings are applied. The result is rounded to an integer and banded.

Species is deliberately absent from rigor. Species is external validity and lives on the human-relevance axis instead. Splitting the two is what lets a tightly controlled mouse experiment read as methodologically clean while still being unable to carry a human claim.

Per-study modulators, with their defaults

• Risk of bias. A 0–100 appraisal, mapped onto 0.35 to 1.0 of the design weight. A harsh appraisal floors a study at 0.35 of its design weight rather than deleting it, because a multiplicative zero would let one reviewer silently remove a study with no audit trail. An unappraised study is assumed to be 50, never 100 — otherwise leaving the field blank is rewarded.
• Sample size. Saturates at 1000 participants; beyond that, power is rarely the binding constraint, bias is. An unreported participant count is assumed to be 30.
• Duration. Saturates at 52 weeks. Longevity outcomes need at least a year to say anything about durability. An unreported duration is assumed to be 4 weeks.
• Preprints. Multiplied by 0.6 — worth about three fifths of the same paper after peer review. Not zero, because preprints of large registered trials are real evidence; not 0.9, because the failure we care about is a hyped preprint driving an A grade before review catches an error.
• Syntheses. A meta-analysis or systematic review carries 3× the mass of a single trial, because it subsumes a body of primary work. If both a synthesis and its constituent trials are in the base, reviewers are required to down-weight the primaries, or the same trials are counted twice.
• Retractions. Contribute zero mass, and additionally trigger a penalty (below).

The gaps are deliberately uneven. The step from randomised trial (0.85) to cohort (0.58) is the confounding cliff, and it is far larger than the step from meta-analysis to systematic review, which is mostly about whether the synthesis pooled numerically or only narratively.

Two values sit outside the hierarchy because neither names a methodology. A preprint is weighted as its design would be and then discounted to 0.6 of it, because it has not been peer reviewed. A study we have recorded but not yet classified carries 0.02 — below every real design, so that classifying it can raise a grade and can never lower one. An unclassified evidence base is therefore always the most pessimistic reading of itself, and a paper we have not read yet can never be the reason something looks proven.

There is one deliberate asymmetry in the whole engine and it is here. Harm mass is computed without the species discount. A clean rodent toxicity signal is a reason to warn a human; a rodent efficacy signal is not a reason to recommend to one. We would rather over-warn than under-warn.

• No human evidence at all: the score is capped at 34, which sits inside band D. Such a base is additionally refused a letter entirely and graded INSUFFICIENT.
• Under 34% of evidence mass human: capped at 64, the top of band C. A claim where two thirds of the mass is preclinical may be “limited”, but it must never render as an actionable A or B.
• At or above 34% human mass: no cap.

Quality and quantity are direction-blind by construction: they measure how much good evidence exists, not which way it points. Without this gate, five large, well-conducted trials that all refute a claim would read as an excellent literature and land mid-scale. The consistency sub-score knows the direction, so it is turned into a hard cap.

The mixed threshold is 55 rather than 50 because the additive smoothing on consistency parks a small, evenly split base slightly above the midpoint.

• Retractions. 3 points per retracted study, to a maximum of 10. The retracted study already contributes zero mass; the extra points exist because a literature that contained retracted work is itself a signal about that field’s reliability.
• Conflict of interest. Handled once, at the level of the whole base, rather than per study. Industry funding does not reliably invalidate an individual trial, but a literature where most of the mass is conflicted has a known publication-bias problem. The penalty starts once conflicted mass passes 50% of the base and ramps to 8 points at 100%.

80 for A and 65 for B come from worked examples: a meta-analysis plus four large unanimous trials lands around 83; three good trials with no synthesis lands around 66. That is the intended bar — A means settled, B means solid. Only A and B may ever be presented as something to do; C and D read “discuss with your clinician” everywhere on the site, including inside the calculators and the AI summaries, because the rule is enforced by one shared function rather than by editorial habit.

When a computed score lands within two points of a band boundary, the letter is treated as a judgement call and a named medical reviewer has to confirm it before it publishes.

A through F answer one question: how strong is the evidence that this helps? Two situations cannot be answered on that scale at all, so they are not put on it.

HARMFUL

Assigned when harm-direction evidence mass is at least as large as benefit-direction mass, and reaches an absolute floor of 0.3 mass units — roughly one adequately powered human trial or one strong cohort. The floor stops a single case report or an in-vitro hint from overriding an established benefit. It is checked before everything else, including the insufficiency test: a harm verdict resting on a thin base still publishes, because under-warning is the more expensive error, but a reviewer has to confirm the wording.

INSUFFICIENT

Assigned when the base is too thin to band at all. Three independent floors, any one of which is enough:

• Fewer than 2 independent evidence units (a synthesis counts as 3). One primary study cannot be banded.
• Total effective mass below 0.75.
• Human mass below 0.25 — roughly one small cross-sectional study. Below that the human signal is decorative.

Four channels, weighted: search interest 0.35, social mentions 0.3, news 0.2, podcasts 0.15. Search leads because it is volunteered rather than pushed; social is fastest but noisiest; news and podcasts are slower and more editorially mediated. A missing channel is renormalised away rather than counted as zero, so an absent podcast feed does not read as “nobody is talking about this”.

Each channel is normalised within its category against that category’s own median and 95th percentile: take the log of the count, express it as “how many median-to-p95 steps above the category median is this”, then squash it with a logistic curve whose steepness is chosen so the category median lands at exactly 0.50 and the category p95 at exactly 0.95.

Why within a category: 1,200 mentions means something completely different for a supplement than for a surgical procedure, and comparing them would make the number meaningless. Why not min–max: one viral week would redefine the top of the scale for the whole category and peg everything else near zero. Under this scheme an observation ten times the category p95 scores about 0.999 on that channel instead of rewriting the scale.

A gap of 20 points or more between hype and evidence is flagged as a real divergence. That flag is the most useful thing on many of our pages: it is the list of things a lot of people are being told to take, on not very much.

• Every graded page is re-reviewed at least every 12 months.
• Pages above 5000 views per 28 days are re-reviewed every 6 months, because a stale grade on a high-traffic page is a live recommendation nobody is checking.
• The review queue is prioritised by how overdue a page is, how much traffic it gets, and how costly it would be for that particular grade to be stale. HARMFUL and F rank highest because they are safety surfaces; a stale A ranks above a stale C because A is rendered as actionable.
• New evidence can also trigger an out-of-cycle review. Every grade change is published in the grade change log with the reviewer’s written justification, verbatim and unedited, with a 80-character minimum that a validator enforces before the change can be saved.

The evidence formula is currently 1.0.0 and the hype formula 1.0.0. The version is bumped whenever any weight, threshold, ceiling or step changes — not only when the shape of the formula changes. Each stored grade records the version that computed it, so a grade that moved because we changed the formula is distinguishable from a grade that moved because the evidence changed. Those are different events and conflating them would make the change log worthless.

Recomputing the whole corpus under a new version is a deliberate, logged operation, and every letter that moves as a result appears in the change log with the version bump named as the reason.

Why is the Evidence Score a geometric mean rather than an average?

Because the four dimensions are not substitutes for each other. An arithmetic mean lets three strong axes buy off one catastrophic axis — a body of work could have superb design, enormous volume and perfect agreement and still be worthless for a human claim because it is all in mice. A geometric mean cannot be rescued that way: if any factor approaches zero, so does the product.

Why does adding more studies not always raise the score?

Adding a non-retracted, non-conflicted, supporting human study can never lower the score — that is a guaranteed property of the formula. Adding a weak, contradicting, animal or conflicted study can lower it, because quantity saturates while consistency, human relevance and the conflict penalty do not.

Why can a mouse study never produce a good grade?

Because the human-relevance gate is a hard ceiling, not a discount. A base with no human evidence is capped at 34, which sits inside band D, and separately refused a letter at all. A discount can be out-earned by piling on more animal studies; a ceiling cannot.

Is HARMFUL just a very low grade?

No. A through F measure how strong the evidence for a benefit is. HARMFUL is a different statement: the evidence of harm outweighs the evidence of benefit. It is a warning, not a position on a scale, which is why it is not a band.

Does the Hype Score affect the Evidence Score?

No, in either direction. They are computed by separate functions from separate inputs and pinned to separate version numbers. The point of showing both is that the gap between them is information — and a gap is only information if neither number knows about the other.