> ## Content Index
> Fetch the complete content index at: https://www.openresearch.wtf/llms.txt
> Use this file to discover other available public pages before exploring further.

# Preprints.ai: How Much of Peer Review Can We Automate?
- URL: https://www.openresearch.wtf/preprints-ai-how-much-of-peer-review-can-we-automate/
- Published: 2026-03-23T02:35:17.000Z
- Updated: 2026-07-31T12:35:53.000Z
- Description: A large share of peer review is mechanical checking. Preprints.ai tests how much of it can be scored automatically, at preprint scale.
- Author: Mark Hahnel
- Tags: Peer-review, Preprints, AI, Experimentation, #card

![](https://storage.ghost.io/c/45/ef/45ef5edd-7737-4b73-a169-a85c335bb1f1/content/images/2026/03/data-src-image-ccdfdf7f-1495-48bd-bdd7-149c2f5645ed.png)

## **What problem are we trying to solve?**

[Peer review is structurally failing](https://www.openresearch.wtf/have-we-already-hit-the-peer-review-breaking-point/). The volume of preprint submissions has grown faster than the reviewer pool for decades. Average review turnaround times have lengthened. Reviewer fatigue is well-documented. And review quality is inconsistent in ways that are impossible to fix when the process depends entirely on the goodwill of overcommitted academics. More simply, can we score every preprint.

At the same time, a significant fraction of what peer reviewers actually do is mechanical. Checking that statistical methods match reported results. Verifying that cited claims are accurately represented. Assessing whether the methodology section contains enough detail to reproduce the experiments. Scanning for common red flags - tortured phrases from paper mills, image duplication, impossible p-values. 

![](https://storage.ghost.io/c/45/ef/45ef5edd-7737-4b73-a169-a85c335bb1f1/content/images/2026/03/data-src-image-6376e492-4ea3-46a5-85fd-d8bfd4d0b659.png)

Preprints.ai asks a sharper question than "can AI help with peer review?" It asks “if we were designing peer review from scratch for a world where powerful LLMs exist, what would we actually need humans for, and what could we comfortably automate?”

The platform currently runs a multi-agent review pipeline against preprints from [bioRxiv](https://www.biorxiv.org/?ref=openresearch.wtf) and [medRxiv](https://www.medrxiv.org/?ref=openresearch.wtf), assessing two dimensions: *research integrity* (methodology, statistical validity, reproducibility, citation accuracy) and *novelty* (does the core claim already exist in the literature?). The output is a structured assessment graded on an A5-to-E1 rubric, with positivity bias actively recalibrated using publication outcome data.

## **What tools have inspired or been used to build this platform?**

### **LLM providers and hybrid routing**

We use three LLM providers in production, with each agent in the review panel assigned to a *different* provider to ensure genuine inter-model independence:

| Provider                                                                                    | Model                     | Role                                    | Cost / 1M tokens     |
| ------------------------------------------------------------------------------------------- | ------------------------- | --------------------------------------- | -------------------- |
| [Claude](https://docs.anthropic.com/en/docs/about-claude/models?ref=openresearch.wtf)       | claude-sonnet-4-20250514  | Primary reviewers (methodology, domain) | $3 in / $15 out      |
| [Claude Haiku](https://docs.anthropic.com/en/docs/about-claude/models?ref=openresearch.wtf) | claude-haiku-4-5-20251001 | Fast checks (ethics, transparency)      | $0.80 in / $4 out    |
| [GPT-4o](https://platform.openai.com/docs/models?ref=openresearch.wtf)                      | gpt-4o-2024-11-20         | Secondary reviewer for consensus        | $2.50 in / $10 out   |
| [Gemini](https://ai.google.dev/gemini-api/docs/models?ref=openresearch.wtf)                 | gemini-2.5-flash          | Secondary reviewer for consensus        | $0.15 in / $0.60 out |

This isn't just cost optimisation. When three different model families independently agree on a finding, the probability that all three share the same hallucination is substantially lower than any single model's error rate. Round-robin provider assignment prevents correlated model bias in the consensus score.

### **The review pipeline: Layer 1 (deterministic, zero LLM cost)**

Before any LLM sees the paper, a battery of rule-based checks runs in under 2 seconds:

PDF Quality Gate → Language Detection → Paper Mill Detection → Image Forensics → Statistical Consistency → Trust Markers

These are inspired by and build on established tools - such as Ripeta’s Trust Indicators.

- Paper mill detection uses a database of 8,000+ [tortured phrases](https://pubpeer.com/?ref=openresearch.wtf) (e.g., "bosom malignancy" for "breast cancer") and 257 [SCIgen](https://pdos.csail.mit.edu/archive/scigen/?ref=openresearch.wtf) patterns for computer-generated text
- Statistical consistency implements the [GRIM test](https://jamesheathers.github.io/sprite/index.html?ref=openresearch.wtf) (Granularity-Related Inconsistency of Means) and [statcheck](http://statcheck.io/?ref=openresearch.wtf)\-style p-value recalculation from reported test statistics
- Image forensics is inspired by [ELIS](https://www.biorxiv.org/content/10.1101/2022.01.24.477532v1?ref=openresearch.wtf) (Error Level Image Similarity) — detecting duplicate figure regions, inconsistent compression, and manipulation artefacts using [imagehash](https://github.com/JohannesBuchner/imagehash?ref=openresearch.wtf) and [Pillow](https://pillow.readthedocs.io/?ref=openresearch.wtf)
- Fabrication detection checks for [Benford's Law](https://en.wikipedia.org/wiki/Benford%27s%5Flaw?ref=openresearch.wtf) deviation in reported data and suspiciously perfect statistical distributions
- Trust markers detect the presence (or absence) of data availability statements, code repositories, ethics approvals with protocol numbers, pre-registration links ([ClinicalTrials.gov](https://clinicaltrials.gov/?ref=openresearch.wtf), [OSF](https://osf.io/?ref=openresearch.wtf)), ORCID identifiers, and CRediT author contributions
- Reporting guideline compliance checks against [ARRIVE](https://arriveguidelines.org/?ref=openresearch.wtf) (animal research), [CONSORT](https://www.consort-statement.org/?ref=openresearch.wtf) (clinical trials), [PRISMA](https://www.prisma-statement.org/?ref=openresearch.wtf) (systematic reviews), and [MIQE](https://www.gene-quantification.de/miqe-bustin-et-al-clin-chem-2009.pdf?ref=openresearch.wtf) (qPCR)

![](https://storage.ghost.io/c/45/ef/45ef5edd-7737-4b73-a169-a85c335bb1f1/content/images/2026/03/data-src-image-63fa5891-49cd-45e5-a7fa-065fa7be545e.png)

### **The review pipeline: Layer 2 (multi-agent LLM review)**

The agent architecture is heavily inspired by [eLife's deliberative review model](https://elifesciences.org/inside-elife/e01a1e6e/elife-s-new-model-changing-the-way-you-share-your-research?ref=openresearch.wtf). We run nine specialist agents, each with a distinct role and scope boundary:

| Agent                 | Focus                                                        | Scope boundary                      |
| --------------------- | ------------------------------------------------------------ | ----------------------------------- |
| Methodologist         | Experimental design, controls, sample size, confounding      | NOT statistics (statistician's job) |
| Statistician          | Test appropriateness, p-values, effect sizes, power analysis | NOT study design                    |
| Domain Expert         | Novelty assessment, field context, missing citations         | Full eLife Reviewer #1 depth        |
| Reproducibility       | Method detail, data/code availability, reporting guidelines  | NOT statistical correctness         |
| Ethics & Transparency | IRB approvals, COI disclosure, funding, pre-registration     | Ethics only, not science            |
| Scientific Validity   | Pseudoscience gate: theoretical coherence, plausibility      | Gate only — not quality             |

Plus three additional domain-specialist agents routed based on the paper's bioRxiv category, drawn from a pool of 31 field-specific configurations covering everything from biochemistry to systems biology. Each configuration specifies methodology standards, common issues, reporting guidelines, statistical expectations, and key databases for that field.

### **Deliberation: from independent reviews to consensus**

Independent review is necessary but not sufficient. In human peer review, the reviewing editor synthesises reviewer comments into a coherent assessment. We replicate this with a structured deliberation protocol inspired by eLife:

1. Round 0 (Independent review) - All 9 agents review the paper in parallel, each with a different LLM provider. No agent sees any other agent's output.
2. Round 1 (Consultation) - Agents see anonymised summaries of each other's reviews. Each can agree, challenge, or add nuance.
3. Round 2 (Reconciliation) - A senior editor agent synthesises the consultation into a single eLife-style assessment: *"This study presents \[significance\] findings on \[topic\]. The evidence is \[strength\], with \[justification\]."*
4. Dissent recording - Any agent that disagrees with the final assessment can lodge a recorded dissent, preserved in the output for transparency.

This substantially reduces variance compared to single-pass review. We track inter-agent agreement (standard deviation of scores across agents) and use it as a confidence signal. eg. high disagreement triggers lower confidence scores and flags the paper as borderline.

### **External data sources**

The platform integrates with multiple academic data APIs, each serving a distinct purpose:

| Source                                                                                                     | Purpose                                                  | Usage                                                                 |
| ---------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- | --------------------------------------------------------------------- |
| [Semantic Scholar](https://api.semanticscholar.org/?ref=openresearch.wtf)                                  | Citation network, reference verification, passage search | 200M+ papers; checks cited claims are accurately represented          |
| [OpenAlex](https://docs.openalex.org/?ref=openresearch.wtf)                                                | Journal tier mapping, h-index, publication outcomes      | Calibration: maps journal prestige to validate our novelty scores     |
| [Crossref](https://api.crossref.org/?ref=openresearch.wtf)                                                 | DOI metadata, retraction status, publication relations   | Detects retracted papers and tracks preprint-to-publication links     |
| [Europe PMC](https://europepmc.org/RestfulWebService?ref=openresearch.wtf)                                 | Publication discovery, fulltext links                    | Fallback for finding published versions of preprints                  |
| [bioRxiv API](https://api.biorxiv.org/?ref=openresearch.wtf)                                               | Preprint metadata ingestion                              | Primary source for paper metadata and PDF URLs                        |
| [Retraction Watch](https://retractionwatch.com/retraction-watch-database-user-guide/?ref=openresearch.wtf) | Known-retracted papers for validation                    | Ground truth: can our system detect papers that were later retracted? |

### 

### **The scoring rubric**

We grade on two independent dimensions, producing a combined grade like B4 or C3:

Integrity (A–E): Strength of evidence. A = compelling, reproducible methodology with full transparency. E = fundamental unfixable problems (fabrication signals, pseudoscience, paper mill indicators).

Novelty (5–1): Significance of findings. 5 = landmark, field-shifting (almost never assigned). 3 = important, real contribution beyond a single subfield. 1 = useful but very limited — one more data point.

Each grade maps to an editorial decision in journal-standard language: Accept, Minor Revision, Major Revision, or Reject — with modifiers for low confidence, abstract-only assessment, or high reviewer disagreement.

![](https://storage.ghost.io/c/45/ef/45ef5edd-7737-4b73-a169-a85c335bb1f1/content/images/2026/03/data-src-image-1ace464e-673d-4aa4-a452-c0d5e520763d.png)

### **Fighting positivity bias using recalibration**

LLMs are famously sycophantic. Left uncalibrated, every paper gets a B. We fight this at multiple levels:

- Downward recalibration rules: 2+ critical weaknesses forces a D floor. 4+ major weaknesses caps at low-C. A "reject" recommendation from any agent caps integrity at 0.42.
- Per-agent accuracy tracking: After each calibration run, we compute every agent's mean absolute error and bias against publication outcomes. Agents with measured bias > ±0.15 get a steering note injected into their next prompt: *"Historical data shows you tend to overscore novelty by \~20%. Be especially critical of incremental contributions."*
- Inverse MAE weighting: In the consensus aggregation, agents with lower historical error get higher weight. An agent that has been consistently accurate on past papers has more influence on the final score than one that hasn't.
- Publication outcome calibration: We match assessed preprints against publication records [using Crossref’s open data](https://www.crossref.org/blog/discovering-relationships-between-preprints-and-journal-articles/?ref=openresearch.wtf). If we scored a paper as novelty-2 but it was published in a tier-5 journal, that delta feeds back into agent weights and prompt tuning.

### **Confidence and transparency**

Every assessment includes:

- Confidence score (0–100%) derived from agent agreement, mean agent self-reported confidence, and score dispersion. High disagreement between agents lowers confidence.
- Individual agent reviews. Every agent's scores, strengths, weaknesses, and recommendations are available in the full report, not just the consensus.
- A 2–3 sentence plain-language summary of the core finding, for a non-specialist audience.

![](https://storage.ghost.io/c/45/ef/45ef5edd-7737-4b73-a169-a85c335bb1f1/content/images/2026/03/data-src-image-f5970253-ca5a-422a-b951-dc680bacd50f.png)

### **Cost per paper**

With hybrid routing across Claude, GPT-4o, and Gemini Flash, the current cost per full assessment is approximately $1.50–2.00 for a consensus review with 9 agents. Layer 1 deterministic checks add negligible cost. At scale, with aggressive caching of reference verification results and local model routing for lower-signal checks, we expect this to converge toward $0.50–1.00 per paper.

## **What we can look to do next**

### **Validation is the open question**

The most important next step is systematic validation. How do Preprints.ai assessments compare to human peer review on the same papers? We've built a [/validate](https://preprints.ai/validate?ref=openresearch.wtf) page where domain experts can agree, disagree, or override AI grades but we need hundreds of expert validations across diverse fields before the calibration claims are defensible. There is no incentive for subject experts to do this, so we can assume they wont.

We're also running consistency tests. We are assessing the same papers multiple times to measure grade variance. If the system can't reproduce its own grades, nobody should trust them. Our target is >70% exact grade stability and >90% within-one accuracy.

### **Field-specific scoring norms**

A C3 in clinical medicine means something different than a C3 in pure mathematics. We're computing per-discipline score distributions from assessed papers and using z-score normalisation to generate field-specific grade thresholds. Fields with sufficient data (20+ assessed papers) get their own norms; others fall back to global defaults.

### **The hybrid model question**

Longer term, the interesting question is whether a hybrid model - AI as first-pass screen, human expert for borderline cases - could work operationally within a real journal workflow. The system already identifies which papers need human attention (low confidence, high agent disagreement, borderline C-range grades). The question is whether editors would trust it enough to act on.

I think the answer is yes, but only after the validation data proves it.If you're working on any of these problems and want to compare notes, or if you spot something obviously wrong with the approach, I'd be glad to hear from you.