Literacy Foundation

The Good Tools Run Both, Then Re-Rank

• Consensus described its 2025 pipeline as "a hybrid search approach" combining embeddings and BM25. ^[1]
• Step 2 re-ranked the top 1,500 on recency, citation count and journal impact; step 3 re-scored only the top 20. ^[1]
• In 2026 it rebuilt around "Scholar Agent" — a GPT-5-based multi-agent system: planning, search, reading, analysis. ^[35] The architecture changed; the lesson did not.
• SCiNiTO exposes Boolean AND / OR / NOT alongside its semantic search. ^[19]
• So "is it semantic or keyword?" is the wrong question. Ask: what did the re-ranker throw away before I ever saw it?

The Evidence

Measured Head-to-Head: Sensitivity vs. Precision

• Four completed evidence syntheses, re-run through Elicit Pro in Review mode, measured Feb–Mar 2025. ^[3]
• It missed most of what belonged — and it found included studies the original searches had not. ^[3]
• Authors' verdict: not sensitive enough to replace traditional searching; genuinely useful as an adjunct. ^[3]
• Currency caveat: Elicit shipped a different product in Aug 2026 (Research Agent) ^[33] — no independent evaluation of it exists yet. That gap is itself the finding.

The Evidence

Before We Get Smug: Human Search Is Biased Too

• An LLM judge rated only 51% of human citations as moderately relevant or better — against 86–88% for the best AI re-rankers. ^[4]
• On the co-authorship graph, humans were 2.5× more likely than the best AI re-ranker to cite a direct collaborator. ^[4]
• A reference list is a record of what an author found and chose — not a gold standard. ^[4]
• The honest framing: both methods are biased. Use them against each other.

Tier 2 — Research-Specific Tools

What "Citation-Grounded" Actually Buys You

• The tool retrieves first, generates second. The papers exist before the prose does. ^[1]
• Consensus puts it plainly: "We only use AI after we search the scientific literature." ^[1]
• That kills fabricated references — but not misread ones: the model can still cite a real paper for something it never said. ^[1]
• Session 1's 2023 baseline (GPT-3.5 / GPT-4 era): even when a citation was real, 24–43% carried substantive errors. ^[26]
• Grounded ≠ correct. It moves the failure from "does it exist?" to "does it say that?" — and on current systems that is now the dominant failure. ^[31]

Critical Literacy

Read the Corpus Number Critically

• Elicit's home page says 125M papers; its own pricing page says 138M. ^{[6] [7]}
• Scite's home page says 280M+ articles; its own pricing page says 300M+. ^{[13] [14]}
• SCiNiTO states 500M+ works on OpenAlex — OpenAlex's own catalogue reports 243M works. ^{[18] [20]}
• Nobody is lying: "works", "papers", "documents" and "sources" count different things.
• A bigger index is not a better search. The independent peer-reviewed head-to-head measured 39.5% sensitivity (Feb–Mar 2025) ^[3] — vendors' own newer benchmarks are self-graded. ^[33]

All corpus figures on this slide and on slides 13–14 are vendor self-reported, taken from each tool's own current pages; none is independently audited.

Tier 3 — Deep-Research Agents

What the Agent Actually Does

• Google: Deep Research and Deep Research Max, both built on Gemini 3.1 Pro — an agent that "autonomously plans, executes, and synthesizes multi-step research tasks". ^{[22] [32]}
• OpenAI: it "creates a proposed research plan. You can review and modify it before the research begins" — and the button is not the chat flagship: deep research has run a GPT-5.2-based model since Feb 2026. ^{[21] [34]}
• Anthropic: Claude Research (paid plans; model not publicly named) conducts "multiple searches that build on each other". ^[23]
• All three: minutes, not seconds, and a report with clickable sources. ^{[21] [22] [23]}
• The Tier 2 / Tier 3 line is blurring: agents search the open web — and via MCP can now call scholarly tools directly (Scite, Elicit, Consensus). ^{[13] [33] [36]}

Tier 3 — Deep-Research Agents

Where Agents Genuinely Win

• Iterating beats one-shot retrieval: expanding breadth-first along bibliographies raised recall on a 250-paper benchmark from under 20% to over 80%. ^[4]
• They cross into grey literature, policy documents and preprints that a scholarly index will not surface. ^[21]
• They will read a plan you wrote and follow it — collaborative planning is a real lever. ^{[21] [22]}
• They are excellent at orientation: "what are the camps in this debate, and who is in them?"
• But the ceiling is real: the best agent on ScholarQuest reached only 0.314 Recall@100. ^[5]

The Evidence

Documented Failure Modes

• Unsupported statements — the 2025 baseline. DeepTRACE audited that year's public systems: citation accuracy 40–80%, "large fractions of statements unsupported by their own listed sources". ^[24]
• Still true on the current generation. A May-2026 preprint audited 14 models (incl. Opus 4.5/4.6, GPT-5.2/5.4, Gemini 3.1 Pro): link validity above 94% — yet only 39–77% factual accuracy. The failure moved from fake URL to real-URL-unsupported-claim. ^[31]
• One-sidedness. Deep-research modes stayed "highly one-sided on debate queries" (2025 audit; no newer re-measurement). ^[24]
• Depth hurts. Fact-check accuracy dropped ~42% as tool calls scaled from 2 to 150 — more retrieval, not more accuracy ^[31]; agents hallucinate URLs at 10.7% pooled vs 4.8% for search-augmented models. ^[25]

Critical Literacy

The Same Tool Scored 25% and 69%

• One tool, one mode, one team, about two weeks — and sensitivity ranged from 25.5% to 69.2% across four evidence syntheses. ^[3]
• The 39.5% headline is the mean of those four. It describes none of them. ^[3]
• Precision barely moved (35.6–46.2%) — while the Boolean searches' precision ran from 0.65% to 14.7%. ^[3]
• What counts as a "hit" depends on the review's inclusion criteria and the query — not the tool alone. ^[3]
• Same pattern one tier up: citation accuracy spanned 40–80% across 2025 systems ^[24], factual accuracy 39–77% across 14 current models. ^[31]
• Rule: before you believe a headline number, ask what counted as a hit — and how many topics it was averaged over.

Every performance figure in this deck comes from independent evaluation ^{[3] [24] [25] [31]} — no vendor self-reported performance claim appears on any slide.

Caution Zone 1

Systematic Reviews Have Rules — and They Now Cover AI

• Cochrane, Campbell, JBI and CEE issued a joint position statement in 2025. ^[27]
• You may use AI "as long as you can demonstrate that it will not compromise the methodological rigour or integrity" of the synthesis. ^[27]
• "AI and automation in evidence synthesis should be used with human oversight." ^[27]
• Any AI use that "makes or suggests judgements" must be fully and transparently reported. ^[27]
• The scoping evidence agrees: LLMs "should only be used with caution and under human supervision". ^[28]

Caution Zone 2

Coverage Bias: The Index Shapes the Answer

• Of 62,701 active open-access journals, Web of Science indexes 6,157, Scopus 7,351, OpenAlex 34,217. ^[29]
• Only 4,094 of them appear in all three. ^[29]
• Africa is underrepresented in all three; high-income economies are consistently over-represented in Scopus and WoS. ^[29]
• Open is not simply worse — OpenAlex's reference coverage is comparable to WoS and Scopus on shared publications. ^[30]
• Every AI tool inherits its index's shape. Ask which index, then ask who it leaves out.

Caution Zone 3

Paywalls: Most Tools Are Reading Abstracts

• OpenAlex catalogues 243M works — 48M of them open access, roughly one in five. ^[20]
• For the other four in five, an open tool sees metadata and an abstract, not the methods section.
• Effect sizes, limitations and the sample description live in the full text you were not shown.
• Scite's answer is licensing: agreements with 30+ publishers to read behind the paywall. ^[13]
• SCiNiTO's institutional tier is the same bargain: full-text access comes from your library. ^[18]
• Practical move: route the shortlist through your institutional access before you summarise it.

Caution Zone 3

And No, Google Scholar Is Not the Safe Harbour

• Across 120 systematic reviews, Google Scholar's coverage was 97.2% — it holds nearly everything. ^[2]
• Its recall was 72.8%, against 81.6% for Embase and MEDLINE combined. ^[2]
• Only 46.4% of included studies sat inside the downloadable first 1,000 results. ^[2]
• Verdict: "neither GS nor one of the other databases investigated, is on its own, an acceptable database to support systematic review searching." ^[2]
• Holding a paper and surfacing it are different problems. That is true of every tool on slide 13.

Series Artefact

The Citation-Verification Workflow

1. Resolve it. Necessary, no longer sufficient: frontier systems now keep link validity above 94% ^[31] — in 2025 audits 5–18% still failed here. ^[25]
2. Match it. Do authors, year, journal and title agree with the record — all four?
3. Open it. Find the sentence in the source that carries the claim.
4. Compare it. Does the source say that, or something weaker, narrower or opposite? ^{[24] [31]}
5. Check the direction. Was the paper later contradicted? ^[13]

Steps 1–2 catch fabrication; steps 3–5 catch misreading — now the dominant failure: with links valid, factual support runs 39–77% ^[31], and grounded tools do not remove it ^{[1] [26]}.

Live Demo & Takeaways

What We Do Next — and What You Take Home

• Demo: one real question, run through a citation-grounded tool and a deep-research agent, side by side.
• Then: we pull three citations out of the agent's report and check every one, live.
• Take home: the Literature-Discovery Tool Matrix, the citation-verification checklist, and prompt templates for both tool types.
• Next session: you have a shortlist — Session 3 turns it into structured notes you can compare.