AI for Researchers · Visual Deck
What it can do with your data, where it fails invisibly, and the four habits that make either one defensible
{{Presenter Name}}
Landscape as of August 2026
2-Minute Recap
flowchart TB a1["S1
Capability/Failure Map
+ 2026 Tool Taxonomy"]:::done --> cap a2["S2
Literature-Discovery
Tool Matrix"]:::done --> cap a3["S3
Summary-Trust
Triage"]:::done --> cap a4["S4
Publisher AI-Policy
Comparison Table"]:::done --> cap cap(["Session 5 · today
the capstone — one page you take home"]):::today classDef done fill:#23204c,stroke:#8f8cb8,color:#b9b7d6,font-size:17px classDef today fill:#ab7d22,stroke:#d9b36c,color:#14132b,font-size:18px
CRIT — Context · Role & register · Instructions & constraints · Task, then iterate — travels with every stage.
"Generating analysis code" sits in the Unreliable zone: never the only pass, always spot-checked.
Learning Objectives
Objectives 4–6 next · full verbatim wording in the reference deck and curriculum
Learning Objectives
Full verbatim wording in the reference deck and curriculum
Section 01
Three jobs, one mechanism — and the mechanism is identical whether or not you ever see a line of code.
Literacy Foundation
These fail in different ways — only one fails loudly. All three sit in the Unreliable zone, and Rule 1 is the entry condition: if you could not catch the error, the task is not in the Safe zone.
The Mechanism, In Their Own Words
"For some data-analysis tasks, ChatGPT writes and runs Python code in a stateful Jupyter notebook environment."
OpenAI, Data analysis with ChatGPT [1]
The question is not "do I code?" — it is "can I get the code out?" A property of the tool, chosen before you upload.
Before You Upload Anything
Their own warning: a file-handling assistant "can be tricked into sending information from its context… to malicious third parties" [2]. Excel: "runs within the compliance boundary of your organization" [5]. Your institution's data-governance policy outranks every line above — public, consented, de-identified or published data is the only safe default.
The Evidence — Measured, Not Asserted
ScienceAgentBench superseded by AstaBench: 58.0% overall, ~3% perfect end-to-end discovery [48] · DSBench: no re-run found [13] · SciCode board stale; 60.2% via Artificial Analysis [14] [50] · CORE-Bench 95.5% manually scored; HAL paused [15] [49] · StatQA/StatABench [10] [17] — rows comparable to your expectations, not to each other [9]
Critical Literacy — The Key Slide
"LLMs primarily make applicability errors, whereas humans mostly make statistical task confusion errors."
Zhu et al., StatQA — 11,623 items, NeurIPS 2024 [10]. An applicability error is silent: the test ran, the arithmetic is right, the assumptions did not hold — and nothing in the output says so.
Fair counterweight: six models chose the correct test in 100% of 20 textbook scenarios — basic, free tiers, one day in August 2025 [19]. Choosing the test is close to solved. Applying it to your data is not — discrepancies cluster in post-hoc tests, confidence intervals, sign conventions, non-parametric tests [18]
The Evidence
"Setting the temperature to 0 does not guarantee determinism" [44] — the authors' own fix: run it multiple times independently and consider the distribution [43]
The Evidence
Stated precisely: 43.53% is a share of the hallucinations found, not of all code [20]. The one to fear: code that runs, returns a number, and answers a question you did not ask.
The Evidence — Two RCTs That Disagree
Honest caveats: n = 16 developers on codebases they knew for five years [24]; n = 47 students, 2022-era model [23]. The durable finding is the calibration gap, not the sign of the effect.
Section 02
Four habits. Three of them were published in 2013. AI did not create the problem — it multiplied the volume of code flowing into it.
The Evidence — The Pre-AI Baseline
The dominant cause was not bad science — it was undeclared dependencies and hard-coded file paths [27]. Of 296 deposited R projects, 98.8% carried no formal dependency description; 25.87% ran after automated environment rebuilds [45]. The question is not whether AI is worse than a human analyst — it is whether your result survives being re-run.
Series Artefact
Guardrails 1–3 are Sandve et al.'s Rules 1, 3 and 6 — published in 2013 [29]. Guardrail 4 is the one AI adds: the analyst who made the choices cannot be asked why.
The Rules, In Their Words
"…declare this in detail in the Methods section… The same standards for the validity and reproducibility of code apply whether the code was produced with AI assistance or not."
Elsevier, Generative AI policies for journals, updated June 2026 [42]
Session 4's rule holds: the publisher page gives the shape, the journal's guide for authors gives the wording. Check it the week you submit.
Section 03
Everything so far applies to you unchanged — and one column of the next table decides whether you can obey guardrail 1 at all.
Series Artefact
Every cell from that tool's own documentation, fetched 2026-07-28 — paths, not endorsements. New since compilation: Gemini Notebook (AI Pro) adds cloud code execution — press-reported, not vendor-documented [52] [53]; check code visibility first.
Section 04
From "write me a script" to "do the analysis" — that shift already shipped: Claude Science, Gemini Notebook's code execution, Elicit's Research Agent, Codex as a general work agent. The approval prompt is the last place a human is guaranteed to be standing.
The Mechanism, In Their Own Words
The tool will "determine which files to make changes to, offer code changes and terminal commands for the user's approval, and iterate to remediate issues until the original task is complete."
GitHub, Copilot features — agent mode [33]
Their own warnings: "no system is completely immune to all attacks" [34]; "prompt injection can cause the agent to fetch and follow untrusted instructions" [35]. The approval prompt is not a formality. It is the control.
The Evidence — The Trajectory, Re-Measured
Well-specified, metric-scored tasks moved 40–50 points in two years [16] [49]. Stated honestly: MLE-bench top entries self-reported, submissions paused [16]; HAL paused — 77.8% is an Opus 4.5-class agent, not the current frontier [49]; AstaBench is Ai2's own leaderboard [48]
Practice
If there is no task set, it is not a result. It is an advertisement.
Section 05
Five sessions, five artefacts, one page. Two rows are deliberately empty — Sessions 6 and 7 fill them in.
Series Artefact · The Capstone
One page, one project, one date — rows are stages of your work, not products, so the canvas survives your tools being replaced. Fillable version: last page of the handout.
Series Artefact
And leave rows 6 and 7 empty — next session's homework, and the session after that. The canvas names tasks, tiers and checks; nothing on it becomes wrong when a product is renamed.
Demo Preview
flowchart LR U["upload
344 penguins · CC-0
public dataset"]:::n --> E["explore
missing values
per column"]:::n --> T["one test
Welch's t-test
seed 42"]:::n --> F["one chart
~thirteen minutes
in all"]:::n --> V["seven minutes
of checking
re-run outside the chat ·
recompute one group mean ·
check the source paper"]:::hot classDef n fill:#23204c,stroke:#8f8cb8,color:#eceaf8,font-size:18px classDef hot fill:#1c1a3f,stroke:#d9b36c,color:#d9b36c,font-size:18px
C "a public CC-0 dataset of 344 penguins… a researcher, not a programmer" · R "a careful research statistician… plain language" · I "show the code every time · drop nothing without telling me · Welch's t-test · seed 42 · print every library version" · T "describe this dataset and its data-quality problems" — then iterate, then verify [40] [41]
References
References
References