← series index
# Session 5: Data, Code & Your AI Workflow
*AI for Researchers — attendee handout · Landscape as of August 2026*
---
## At a glance
| | |
|---|---|
| **Session** | 5 of 7 — Data, Code & Your AI Workflow (capstone) |
| **You'll learn** | What AI actually does when it "analyses your data", which of its errors are invisible in the output, the four guardrails that make an AI-assisted analysis defensible, and how the no-code paths differ on the one property that matters. |
| **You'll practise** | Taking a real open dataset through AI-assisted exploration, one test and one chart — then re-running the code, recomputing one number by hand, and checking the result against the published paper. |
| **Prerequisite** | None, and **no coding**. Every technique here has a no-code path. |
| **Tools referenced** | None endorsed. The guardrails and the canvas name tasks and checks, so nothing here becomes wrong when a product is renamed. |
---
## The one-paragraph version
AI can genuinely accelerate an analysis, and the evidence now comes in pairs — a 2024 baseline and a 2026 re-measurement. Where the task is well specified and scored by a metric, two years moved the numbers enormously: reproducing a published result *with the code and data already supplied* went from **21%** to **77.8%** (95.5% with manual scoring) for an Opus 4.5-class agent [15][49], and the Kaggle-medal rate on 75 ML-engineering contests went from **16.9%** to **64.44%** [16]. Where the task is your actual job, it did not: on 102 expert-validated science tasks the best agent solved **32.4%**, and that benchmark froze at **33.3%** (o3, April 2025) [9][49]; its successor, AstaBench, scores the best 2026 agent at 58.0% overall but only **~3% perfect completion** on end-to-end discovery [48]. The failure that should change your habits is still not arithmetic. On 11,623 statistical items the best mid-2024 model scored **64.83%** — and the key finding was that "LLMs primarily make **applicability errors**, whereas humans mostly make statistical task confusion errors" [10]: the test ran, the number is right, the assumptions did not hold, and nothing in the output says so. Two years on, GPT-5.1 reaches only **68.6%** on the successor StatABench, with tool selection still half the audited failures [17]. Given the same questions as expert analysts, agents reproduced **under 13%** of the experts' modelling decisions [11]. Ask twice and you often get a different workflow: reproducibility ran **21–81%** across three task sets, and inconsistent answers were measurably *less accurate* (64.4% vs 53.5%, p < 0.001) [12][43]. No setting fixes it — "setting the temperature to 0 does not guarantee determinism in code generation" [44]. None of which makes AI uniquely dangerous. Before any of it, **74%** of 9,000+ deposited R files failed to run [26], **98.8%** of 296 deposited R projects carried no dependency description [45], and only **26%** of *Science* papers under a code policy could be reproduced [28]. AI does not create that problem; it multiplies the volume of code flowing into it. The four habits below are the fix, and three were published in 2013 [29].
---
## Three analysis prompt templates
Adapt the bracketed parts. All three work in any assistant that can run code on an uploaded file, and in the no-code tools below.
**A. First pass on a dataset** — makes the tool surface its choices instead of burying them.
```
Context: I have uploaded [FILE]. It is [ONE SENTENCE: what the data is,
how many rows roughly, where it came from]. I am a researcher, not a
programmer.
Role and register: Act as a careful research statistician doing a first
pass on someone else's dataset. Plain language, no jargon without a gloss.
Instructions and constraints:
- Show me the code you run, every time, before the result.
- Report the exact number of rows and columns, and each column's type.
- Report missing values PER COLUMN as counts, not percentages.
- Do not drop, impute or clean anything yet. Tell me what you would drop
and why, and wait for me.
- State any assumption you had to make in a line beginning "ASSUMPTION:".
- Do not suggest a statistical test yet.
Task: Describe this dataset and its data-quality problems.
```
*When to use:* always, first. *Watch out for:* a tool that will not show you its code cannot be part of a reproducible workflow — guardrail 1, non-negotiable.
**B. One statistical test, defensibly** — the constraints are the whole template.
```
Instructions and constraints:
- Restrict to [SUBSET]. Compare [OUTCOME] between [GROUP A] and [GROUP B].
- Exclude rows where [VARIABLE] is missing. Report exactly how many rows
you excluded and from which group.
- Use [NAMED TEST] and do not assume [ASSUMPTION]. Name the test you ran
in the output.
- Report in a table: n, mean and SD per group; the difference in means;
the test statistic; the degrees of freedom; the p-value; the 95%
confidence interval; and an effect size.
- Set the random seed to [NUMBER] anywhere a random choice is made, and
say where you did.
- Print the exact version of every library you used.
- Show the full code. Do not summarise it.
Task: Run that test and report the numbers. Do not interpret them yet.
```
*When to use:* any confirmatory test. *Watch out for:* check the **degrees of freedom**, not just the statistic — that is where an ignored "do not assume equal variances" shows up, and nobody reads it.
**C. The adversarial check** — run this in a *fresh session*, pasting only the code and the numbers.
```
Context: Below is analysis code and the results it produced. I did not
write it and I am checking it.
Instructions and constraints:
- List every assumption this test requires, and whether the code checks
it. Do not run the analysis again.
- List every row or case this code silently excludes.
- Name any place the code answers a subtly different question from the
one stated.
- If you would have chosen a different method, say which and why, in one
sentence, without rewriting anything.
Task: Produce those four lists. Do not reassure me.
```
*When to use:* before any AI-assisted result leaves your machine. *Watch out for:* the fresh session matters — in-thread, the tool defends its own earlier work.
---
## No-code analysis paths compared
Every cell from that tool's own documentation, fetched 2026-07-28. Rows are **paths**, not endorsements.
| Path (examples) | What it does for you | Can you get the code out? | A documented limit |
|---|---|---|---|
| **Assistant + code execution** (ChatGPT, Claude) | Upload a file, ask in plain English; it writes and runs Python [1] | **Yes** — ask for it and save it. "For some data-analysis tasks, ChatGPT writes and runs Python code in a stateful Jupyter notebook environment" [1] | Sandbox has no internet; ~50 MB CSV ceiling; containers expire [1][2] |
| **Spreadsheet copilot** (Copilot in Excel) | Answers questions about the workbook you already have | **Yes** — "expand the code to see how the answer was generated" [5] | Results "aren't refreshable"; "Copilot does not modify your workbook" [5] |
| **Notebook agent** (Colab Data Science Agent) | Plain-English request becomes code proposed into your notebook | **Yes, by design** — *Accept*, or *Accept and run* [3] | CSV and BigQuery sources; runs only in your runtime [3] |
| **Dedicated analysis tool** (Julius) | Upload, ask, get tables and charts; reads SPSS files | **Yes** — notebooks "re-run, version, and share… for reproducibility" [6] | Free-tier files deleted after 1 h idle; per-file size not documented [6] |
| **Point-and-click stats + AI** (JASP 0.98, July 2026) | Menu-driven tests; optional AI to "annotate analyses, interpret results" [7] | The .jasp file "besides the data also holds analyses and interpretations" [7] | AI is opt-in with your own API key; data handling not documented [7] |
| **Point-and-click stats, no AI** (jamovi) | Menu-driven tests, publication-formatted output | **Yes** — Syntax mode emits "equivalent R code for each analysis" [8] | Sharing needs colleagues to hold the same R packages [8] |
*Capabilities change monthly — verify before relying on a cell.* **Column 3 decides whether you can obey guardrail 1.** One honest gap: at least one major assistant's own upload documentation does not state that it writes or runs visible Python at all — re-verified 2026-09-02, still the case [4]. And one addition since the table was compiled: Gemini Notebook (the renamed NotebookLM [52]) is rolling out cloud code execution for AI Pro subscribers, "complex data analysis grounded in your sources" — press-reported [53], not yet in Google's own product docs, so check code visibility before it becomes your path.
**Before you upload anything.** Files may be used for training unless you turn it off — "Improve the model for everyone", under Settings > Data Controls [1]. Retention differs sharply by product and tier [6]. One vendor warns its own file tool "can be tricked into sending information from its context… to malicious third parties" [2]. Your institution's data-governance policy outranks all of it.
---
## The reproducibility checklist
The four guardrails, as ten things you do. Guardrails 1–3 are Sandve et al.'s Rules 1, 3 and 6, from 2013 [29]; guardrail 4 is the one AI adds.
**1 · Keep the code**
- [ ] **Ask to see the code every time**, and save it beside the data — not in a chat transcript. Sandboxes are ephemeral; one vendor's containers "expire 30 days after creation" [2].
- [ ] **Re-run it once outside the tool.** If it fails on an undefined variable, the chat had state you do not — the commonest way an AI analysis turns out not to be reproducible [27].
- [ ] **If your tool cannot show the code, change tools for this task.** That is a property of the product, not of your skills.
**2 · Pin the versions**
- [ ] **Make the tool print every library version**, and record the model name, version and date with them [46]. Undeclared dependencies were the dominant failure cause across 27,271 notebooks from 3,467 biomedical papers [27].
- [ ] **Export an environment file** (`requirements.txt`, `environment.yml`, `renv.lock`) as you finish, not at submission.
**3 · Set the seed**
- [ ] **Put the seed in the prompt and in the code**, and ask where randomness entered. Rule 6: "For Analyses That Include Randomness, Note Underlying Random Seeds" [29].
- [ ] **Do not assume a temperature setting fixes it** — "temperature to 0 does not guarantee determinism in code generation" [44].
**4 · Verify**
- [ ] **Recompute one reported number by hand** — a group mean, a count, a percentage. Thirty seconds. It catches the error where the tool analysed a different subset than you asked for, which is invisible in a correct-looking output.
- [ ] **Check *n* and ask what was dropped.** If the numbers do not add up to your row count, find the missing cases before the p-value.
- [ ] **Run the whole analysis twice, in two fresh sessions**, and compare — inconsistent answers are measurably less accurate [12][43].
- [ ] **Validate the output against human judgment** [46] — and in a group, agree who does it: "any AI generated code needs to be carefully verified by lab members with programming experience" [47].
**Then disclose it.** COPE: AI used "in the collection and analysis of data" must be disclosed "in the Materials and Methods… how the AI tool was used and which tool was used" [31]. ICMJE puts writing help in the acknowledgments but "data collection, analysis, or figure generation" **in the methods** [32]. Elsevier is most explicit on code: declare it "in detail in the Methods section", because "the same standards for the validity and reproducibility of code apply whether the code was produced with AI assistance or not" [42]. Deposit the code permanently — Springer Nature requires "a repository that assigns a permanent identifier, such as Code Ocean or Zenodo… providing a GitHub link only is not sufficient" [30].
---
## The Personal AI Workflow Canvas — fillable
One page, **one project**, one date. Fill the *check* column first: if you cannot name the check, you have not decided to delegate the task — you have decided to hope.
**Project:** ______________________ **Dataset:** ______________________ **Date filled:** __________ **Target journal:** __________
| Lifecycle stage | What I hand to AI | Tier · Zone | My non-negotiable check | Artefact / rule |
|---|---|---|---|---|
| **1 · Frame** (S1) | | Tier ___ · ☐ Safe ☐ Unreliable ☐ Dangerous | | CRIT |
| **2 · Discover** (S2) | | Tier ___ · ☐ Safe ☐ Unreliable ☐ Dangerous | | Literature-Discovery Tool Matrix |
| **3 · Read** (S3) | | Tier ___ · ☐ Safe ☐ Unreliable ☐ Dangerous | | Summary-Trust Triage |
| **4 · Analyse** (S5) | | Tier ___ · ☐ Safe ☐ Unreliable ☐ Dangerous | ☐ keep code ☐ pin versions ☐ seed ☐ recompute one number | The Four Reproducibility Guardrails |
| **5 · Write** (S4) | | Tier ___ · ☐ Safe ☐ Unreliable ☐ Dangerous | | Publisher AI-Policy Comparison Table |
| **6 · Cross-field** (S6) | *— fill in during Session 6 —* | | | |
| **7 · My field's tools** (S7) | *— fill in during Session 7 —* | | | |
**Three rules.** (1) The check column first. (2) One project, not a career — a canvas for "my research" is a wish. (3) Date it and re-check in six months: every tool named across these five sessions changed while we ran them; none of the *stages* did.
---
## Further reading
- Sandve, G. K., et al. (2013). *Ten Simple Rules for Reproducible Computational Research*. PLoS Comput. Biol. 9(10), e1003285. https://doi.org/10.1371/journal.pcbi.1003285 — four pages, twelve years old, still the whole answer. Rules 1, 3 and 6 are guardrails 1–3.
- Zhu, Y., Du, S., Li, B., Luo, Y., & Tang, N. (2024). *Are Large Language Models Good Statisticians?* NeurIPS 2024 Datasets & Benchmarks. — the applicability-error finding: the most useful thing to know about how AI gets statistics wrong.
- Cui, J., & Alexander, R. (2026). *Same Prompt, Different Outcomes: Evaluating the Reproducibility of Data Analysis by LLMs*. arXiv:2602.14349 — 480 runs, one actionable conclusion.
- Trisovic, A., et al. (2022). *A large-scale study on research code quality and execution*. Scientific Data 9, 60. https://doi.org/10.1038/s41597-022-01143-6 — the pre-AI baseline; read it before concluding AI is the problem.
- Becker, J., et al. (2025). *Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity*. arXiv:2507.09089 — forecast 24% faster, felt 20% faster, measured 19% slower. Small study, large lesson on calibration.
- Elsevier. *Generative AI policies for journals*. https://www.elsevier.com/about/policies-and-standards/generative-ai-policies-for-journals — the clearest publisher statement on AI-written **code**. Read the FAQ, then your target journal's.
*Full annotated source list — 53 sources, with the verified wording and the caveat behind every number: see `sources.md`.*
---
*AI for Researchers · Session 5: Data, Code & Your AI Workflow · Landscape as of August 2026*