← series index

# Sources — Session 5: Data, Code & Your AI Workflow

*AI for Researchers · dated, annotated reference list · compiled 2026-07-28 · revised 2026-08-21 (current-generation companion figures added; entries [48]–[53]) · Landscape as of August 2026*

Entry format (one line per source, numbering matches the `[n]` footnote markers used in `slides.html` and `handout.md`):

`- [n] Author/Org (Year). *Title*. URL — accessed YYYY-MM-DD. [type: peer-reviewed | policy | primary-doc | news] — one-line note on what claim(s) it grounds.`

Allowed `type` values:
- `peer-reviewed` — peer-reviewed paper or arXiv/preprint literature
- `policy` — official publisher/funder/institutional policy page
- `primary-doc` — primary tool/vendor documentation (not a blog or listicle)
- `news` — recent news/benchmark article; informs tool discovery only, never the sole ground for a factual claim

**Evidence rule for this session.** Every number this deck puts on a slide about how well AI performs an analysis comes from a *published evaluation study* with a stated task set, not from a vendor's claim about its own model. Where a benchmark result is contested, dated, or measured on a narrow task set, the caveat is stated on the slide, not only here. Facts about what a *product does* come from that product's own current documentation and nowhere else.

---

## References

### A. What these tools actually do — primary documentation for the no-code and low-code paths

- [1] OpenAI (2026). *Data analysis with ChatGPT* (and companion page *File storage and Library in ChatGPT*). https://help.openai.com/en/articles/8437071-data-analysis-with-chatgpt and https://help.openai.com/en/articles/20001052-file-storage-and-library-in-chatgpt — accessed 2026-07-28. [type: primary-doc] — Grounds the deck's central mechanical claim that the "no-code" path is a code path with the code hidden. Quoted verbatim: "For some data-analysis tasks, ChatGPT writes and runs Python code in a **stateful Jupyter notebook environment**"; "When ChatGPT uses Python for analysis, **review the generated code, outputs, and assumptions before relying on the result.** If the answer depends on a specific method, ask ChatGPT to show or adjust that method"; "The Python environment used for data analysis cannot make external web requests or API calls." Documented failure mode quoted on the data-governance slide: "ChatGPT may not reliably extract exact values from image-based tables, scanned files, or files with complex visual layouts." Limits from the companion page: "hard limit of 512MB per file", "capped at 2M tokens per file. This limitation does not apply to spreadsheets", "For CSV files or spreadsheets, the file size cannot exceed approximately 50MB". Training setting quoted: OpenAI "may use content (including files you upload)… to improve model performance if you have 'Improve the model for everyone' turned on. You can turn it off anytime in Settings > Data Controls." **Naming note stated on the slide:** the current official page is titled *Data analysis with ChatGPT*; the older "Advanced Data Analysis" branding does not appear on it, so the deck does not use that name.
- [2] Anthropic (2026). *Create and edit files with Claude* and *Code execution tool* (Claude Platform docs). https://support.claude.com/en/articles/12111783-create-and-edit-files-with-claude and https://platform.claude.com/docs/en/agents-and-tools/tool-use/code-execution-tool — accessed 2026-07-28. [type: primary-doc] — Second row of the no-code comparison table, and the source of the deck's sandbox-limits line. Quoted: files are produced in "an **isolated, sandboxed container**"; "no sandbox environments are ever shared between users"; "The maximum file size is 30MB per file for both uploads and downloads"; available to "all Claude users (Free, Pro, Max, Team, and Enterprise)". Platform docs give the exact resource envelope quoted in the handout: **5 GiB RAM, 5 GiB disk, 1 CPU**, "Internet access: Completely disabled for security", "Containers expire 30 days after creation", pre-installed "pandas, numpy, scipy, scikit-learn, statsmodels… matplotlib, seaborn", and "The container has no internet access, so Claude can't download or install additional packages at runtime: only the pre-installed libraries are available." Security warning quoted on the data-governance slide: users should "monitor Claude while using the feature" because Claude "can be tricked into sending information from its context… to malicious third parties."
- [3] Google (2026). *Use the Data Science Agent* (Colab Enterprise documentation). https://docs.cloud.google.com/colab/docs/use-data-science-agent — accessed 2026-07-28. [type: primary-doc] — The clearest documented example of a no-code path that *hands you the code*, which is why it earns a "yes" in the table's "can you get the code out?" column. Quoted: the agent "can help you with tasks ranging from exploratory data analysis to generating machine learning predictions and forecasts"; and, on every suggestion, "you can click: **Accept** to add the code to your notebook. **Accept and run** to add the code to your notebook and run the code." Supported sources: "CSV files [and] BigQuery tables". Constraint quoted: "The code produced by the Data Science Agent only runs in your notebook's runtime."
- [4] Google (2026). *Upload & analyze files in Gemini Apps*. https://support.google.com/gemini/answer/14903178 — accessed 2026-07-28. [type: primary-doc] — Documents the consumer-app upload limits used in the comparison table: "Up to 10 files (subject to availability) can be uploaded in the same prompt"; "All other supported file types can be up to 100 MB"; charts can be generated from uploaded spreadsheets. **Explicit non-claim, stated in the table:** this page does *not* document that Gemini Apps writes or executes visible Python, so the deck does not say that it does; the Colab agent [3] is cited instead where code visibility matters. **Re-verified 2026-09-02:** the page still documents upload limits, supported file types and chart generation only — no mention of writing or executing visible Python — so the non-claim, and the deck's "one major assistant" line, stand unchanged.
- [5] Microsoft (2026). *Get direct answers to your data analysis questions* (Copilot in Excel) and *Data security and Python in Excel*. https://support.microsoft.com/en-us/excel/get-direct-answers-to-your-data-analysis-questions and https://support.microsoft.com/en-us/office/data-security-and-python-in-excel-33cc88a4-4a87-485e-9ff9-f35958278327 — accessed 2026-07-28. [type: primary-doc] — The spreadsheet row of the table and the deck's single best "the no-code tool is running code" quote: "Copilot uses **Python-based analysis** to interpret your question, identify relevant data in your workbook, and compute a response", and users can "**expand the code to see how the answer was generated**" and ask Copilot to "generate a summary of the code using large-language model (LLM) capabilities to verify its logic". Caveat quoted in the table: "Copilot does not modify your workbook. Sometimes, it may return a static table or image visualization that you can insert into your sheet, but these aren't refreshable." Security page quoted on the governance slide: Python code "runs within the compliance boundary of your organization on hypervisor isolated containers", "Data isn't persisted in the Microsoft Cloud", "The Python code doesn't have network access."
- [6] Julius AI (2026). *What is Julius AI?*, *Files*, and *Containers* (product documentation). https://julius.ai/docs/get-started/what-is-Julius-AI , https://julius.ai/docs/get-started/files , https://julius.ai/docs/get-started/containers — accessed 2026-07-28. [type: primary-doc] — The dedicated no-code analysis product in the table. Quoted: it "generates SQL and Python when needed", and its notebooks let you "View, edit, and customize generated code (e.g., SQL, Python)" and "Re-run, version, and share notebooks for reproducibility". Accepts .csv, .xlsx, .json, .parquet, **.sav (SPSS)**, .db/.sqlite and more. Retention quoted in the governance line: free-tier files are deleted after 1 hour of inactivity, paid-tier after 7 days. Container tiers documented: standard 1 hour / 8 GB RAM; Pro+ 3 hours or 32 GB RAM. **Explicit non-claim:** no official page documents a per-file size limit or a model-training policy, and the widely repeated "handles 32 GB files" claim is a third-party misreading of the 32 GB *RAM* tier — the deck says "not documented" rather than guessing.
- [7] JASP / University of Amsterdam (2026, published 2 July 2026). *Introducing JASP 0.98: Fully Integrated AI Support* (and *Getting started*). https://jasp-stats.org/2026/07/02/introducing-jasp-0-98-fully-integrated-ai-support/ and https://jasp-stats.org/getting-started/ — accessed 2026-07-28. [type: primary-doc] — The point-and-click statistics row, and the newest thing in the session: as of 2 July 2026, JASP ships optional AI support that "needs to be activated through the JASP Preferences setting, where the user is asked to provide an API key", after which "the AI is available to annotate analyses, interpret results, conduct alternative analyses". Reproducibility property quoted in the table: "The JASP-file besides the data also holds analyses and interpretations", which you can "edit, reorder or extend". **Caveat stated on the slide:** neither announcement names the model provider or documents where your data goes, so the deck labels it "opt-in, bring-your-own-key, data handling not yet documented".
- [8] jamovi (2026). *Combining jamovi and R* (user manual). https://docs.jamovi.org/usermanual/um_6_jamovi_and_R.html — accessed 2026-07-28. [type: primary-doc] — The non-AI reproducible-GUI baseline in the table, included because it is the cheapest way for a non-programmer to satisfy guardrail 1. Quoted: jamovi is "built on top of the R statistical language"; enabling `Syntax mode` produces "equivalent R code for each analysis", which you can "copy and paste… into an R session"; all built-in analyses are also "available within an R session through the R package jmv". Documented limitation quoted: with System R, "sharing jamovi files with colleagues becomes a bit more complicated" because they need the same packages installed. As of this check jamovi's documentation describes no AI feature — a deliberate contrast with [7].

### B. How well AI actually performs an analysis — the evaluation literature

- [9] Chen, Z., Chen, S., Ning, Y., Zhang, Q., Wang, B., et al. (2025). *ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery*. ICLR 2025. https://arxiv.org/abs/2410.05080 — accessed 2026-07-28. [type: peer-reviewed] — **The headline evaluation number in the deck, and the one closest to what an attendee would actually ask an agent to do.** 102 tasks extracted from 44 peer-reviewed publications across four disciplines, validated by nine subject-matter experts; every task's target output is a self-contained Python program. Quoted verbatim: "Given three attempts for each task, the best-performing agent can only solve **32.4%** of the tasks independently and **34.3%** with expert-provided knowledge." OpenAI o1-preview reached **42.2%** via self-debug "with more than 10 times the cost of other LLMs". Caveats stated on the slide: pass@3 rather than single-shot; tasks derived from published papers, so some contamination risk; models tested are 2024–early-2025 vintage. **August-2026 status:** the benchmark is effectively frozen — the official project page hosts no leaderboard, HAL's best entry is **33.3%** ("SAB Self-Debug", o3 Medium, April 2025), and HAL has paused new-model updates [49]; the community's successor suite for agentic science is AstaBench [48], which the slide now names alongside this row.
- [10] Zhu, Y., Du, S., Li, B., Luo, Y., & Tang, N. (2024). *Are Large Language Models Good Statisticians?* (the StatQA benchmark). Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track. https://proceedings.neurips.cc/paper_files/paper/2024/hash/729786203d330da046dd8091c2d92a66-Abstract-Datasets_and_Benchmarks_Track.html — accessed 2026-07-28. [type: peer-reviewed] — **The most important slide in the session's evidence section**, because it identifies *which kind* of error to look for. StatQA comprises 11,623 examples. Quoted verbatim: "even state-of-the-art models such as **GPT-4o achieve a best performance of only 64.83%**"; and, crucially, "our comparative human experiments highlight a striking contrast in error types between LLMs and humans: **LLMs primarily make applicability errors, whereas humans mostly make statistical task confusion errors.**" An applicability error is running a test whose assumptions do not hold for the data — an error that leaves no trace in the output, which is exactly why the deck builds a verification habit rather than an arithmetic check. Caveat on the slide: GPT-4o-era (mid-2024), derived hypothesis-testing tasks rather than full analyses of real study data.
- [11] Gu, K., Shang, R., Jiang, R., Kuang, K., Lin, R.-J., et al. (2024). *BLADE: Benchmarking Language Model Agents for Data-Driven Science*. Findings of EMNLP 2024. https://arxiv.org/abs/2408.09667 — accessed 2026-07-28. [type: peer-reviewed] — Grounds the deck's "it gave you *an* analysis, not *the* analysis" claim. Ground truth is the set of analysis decisions made by 11 trained expert annotators on 12 datasets and research questions drawn from the literature, yielding 118 conceptual-variable decisions, 246 transform decisions and 172 modelling decisions. Result quoted: coverage of ground-truth statistical models with conceptual variables stayed "below 13%" (coverage@10) across all models tested, and coverage for variable operationalisation stayed below 27%. Caveats stated: only 12 research questions; 2024-era models; coverage@10 means ten sampled analyses per model.
- [12] Zeng, Q., Jin, C., Wang, X., Zheng, Y., & Li, Q. (2025). *AIRepr: An Analyst-Inspector Framework for Evaluating Reproducibility of LLMs in Data Science*. Findings of EMNLP 2025. https://arxiv.org/abs/2502.16395 — accessed 2026-07-28. [type: peer-reviewed] — The direct evidence for the deck's "ask it twice" rule. 15 analyst-inspector LLM pairs across 1,032 tasks from three public benchmarks (DiscoveryBench, QRData, StatQA). Under standard chain-of-thought prompting, workflow reproducibility ranged from **21.34%–55.23%** on DiscoveryBench, 30.28%–75.06% on QRData and 48%–81% on StatQA, depending on model; human experts scored 66.53% on DiscoveryBench. The finding the slide leads with, quoted: "reproducible solutions were significantly more accurate than irreproducible ones (mean **64.4% vs. 53.5%**; one-sided paired t-test, p < 0.001)." Caveat stated on the slide: "reproducibility" here means consistency of the generated *workflow*, judged by an LLM inspector — not bit-identical numeric output. **Consistency note (August-2026 pass):** the deck and handout now quote the full cross-benchmark span, **21–81%**, everywhere this study is summarised in one range, with the per-benchmark ranges (21–55% / 30–75% / 48–81%) stated where space allows; an earlier draft's "19–79%" did not match the paper and has been corrected.
- [13] Jing, L., Huang, Z., Wang, X., Yao, W., Yu, W., et al. (2025). *DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?* ICLR 2025. https://arxiv.org/abs/2409.07703 — accessed 2026-07-28. [type: peer-reviewed] — The realistic-task corroboration of [9], on tasks with long contexts, multiple tables and large data files. Quoted: the benchmark "includes **466 data analysis tasks and 74 data modeling tasks**, sourced from Eloquence and Kaggle competitions", and "the best agent solv[es] only **34.12%** of data analysis tasks". Caveat: competition-derived tasks are more tightly specified than real research questions, which if anything makes 34% an optimistic figure for research use.
- [14] Tian, M., Gao, L., Zhang, S. D., Chen, X., Fan, C., et al. (2024). *SciCode: A Research Coding Benchmark Curated by Scientists*. NeurIPS 2024, Datasets and Benchmarks Track; official repository and leaderboard at https://github.com/scicode-bench/SciCode . https://arxiv.org/abs/2407.13168 — accessed 2026-07-28. [type: peer-reviewed] — The hardest number in the deck, used to bound expectations for genuinely research-level code. 338 subproblems decomposed from 80 main problems across 16 natural-science sub-fields. Quoted from the paper: "Claude3.5-Sonnet, the best-performing model among those tested, can solve only **4.6%** of the problems in the most realistic setting"; the project's own README reports **7.7%** for o1-preview, and the leaderboard's best main-problem resolve rate is **10.8%** (o3-mini-low). **The deck quotes the model alongside the number** because the 4.6% and 7.7% figures come from different versions of the same work. "Most realistic setting" means without scientist-written background hints. **August-2026 status:** the official leaderboard is **stale** — its top entry is still o3-mini-low at 10.8% main-problem / 33.3% subproblem, with no 2026 frontier models listed. Artificial Analysis independently runs the 288 test-set subproblems on current models: **Claude Fable 5 60.2%**, Gemini 3.1 Pro Preview 58.9%, Kimi K3 58.7% [50]. Those are **subproblem-level** figures — comparable to the ~26–33% subproblem rates of the 2024–25 cohort, NOT to the 4.6%/10.8% main-problem rates; no verified 2026 main-problem figure exists, and the slide says so.
- [15] Siegel, Z. S., Kapoor, S., Nadgir, N., Stroebl, B., & Narayanan, A. (2024, rev. 2026). *CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark*. arXiv:2409.11363. https://arxiv.org/abs/2409.11363 — accessed 2026-07-28. [type: peer-reviewed] — Cited on the reproducibility slide because it measures the *easiest possible* version of the task: reproduce a published result when the code and data are already handed to you. 270 tasks based on 90 scientific papers across computer science, social science and medicine. Quoted: "The best agent achieved an accuracy of **21% on the hardest task**, showing the vast scope for improvement in automating routine scientific tasks." Caveat: baselines were GPT-4o-era agents; preprint. **August-2026 status:** HAL's CORE-Bench Hard board shows **77.8%** accuracy for an agent on **Claude Opus 4.5 using Claude Code** (submitted by Nicholas Carlini, $87.16/run), with **manual scoring reaching 95.5%** [49]. HAL is paused for new models, so no Fable 5 / GPT-5.6-class entries exist — but 21% → 77.8% means the hard computational-reproducibility split is close to solved at Opus 4.5 class, and the trajectory slide now argues from that.
- [16] Chan, J. S., Chowdhury, N., Jaffe, O., Aung, J., Sherburn, D., et al. (2025). *MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering*. ICLR 2025 version, arXiv:2410.07095; live leaderboard at https://github.com/openai/mle-bench . https://arxiv.org/abs/2410.07095 — accessed 2026-07-28. [type: peer-reviewed] — The trajectory slide's evidence, and the honest counterweight to the pessimistic benchmarks. The paper curates "**75 ML engineering-related competitions from Kaggle**" and reports that "the best-performing setup — OpenAI's o1-preview with AIDE scaffolding — achieves at least the level of a **Kaggle bronze medal in 16.9% of competitions**" (October 2024). The project's own leaderboard's top entry is now **64.44%** any-medal — Famou-Agent 2.0 on Gemini-3-Pro-Preview, dated 2026-02-23 (README re-checked 2026-09-02; next entries: 63.11% on Claude-Opus-4.6, 62.67% and lower on Gemini-3-Pro-Preview). **Caveats stated on the slide:** the benchmark is authored by a model vendor; a bronze medal is a modest bar; and the README states, verbatim, "We are currently not taking any new submissions to the leaderboard while we develop an improved process for ensuring submissions are fair and comparable" — so the 2026 figures are self-reported entries, not an audited ranking.
- [17] Zhu, Y., Ding, Y., Lai, P., Wang, L., Jing, B., & Chen, G. (2026, 22 June 2026). *StatABench: Dataset and Framework for Evaluating Statistical Analysis Capabilities of LLMs*. arXiv:2606.22977. https://arxiv.org/abs/2606.22977 — accessed 2026-07-28. [type: peer-reviewed] — The most current statistics-specific measurement available, used to show the gap has narrowed without closing. 404 closed-form questions across 18 statistical topics plus 30 open-ended modelling tasks. Quoted: "even **GPT-5.1 achieves only 68.6%** on Stat-Closed tasks, while the best open-source model reaches 60.6%." Error analysis of the failing cases: tool-selection failure 50%, compliance and execution 26%, result interpretation 24%. **Caveats stated on the slide:** preprint, not yet peer-reviewed; the error breakdown rests on only 38 cases; human–machine grading agreement was moderate (Fleiss' κ 0.41–0.51). Re-checked 2026-08-21: still the current statistics-specific measurement; on the open-ended Stat-Open tasks the best agent framework averages 61.86. The deck now also cites this row on the applicability-error slide and the trajectory slide, as the current-generation evidence that the session's statistics caution survives.
- [18] Shahrul, A. I., & Syed Mohamed, A. M. F. (2024). *A Comparative Evaluation of Statistical Product and Service Solutions (SPSS) and ChatGPT-4 in Statistical Analyses*. Cureus 16(10), e72581. https://doi.org/10.7759/cureus.72581 — accessed 2026-07-28. [type: peer-reviewed] — Grounds the deck's claim that the failures cluster in specific, predictable places rather than being random. ChatGPT-4 matched SPSS exactly on independent and paired t-tests, Pearson correlation and simple linear regression, but diverged on Tukey HSD post-hoc comparisons (comparison order reversed, flipping the sign of mean differences and swapping confidence-interval bounds), Wilcoxon signed-rank (Z = −1.246 vs −1.46), Mann–Whitney U (U = 449.000 vs 451.000, Z of opposite sign) and ICC confidence intervals. Conclusion quoted: "discrepancies in post-hoc analyses, confidence intervals, and more complex tests indicate that **careful validation is required**… Researchers should exercise caution and cross-validate results with established tools such as SPSS." **Caveats stated on the slide:** synthetic datasets with n = 30–100 per test, a single analyst, no repeated runs, and a low-barrier peer-reviewed venue — so it is quoted for the *pattern* of failure, not as an error rate.
- [19] Shukla, M., Pandey, D., Kaur, S., Agarwal, M., Goyal, A., & Sharma, H. (2025). *Evaluating the Accuracy and Explanatory Quality of Large Language Models ChatGPT, Claude, DeepSeek, Gemini, Grok, and Le Chat in Statistical Test Selection for Hypothesis Testing Decisions*. Cureus 17(10), e94949. https://doi.org/10.7759/cureus.94949 — accessed 2026-07-28. [type: peer-reviewed] — **The deliberate counterweight, included so the session is not a one-sided warning.** Twenty standardised hypothesis-testing scenarios, six models, five independent biostatistician raters. Quoted: "All six LLMs achieved **100% accuracy** in statistical test selection across all 20 hypothesis scenarios", though "significant variations emerged in explanatory quality" (statistical-reasoning ratings ranged 3.16–4.66 on a five-point scale, with regression methods rated lowest). **Caveats stated verbatim on the slide, because they carry the argument:** testing occurred on a single day (21 August 2025); free model versions only; and "scenarios focused on basic biomedical research rather than advanced biostatistics". Read with [10] and [17], the honest summary is: choosing the textbook test is close to solved; *applying* it correctly to your data is not.

### C. What goes wrong inside the generated code

- [20] Zhang, Z., Wang, Y., Wang, C., Chen, J., & Zheng, Z. (2025). *LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation*. ISSTA 2025. https://arxiv.org/abs/2409.20550 — accessed 2026-07-28. [type: peer-reviewed] — The taxonomy behind the deck's "the code runs and answers the wrong question" slide. Manual annotation of 1,380 code snippets (six models × ten snippets × 23 tasks). Distribution quoted: **Task Requirement Conflicts 43.53%** (functional requirement violation 36.66%), Factual Knowledge Conflicts 31.91% (API knowledge 20.41%), Project Context Conflicts 24.56%. **Stated precisely on the slide:** these are percentages *of the hallucinations found*, not of all generated code — the deck does not say "43% of AI code is wrong".
- [21] Spracklen, J., Wijewickrama, R., Sakib, A. H. M. N., Maiti, A., Viswanath, B., & Jadliwala, M. (2025). *We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs*. 34th USENIX Security Symposium. https://arxiv.org/abs/2406.10279 — accessed 2026-07-28. [type: peer-reviewed] — The strongest single citation for "it will confidently tell you to install a library that does not exist". Quoted verbatim: "Using **16 popular LLMs** for code generation and two unique prompt datasets, we generate **576,000 code samples** in two programming languages… the average percentage of hallucinated packages is at least **5.2% for commercial models and 21.7% for open-source models**, including a staggering **205,474 unique examples of hallucinated package names**." Python and JavaScript only; models sampled mid-2024. A 2026 preprint re-measuring five frontier models reports rates of 4.62%–6.10% — a narrowing spread rather than a solved problem — promoted in the August-2026 pass to its own entry [51] and paired on the slide with this study's 2024 baselines; [21] remains the peer-reviewed headline figure.
- [22] Fu, Y., Liang, P., Tahir, A., Li, Z., Shahin, M., Yu, J., & Chen, J. (2025). *Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study*. ACM Transactions on Software Engineering and Methodology. https://arxiv.org/abs/2310.02059 — accessed 2026-07-28. [type: peer-reviewed] — Cited because it measures AI-generated code *as actually committed to real repositories*, not in a benchmark. 733 AI-generated snippets mined from real GitHub projects: security weaknesses in **29.5% of Python** and **24.2% of JavaScript** snippets, spanning 43 CWE categories, 8 of them in the 2023 CWE Top-25. The constructive finding quoted in the handout: pairing static analysis with the assistant fixed **up to 55.5%** of the identified weaknesses. Caveats: snippets identified by developer attribution, so not a random sample; weaknesses detected by static analysis, which produces false positives; 2023-era assistants.
- [23] Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2023). *Do Users Write More Insecure Code with AI Assistants?* ACM SIGSAC Conference on Computer and Communications Security (CCS '23). https://arxiv.org/abs/2211.03622 — accessed 2026-07-28. [type: peer-reviewed] — The mechanism behind the deck's overconfidence argument, quoted verbatim: "participants who had access to an AI assistant… **wrote significantly less secure code** than those without access. Additionally, participants with access to an AI assistant were **more likely to believe they wrote secure code** than those without access." Specifics: 47 participants (33 experiment, 14 control) across five tasks; on the signing task 3% of AI-assisted versus 21% of control produced a secure solution (p = 0.039); 36% of AI-assisted versus 7% of control wrote SQL-injection-vulnerable code (p = 0.041). Also relevant to the prompting habit: participants who trusted the assistant less and iterated more on their prompts produced fewer vulnerabilities. **Caveats stated on the slide:** n = 47, unbalanced arms, student-heavy sample, and a 2022-era model — so the deck presents the *confidence miscalibration*, not the exact percentages, as the durable finding.

### D. Productivity, and the gap between how it feels and what it does

- [24] Becker, J., Rush, N., Barnes, E., & Rein, D. (2025). *Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity*. arXiv:2507.09089 (METR). https://arxiv.org/abs/2507.09089 — accessed 2026-07-28. [type: peer-reviewed] — The randomised controlled trial behind the perception-gap slide, quoted verbatim: "16 developers with moderate AI experience complete 246 tasks in mature projects on which they have an average of 5 years of prior experience… Before starting tasks, developers forecast that allowing AI will reduce completion time by **24%**. After completing the study, developers estimate that allowing AI reduced completion time by **20%**. Surprisingly, we find that allowing AI actually **increases completion time by 19%** — AI tooling slowed developers down." Expert forecasts were 39% (economics) and 38% (ML) shorter. **Caveats stated on the slide, at length, because they matter:** n = 16; tools were Cursor Pro with Claude 3.5/3.7 Sonnet, February–June 2025, which METR itself treats as a historical snapshot; and the population — experts working on codebases they already know deeply — is close to the *opposite* of this session's audience. The transferable finding is the ~39-point gap between felt and measured speed, not the −19% itself.
- [25] Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M. (2023). *The Impact of AI on Developer Productivity: Evidence from GitHub Copilot*. arXiv:2302.06590. https://arxiv.org/abs/2302.06590 — accessed 2026-07-28. [type: peer-reviewed] — Cited on the same slide as [24] so the deck does not present one RCT as the whole literature. Quoted: "The treatment group, with access to the AI pair programmer, completed the task **55.8% faster** than the control group." **Caveats stated on the slide:** the authors are affiliated with the vendor; the task was a single, self-contained, unambiguously specified greenfield exercise (implement an HTTP server); and the abstract reports nothing about whether the resulting code was correct or secure. Together with [24] the honest reading is that the gain scales with how self-contained and novel the task is, and shrinks or reverses as it depends on context you already hold.

### E. Reproducibility — the baseline this session inherits

- [26] Trisovic, A., Lau, M. K., Pasquier, T., & Crosas, M. (2022). *A large-scale study on research code quality and execution*. Scientific Data 9, 60. https://doi.org/10.1038/s41597-022-01143-6 — accessed 2026-07-28. [type: peer-reviewed] — The deck's framing statistic, and the reason the session is not an argument that AI is uniquely unreliable. Analysis of "more than 2000 replication datasets with over 9000 unique R files published from 2010 to 2020" at the Harvard Dataverse, re-executed in a clean environment: "**74% of R files failed to complete without error in the initial execution, while 56% failed when code cleaning was applied.**" Caveat stated on the slide: "failed to complete without error" includes environmental causes such as hard-coded paths and absent packages — which is precisely why the drop from 74% to 56% after automated cleaning is quoted alongside the headline.
- [27] Samuel, S., & Mietchen, D. (2024). *Computational reproducibility of Jupyter notebooks from biomedical publications*. GigaScience 13, giad113. https://doi.org/10.1093/gigascience/giad113 — accessed 2026-07-28. [type: peer-reviewed] — The biomedical corroboration of [26], and the source of the deck's "dependencies are the failure mode" claim. Of 27,271 notebooks from 2,660 GitHub repositories associated with 3,467 publications, 15,817 Python notebooks had declared dependencies and were re-run automatically; for 10,388 all dependencies installed; of those, **1,203 ran without any errors, and 879 produced results identical to those originally reported**. Dominant exceptions were ModuleNotFoundError/ImportError (41.65%) and FileNotFoundError/IOError (7.9%). Authors' conclusion quoted: "the large majority of these notebooks could not be executed automatically, mostly due to issues with the documentation of dependencies."
- [28] Stodden, V., Seiler, J., & Ma, Z. (2018). *An empirical analysis of journal policy effectiveness for computational reproducibility*. PNAS 115(11), 2584–2589. https://doi.org/10.1073/pnas.1708290115 — accessed 2026-07-28. [type: peer-reviewed] — Cited on the policy slide to make the point that a sharing *policy* is not the same as a reproducible *result*. From a sample of 204 papers published in *Science* after its February 2011 data and code policy took effect: "we were able to obtain artifacts from **44%** of our sample and were able to reproduce the findings for **26%**." Caveats: single journal, request-based rather than deposit-based policy, 2018 study.
- [29] Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E. (2013). *Ten Simple Rules for Reproducible Computational Research*. PLoS Computational Biology 9(10), e1003285. https://doi.org/10.1371/journal.pcbi.1003285 — accessed 2026-07-28. [type: peer-reviewed] — **The intellectual anchor of the whole guardrails section**, and the reason the deck can say the rules are not new. Four of the ten rules are the session's four guardrails almost verbatim: Rule 1 "For Every Result, Keep Track of How It Was Produced"; Rule 3 "Archive the Exact Versions of All External Programs Used"; Rule 6 "For Analyses That Include Randomness, Note Underlying Random Seeds"; Rule 10 "Provide Public Access to Scripts, Runs, and Results". Rule 4 ("Version Control All Custom Scripts") and Rule 2 ("Avoid Manual Data Manipulation Steps") support the handout's checklist. Published 2013 — twelve years before the tools this session is about.

### F. What the rules already require of an AI-assisted analysis

- [30] Springer Nature (2026). *Code policy* (open science / editorial policies). https://www.springernature.com/gp/open-science/code-policy — accessed 2026-07-28. [type: policy] — The most operationally specific publisher requirement in the session, quoted verbatim on the policy slide: "Code availability statements should include information on what code is available, where these can be found, and any applicable access terms"; "Code should be deposited in a repository that assigns a permanent identifier, such as Code Ocean or Zenodo, and cited in the reference list"; and the line that converts guardrail 1 into a submission requirement — "**providing a GitHub link only is not sufficient as it does not assign a permanent identifier to the code**". Where code cannot be shared, "code availability should still be stated in the manuscript along with any conditions for access."
- [31] COPE Council (2023, last reviewed 13 February 2023). *COPE position statement: Authorship and AI tools*. https://publicationethics.org/guidance/cope-position/authorship-and-ai-tools — accessed 2026-07-28. [type: policy] — Carried forward from Session 4, but cited here for the clause that session did not need: authors who use AI tools "in the writing of a manuscript, production of images or graphical elements of the paper, **or in the collection and analysis of data**, must be transparent in **disclosing in the Materials and Methods (or similar section)** of the paper **how the AI tool was used and which tool was used**." Also: "AI tools cannot be listed as an author of a paper."
- [32] International Committee of Medical Journal Editors (2025). *Recommendations — Defining the Role of Authors and Contributors (II.A.4, AI-Assisted Technology)*. https://www.icmje.org/recommendations/browse/roles-and-responsibilities/defining-the-role-of-authors-and-contributors.html — accessed 2026-07-28. [type: policy] — The disclosure rule that differs for analysis and for writing, quoted on the policy slide: "If AI was used for **data collection, analysis, or figure generation**, authors should describe this use **in the methods**" — as against writing assistance, which belongs in the acknowledgments. Also: "Humans are responsible for any submitted material that included the use of AI-assisted technologies", and "Chatbots (such as ChatGPT) should not be listed as authors."

### G. Agentic tools — what "agentic" means, from the vendors' own documentation

- [33] GitHub (2026). *GitHub Copilot features*. https://docs.github.com/en/copilot/get-started/features — accessed 2026-07-28. [type: primary-doc] — The deck's operational definition of "agentic", quoted verbatim: agent mode lets the tool "work autonomously in the IDE… determine which files to make changes to, **offer code changes and terminal commands for the user's approval**, and iterate to remediate issues until the original task is complete"; the cloud agent "can research a repository, create an implementation plan, and make code changes on a branch". Oversight quoted: agentic features "typically need **human approval to perform sensitive actions**, such as running commands in a developer's terminal or merging a pull request."
- [34] Anthropic (2026). *Claude Code — Security*. https://code.claude.com/docs/en/security — accessed 2026-07-28. [type: primary-doc] — The safeguards row of the agentic slide. Quoted: "Claude Code uses **strict read-only permissions by default**. When additional actions are needed (editing files, running tests, executing commands), Claude Code **requests explicit permission**"; it "can only write to the folder where it was started and its subfolders"; "**Fail-closed matching**: Unmatched commands default to requiring manual approval." The warning the deck quotes on the slide: "While these protections significantly reduce risk, **no system is completely immune to all attacks**", and "**You're responsible for reviewing proposed code and commands for safety before approval**."
- [35] OpenAI (2026). *Codex — Agent approvals & security*. https://learn.chatgpt.com/docs/agent-approvals-security — accessed 2026-07-28. [type: primary-doc] — Second safeguards source, documenting the same pattern under different names: approval policies **on-request / never / untrusted**; sandbox modes **read-only / workspace-write / danger-full-access**; "By default, the agent runs with **network access turned off**." Risk quoted in the handout: "Use caution when enabling network access or web search in Codex. **Prompt injection can cause the agent to fetch and follow untrusted instructions**."
- [36] Project Jupyter (2026). *Jupyter AI v3 documentation*. https://jupyter-ai.readthedocs.io/en/v3/ — accessed 2026-07-28. [type: primary-doc] — The researcher-facing agentic path, included because it is the one that lands inside a notebook the attendee already has. Quoted: "An open source extension that **connects AI agents to computational notebooks in JupyterLab**", supporting a range of third-party agents, with the permission model stated plainly: "**Agents request permission before writing files, running commands, or using MCP tools.**" Ships with no agent installed by default.

### H. Staying current — venues that are citable, first-party, and still there next year

- [37] arXiv (2026). Category listings: *cs.CL Computation and Language*, *cs.SE Software Engineering*, *stat.ML Machine Learning*. https://arxiv.org/list/cs.CL/recent , https://arxiv.org/list/cs.SE/recent , https://arxiv.org/list/stat.ML/recent — accessed 2026-07-28. [type: primary-doc] — The "staying current" slide's first recommendation, and the one with the shortest lag: on the day of access these listings carried 445, 116 and 129 recent entries respectively. Cited as the primary venue where the benchmarks in section B appear months before they are peer-reviewed — with the corollary the slide states out loud: an arXiv listing is a preprint feed, not a filter.
- [38] Official proceedings: ACL Anthology (https://aclanthology.org/), NeurIPS Proceedings including the Datasets and Benchmarks Track (https://papers.nips.cc/), PMLR / ICML Proceedings (https://proceedings.mlr.press/), ICLR (https://iclr.cc/) — all accessed 2026-07-28. [type: primary-doc] — The peer-reviewed counterpart to [37], and the specific place the evaluation literature in section B lands: [9], [13] and [16] are ICLR 2025; [10] and [14] are the NeurIPS Datasets and Benchmarks Track; [11] and [12] are EMNLP Findings, indexed in the ACL Anthology, which hosts 128,689 papers and licenses post-2016 content CC BY 4.0. The deck names the Datasets and Benchmarks Track specifically, because that is where evaluations of the kind an attendee should trust are published.
- [39] Publisher news desks: *Nature* news (https://www.nature.com/nature/articles?type=news) and *Science* news (https://www.science.org/news/all-news) — accessed 2026-07-28. [type: news] — Recommended on the staying-current slide as editorially staffed, first-party summaries for readers outside computer science; labelled as news, and explicitly not used as the ground for any factual claim in this deck.

### I. The demo dataset

- [40] Horst, A. M., Hill, A. P., & Gorman, K. B. (2020). *palmerpenguins: Palmer Archipelago (Antarctica) penguin data*. R package version 0.1.0. https://allisonhorst.github.io/palmerpenguins/ — DOI 10.5281/zenodo.3960218 (archived record: https://zenodo.org/records/3960218) — accessed 2026-07-28. [type: primary-doc] — The demo dataset, chosen because it is small, genuinely open, needs real cleaning decisions, and has a peer-reviewed paper attached so the demo's conclusion can be checked against published science. Licence quoted verbatim from the package documentation: "Data are available by **CC-0 license** in accordance with the Palmer Station LTER Data Policy." **Licence provenance checked at three independent levels on 2026-07-28**, because a slide should not rest on one page: the repository's `LICENSE.md` carries the full **CC0 1.0 Universal** text (https://raw.githubusercontent.com/allisonhorst/palmerpenguins/main/LICENSE.md); CRAN's package page records `License: CC0` (https://cran.r-project.org/package=palmerpenguins); and the package documentation states the LTER data policy. **Caveat worth knowing:** the Zenodo record's own licence field reads only "Other (Open)", so the demo script cites Zenodo for the DOI and CRAN/`LICENSE.md` for the licence. Contents confirmed against the CSV on 2026-07-28: 344 rows, 8 variables (`species`, `island`, `bill_length_mm`, `bill_depth_mm`, `flipper_length_mm`, `body_mass_g`, `sex`, `year`); 2 rows missing all four measurements and 11 missing `sex`. Direct CSV: https://raw.githubusercontent.com/allisonhorst/palmerpenguins/main/inst/extdata/penguins.csv ; **drift-proof pinned equivalent** (verified byte-identical, same MD5, on 2026-07-28): https://raw.githubusercontent.com/allisonhorst/palmerpenguins/156daa4301838d9fdcc5b018b29b9149bf975552/inst/extdata/penguins.csv — all URLs confirmed to resolve 2026-07-28.
- [41] Gorman, K. B., Williams, T. D., & Fraser, W. R. (2014). *Ecological sexual dimorphism and environmental variability within a community of Antarctic penguins (genus Pygoscelis)*. PLOS ONE 9(3), e90081. https://doi.org/10.1371/journal.pone.0090081 — accessed 2026-07-28. [type: peer-reviewed] — The source paper for [40], and the demo's final verification step: the published finding quoted in the demo script is "Adélies and gentoos were more sexually dimorphic in body mass than chinstraps", which the live analysis reproduces (male − female body mass: Gentoo 805 g, Adélie 675 g, Chinstrap 412 g, computed independently from the CSV on 2026-07-28). Open access under CC BY.

### J. Added on second pass — policy on AI-written code, and the non-determinism problem

- [42] Elsevier (2026, policy updated June 2026). *Generative AI policies for journals* — including the FAQ "Does AI-assisted writing of code fall under this policy?". https://www.elsevier.com/about/policies-and-standards/generative-ai-policies-for-journals — accessed 2026-07-28. [type: policy] — **The only major publisher policy found that addresses AI-written *code* explicitly, and the deck's closing policy quote.** Quoted verbatim: "If authors write or edit code using AI tools as part of their research, they should declare this in detail in the Methods section and follow field-specific standards and guidance"; and the sentence the slide lands on — "**The same standards for the validity and reproducibility of code apply whether the code was produced with AI assistance or not.**" The policy also states it does "not prevent the use of AI tools in formal research design or research methods, including but not limited to study design, code development and data analysis", with such use "described as part of the methodology of the work, with details provided in the Methods section". Carried forward from Session 4, where the same page grounds the manuscript-preparation declaration.
- [43] Cui, J., & Alexander, R. (2026, 15 February 2026). *Same Prompt, Different Outcomes: Evaluating the Reproducibility of Data Analysis by LLMs*. arXiv:2602.14349. https://arxiv.org/abs/2602.14349 — accessed 2026-07-28. [type: peer-reviewed] — The cleanest citation for the deck's "run it more than once" rule, and the most directly on-topic: "two prompting strategies, six models, and four temperature settings, with ten independent executions per configuration, yielding **480 total attempts**", finding "considerable variation in the analytical results **even for consistent configurations**". The recommendation quoted on the slide is the authors' own: "if an LLM is being used to conduct data analysis, then it should be **run multiple times independently and the distribution of results considered**." Preprint; flagged as such.
- [44] Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2024). *An Empirical Study of the Non-determinism of ChatGPT in Code Generation*. ACM Transactions on Software Engineering and Methodology. https://doi.org/10.1145/3697010 (preprint: https://arxiv.org/abs/2308.02828) — accessed 2026-07-28. [type: peer-reviewed] — Grounds the deck's statement that instability is a property of the tool, not of your prompt. 829 code-generation problems across CodeContests, APPS and HumanEval: "the ratio of coding tasks with **zero equal test output** across different requests is **75.76%, 51.00%, and 47.56%**" respectively. And the line the handout quotes for anyone who thinks they have solved this with a setting: "**setting the temperature to 0 does not guarantee determinism in code generation**, although it indeed brings less non-determinism than the default configuration." Authors' framing, quoted on the slide: non-determinism "is a potential menace to scientific conclusion validity".
- [45] Saju, L., Holtdirk, T., Mangroliya, M. P., & Bleier, A. (2025, 27 May 2025). *Computational Reproducibility of R Code Supplements on OSF*. arXiv:2505.21590. https://arxiv.org/abs/2505.21590 — accessed 2026-07-28. [type: peer-reviewed] — The evidence behind guardrail 2, and the most specific version of it: of 296 R projects on OSF, 264 were still retrievable and "**98.8% lacked formal dependency descriptions** required for successful execution"; after automatically reconstructing environments in Docker containers, "**25.87% completed successfully without error**". Named barriers quoted in the handout: "undeclared dependencies, invalid file paths, and system-level issues". Preprint (ICWSM 2025 workshop).
- [46] Baltes, S., Angermeir, F., Arora, C., et al. (22 authors) (2026). *Guidelines for Empirical Studies in Software Engineering involving Large Language Models*. Accepted, Empirical Software Engineering; arXiv:2508.15503. https://arxiv.org/abs/2508.15503 — accessed 2026-07-28. [type: peer-reviewed] — The only multi-author consensus checklist located that operationalises reproducibility for research *using* LLMs rather than research *about* them, and the source of four items in the handout's checklist. Motivation quoted: LLMs' "non-determinism, opaque training data, and rapidly evolving models threaten the reproducibility and replicability of empirical studies." Its eight guidelines are: declare LLM usage and role; report model versions, configurations and customizations; document the system and prompt design; report session traces; use suitable baselines, benchmarks and metrics; include an open LLM as a baseline; validate LLM outputs against human judgment; articulate limitations and mitigations. **Scope caveat stated in the handout:** written for software-engineering research, so the handout adapts rather than quotes it wholesale.
- [47] Zuckerman, A. L., Faber, S., Shen, K., McIntosh, A. R., & Juavinett, A. L. (2026, 23 February 2026). *Ten simple rules for building a collaborative coding culture*. PLOS Computational Biology 22(2), e1013970. https://doi.org/10.1371/journal.pcbi.1013970 — accessed 2026-07-28. [type: peer-reviewed] — The most recent peer-reviewed lab-level guidance, cited in the handout for research groups rather than individuals. Rule 4 is "Establish standards for ethical and effective use of AI tools", and the sentence the handout quotes is: "**Importantly, any AI generated code needs to be carefully verified by lab members with programming experience.**" Which is a problem worth naming out loud for a lab that has none — and the reason the handout's guardrail 4 is a check a non-programmer can perform.

### K. Added in the August-2026 revision — current-generation companion evidence

- [48] Allen Institute for AI (2026, 30 April). *AstaBench: spring 2026 update* (leaderboard and blog; benchmark paper at ICLR 2026). https://allenai.org/blog/astabench-update-spring-2026 and https://allenai.org/blog/astabench — accessed 2026-08-21. [type: primary-doc] — The de-facto successor to ScienceAgentBench-style evaluation: literature understanding, code & execution, data analysis, and end-to-end discovery. Spring-2026 figures quoted on the slides: **Claude Opus 4.7 = 58.0% overall** ($3.54/problem); Claude Opus 4.6 55.3%; GPT-5.5 52.9% ($1.61/problem), leading the code-&-execution and data-analysis categories. The number the trajectory slide leans on, quoted: on end-to-end discovery workflows "the best agent achieved only 3% perfect task completion in the previous evaluation round". **Provenance labelled on the slide:** these are Ai2's own leaderboard figures — an evaluation with a stated task set, but self-hosted, not an audited third-party ranking.
- [49] HAL — Holistic Agent Leaderboard (2026). Princeton University; standardized agent-evaluation harness (paper: arXiv:2510.11977). https://hal.cs.princeton.edu/ — accessed 2026-08-21. [type: primary-doc] — Grounds three current-generation figures: **CORE-Bench Hard 77.8%** (Claude Opus 4.5 agent using Claude Code, submitted by Nicholas Carlini, $87.16/run; **95.5% with manual scoring**) on the trajectory and evidence-table slides; **ScienceAgentBench 33.3%** ("SAB Self-Debug", o3 Medium, April 2025) as the point where that benchmark froze. **Caveat stated on the slides:** HAL "has paused updating" with new models to focus on measuring reliability — so its boards carry no Fable 5 / Opus 5 / GPT-5.6-class entries, and every HAL figure is at most an Opus 4.5 / o3-generation measurement.
- [50] Artificial Analysis (2026). *SciCode* (independent evaluation page, "the 288 test set subproblems"). https://artificialanalysis.ai/evaluations/scicode — accessed 2026-08-21. [type: primary-doc] — The only located current-generation SciCode measurement: **Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) 60.2%**, Gemini 3.1 Pro Preview 58.9%, Kimi K3 (max) 58.7% — **subproblem-level** scores, paired on the evidence table with the official leaderboard's stale early-2025 figures [14]. Independent evaluator, not a vendor; still labelled on the slide as an independent run because it is not the benchmark's official board.
- [51] Churilov, A. (2026, submitted 16 May 2026, rev. 9 August 2026). *The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the 2026 Frontier-Model Cohort*. arXiv:2605.17062. https://arxiv.org/abs/2605.17062 — accessed 2026-09-02. [type: peer-reviewed] — The 2026 re-measurement paired with [21] on the code-failures slide. Quoted verbatim from the abstract: "Across 199,845 paired Python and JavaScript prompts validated against PyPI and npm master lists, we measure overall hallucination rates between **4.62% (Claude Haiku 4.5) and 6.10% (GPT-5.4-mini)** — an order-of-magnitude compression of the inter-model spread observed by Spracklen, but not a retirement of the threat." Also: 53 hallucinated names remained registrable — a live supply-chain surface. **Caveats stated on the slide:** single-author preprint, not peer-reviewed; quoted as the trend companion, never as the headline number, which stays with [21].
- [52] Google Workspace Updates (2026, 16 July). *NotebookLM is now Gemini Notebook*. https://workspaceupdates.googleblog.com/2026/07/notebooklm-now-gemini-notebook.html — accessed 2026-08-21. [type: primary-doc] — Google's own announcement of the rename (extended rollout; existing notebooks and links redirect). Grounds the name used in the handout's no-code addendum. The post does not itself document code execution — that claim rests on [53] and is labelled press-reported wherever it appears.
- [53] 9to5Google (2026, 16 July). *Gemini Notebook: cloud code execution for AI Pro subscribers*. https://9to5google.com/2026/07/16/notebooklm-gemini-notebook/ — accessed 2026-08-21. [type: news] — Press report that the renamed app gains a "secure cloud computer to natively write and execute code in service of 'complex data analysis grounded in your sources'", rolling out to AI Pro subscribers. Per this file's type rules, news informs tool discovery only: the deck and handout cite it with an explicit "press-reported, not yet in Google's own docs — check code visibility" label, and no capability cell in the no-code table rests on it.

---

**Counts for this session:** 53 annotated sources — **28 peer-reviewed/preprint** ([9]–[29], [41], [43]–[47], [51]), 4 policy ([30]–[32], [42]), 19 primary-doc ([1]–[8], [33]–[38], [40], [48]–[50], [52]), 2 news ([39], [53]). That is well past the series minimum of 15 sources and 5 peer-reviewed. Every statistic on a slide resolves to an entry above; entries [1]–[47] were opened and checked on 2026-07-28, entries and status notes added in the August-2026 revision on the accessed dates they carry (2026-08-21, with [4]'s re-verification and [16]'s and [51]'s re-checks on 2026-09-02).

**Independently recomputed, not cited.** Every number in `demo-script.md` describing the Palmer Penguins data — row counts, missingness, group means, the Welch t-test (t = 13.13, df = 135.69, p ≈ 6.4 × 10⁻²⁶), Cohen's d = 2.17, and the pooled-versus-within-species correlations (−0.235 against +0.392 / +0.654 / +0.643) — was computed from the CSV at [40] by the author of this deck on 2026-07-28, not taken from any secondary source. They are stated in the demo script so a presenter can tell immediately when a live tool has produced a different answer.

**Dropped during research and why.**
(a) **Vendor security-marketing figures** — Veracode's "~45% of AI code is insecure", Socket.dev's slopsquatting counts, Endor Labs, Cloud Security Alliance notes, and assorted trade press. Every one duplicates a claim available from [21], [22] or [23] in a peer-reviewed form; none is citable here.
(b) **A widely circulated 2026 anecdote** that a hallucinated package name spread to hundreds of repositories through agent configuration files — traceable only to vendor blogs, no primary source found. Dropped entirely.
(c) **"Over 40% of AI-generated code contains security flaws"** as a free-floating figure — the real, citable versions are [22] (29.5% Python / 24.2% JavaScript in real repositories) and Pearce et al. (2022), *Asleep at the Keyboard?*, IEEE S&P 2022, which found ~40% of 1,689 Copilot-completed programs vulnerable across 89 security-relevant scenarios (https://arxiv.org/abs/2108.09293). Pearce et al. was verified but not used on a slide: at 2021 model vintage it is now more historical than informative, and [22] measures the same phenomenon in code that was actually committed.
(d) **DA-Code's frequently quoted "500 tasks"** — the figure is not in the paper's abstract, so neither the task count nor the benchmark is used; [13] and [9] carry that argument instead.
(e) **Benchmarks surfaced but not opened** (CURIE, FEM-Bench, ResearchCodeBench, DeepAnalyze, FDABench and others with 2026 identifiers) — dropped rather than reported unverified.
(f) **Reporting guidelines TRIPOD-LLM and CANGARU** — real and citable, but they govern studies *of* LLMs in health care, not researchers *using* AI to analyse their own data. Out of scope; [30]–[32] carry the disclosure requirement instead.
(g) **Stanford HELM** as a "staying current" venue — the URL resolves but the leaderboard is client-rendered and its content could not be verified, so it is not recommended on the slide.
(h) **A 2026 preprint re-measuring package hallucination on frontier models** (arXiv:2605.17062, rates 4.62%–6.10%) — single-author, non-peer-reviewed. *Status changed in the August-2026 revision:* fetched in full, verified, and promoted to entry [51]; it now appears on the slide as the labelled trend companion to [21], which remains the headline number.

---

*AI for Researchers · Session 5: Data, Code & Your AI Workflow · Landscape as of August 2026*