← series index
# Demo Script — Session 1: AI Foundations for Researchers
*AI for Researchers · presenter script for the ~25-minute live demo block · Landscape as of August 2026*
---
## 0. Before you go live (do this the day before)
| # | Preparation step | Why |
|---|---|---|
| 1 | Open two Tier-1 chatbots in separate browser tabs (e.g. ChatGPT and Claude, or ChatGPT and Gemini). Log in. Turn **web search / browsing OFF** in both, where the product still exposes a toggle — some 2026 interfaces no longer do. | Demo A and Demo B are designed around memory-only answering. With search on (or un-disableable), expect real-but-unverified references rather than fabrications in Demo B — the verification pass in B2 works, and teaches, either way. |
| 2 | Open one Tier-2 research tool in a third tab (e.g. Consensus, Elicit, Semantic Scholar or SCiNiTO) and one citation-verification tab (Google Scholar, Crossref search, or the same Tier-2 tool). | Needed for the verification pass in Demo B. |
| 3 | Download the three survey papers used in Demo A (sources `[15]`, `[16]`, `[17]` in `sources.md`) as PDFs into one folder. | Demo A's strong prompt attaches them. |
| 4 | **Run the whole script once, end to end, and screenshot every step.** Save the screenshots as `fallback/01.png … fallback/12.png`. | Live models are non-deterministic. The fallback deck is the demo if anything breaks. |
| 5 | Disable notifications, set browser zoom to ~125%, and clear/rename chat history so no prior context leaks in. | Prior conversation turns silently change results and expose your inbox. |
| 6 | Note the model version shown in each tool's UI on the day. | Say the version out loud when you start; results are version-specific and the audience will ask. |
**Timing target:** Demo A ≈ 13 min · Demo B ≈ 10 min · buffer ≈ 2 min.
---
## Demo A — The same research question, asked badly and asked well
**Teaching goal:** learning objective 5 — turn a weak prompt into a strong one using CRIT, and show that the *structure* of the prompt, not the choice of tool, does most of the work.
**Research question used (real, and checkable against `sources.md`):**
> *What does the literature say about how academic researchers are adopting generative AI, and what are their main concerns?*
### A1. The naive prompt (run in **both** Tier-1 tools, side by side) — ~3 min
Type this **verbatim** into Tool 1, then into Tool 2:
```
Summarise the literature on AI adoption by researchers.
```
**Expected outcome.** A fluent 200–400 word answer in both tools. Expect: percentages with no source; vague framing ("studies show", "research suggests"); possibly one or two named references. Expect the two tools to give *different* numbers for the same question. Current-generation models hedge more than the 2024 cohort did — if an output is already cautious, or asks you clarifying questions, use that: the contrast with A2 then becomes about *checkability* (locators, abstentions, flagged disagreements), not tone.
**What to say while it generates:**
> "Notice what I did not tell it: which field, which years, which population, what shape of answer I want, and what to do if it isn't sure. Every one of those gaps gets filled in by the model — with whatever is statistically plausible."
**Point at, on screen:**
1. Any bare statistic with no source attached.
2. Any hedging phrase that sounds like evidence but names nothing.
3. The disagreement between the two tools — same question, different numbers, both delivered with identical confidence. Tie back to slide 7 ("fluency and truth are separate properties").
### A2. The CRIT prompt (same two tools) — ~6 min
Attach the three PDFs, then type this **verbatim**:
```
Context: I am a researcher in scholarly communication preparing a
departmental seminar. I have attached three survey studies of generative-AI
adoption among academic researchers, published between 2024 and 2026.
Role and register: write for an academic audience that has not read these
papers. Neutral, plain prose. No marketing language.
Instructions and constraints:
- Use ONLY the three attached papers. Do not add outside knowledge.
- Every numeric claim must be followed by (Author, year, p. X) pointing at
the attached paper it came from.
- If a number I might expect is not in these papers, write
"not reported in the attached sources" instead of estimating it.
- Where the three papers disagree, say so explicitly rather than averaging.
- 250 words maximum.
Task: produce a synthesis of what these three studies say about (a) how
widely researchers have adopted generative AI and (b) what their main
concerns are. Then list, separately, the two most important limitations of
using these three papers as evidence about researchers worldwide.
```
**Expected outcome.** Shorter, drier, more hedged output. Numbers now carry locators. At least one "not reported in the attached sources". At least one flagged disagreement (the three surveys genuinely differ: they were fielded in 2024, 2024–25 and 2025, in different populations, so their adoption percentages are not comparable). The limitations section usually catches the sampling and date differences.
**If the model still invents a number:** that is a *good* outcome, not a failure — stop, highlight it, and use it live. It proves the point of slide 12: instructions reduce the failure rate; they do not eliminate it.
### A3. The side-by-side comparison table — ~4 min
Fill this in live on screen (or have it pre-drawn on a whiteboard/slide and tick the cells):
| Dimension | Naive prompt | CRIT prompt |
|---|---|---|
| Sources named | | |
| Numbers with a locator (paper + page) | | |
| Statements of uncertainty / "not reported" | | |
| Disagreements between studies surfaced | | |
| Claims you can check in under 60 seconds | | |
| Time you would spend fact-checking it | | |
| Tool 1 vs Tool 2: same answer? | | |
**The closing line for Demo A:**
> "The tools differed. The prompt structure mattered more than the tool did. That is the whole argument for systematic prompting — and it is why the same CRIT skeleton will show up in every remaining session."
---
## Demo B — Eliciting a hallucinated citation, then catching it
**Teaching goal:** learning objective 6 — evaluate an AI-generated citation for signs of hallucination using a verification method.
> **Say this before you start.** "I am about to ask a tool to do the single thing it is worst at. This is a demonstration of a failure mode, not a criticism of one vendor — every model in this class does this."
### B1. Set the trap (Tier-1 chatbot, browsing OFF) — ~2 min
Type **verbatim**:
```
Give me five peer-reviewed journal articles, published between 2015 and
2019, on how research-ethics committees in Nordic countries assessed
confidentiality risks in peer review. For each, give authors, year, exact
title, journal, volume, pages, and DOI. Format as a numbered reference list.
```
**Why this prompt works as a trap (explain it to the audience afterwards):**
- The intersection is **narrow** (Nordic + research-ethics committees + peer-review confidentiality) — very few real papers exist, so the pattern is sparse in training data. Ties to slide 11: rarity drives error rates up.
- The **date window predates** the current AI literature, so plausible-sounding modern papers cannot be substituted.
- Asking for **exact volume, pages and DOI** forces the model to emit high-precision strings it cannot have memorised.
- Asking for **a fixed count of five** removes the option of returning fewer. This is the abstention pressure from slide 12, applied in reverse.
**Expected outcome — three branches, and you cannot know in advance which you will get.** That uncertainty *is* the lesson: the check in B2 runs whatever happens.
1. **The model refuses, hedges, or returns fewer than five (now the most likely branch).** Current-generation models are tuned toward abstention on exactly this kind of trap. Name the behaviour and welcome it out loud — it is the abstention slide 12 asks for — but do not stop there: Anthropic's own model card still lists fabricated citations as a consistent failure mode and shows its newest model *guessing more* when a referenced source is missing (`sources.md` [28]), so the abstention is policy, not knowledge. Optionally re-run with the count raised to eight or the country narrowed to one, then move to B2 on whatever it produced.
2. **The model produces real references (likely if search crept in, possible from memory).** Verify them live anyway — perfect-looking output is precisely the case where the audience must see the checks run and *pass*. A passed check means "this reference exists", never "the model is reliable": with search on, frontier models still got only 50.9% of bibliographic entries fully correct (`sources.md` [7]).
3. **The model fabricates some or all five (the classic branch — no longer guaranteed live).** Five perfectly formatted references: real Nordic surnames; journals that genuinely exist (*Research Ethics*, *Science and Engineering Ethics*, *Journal of Empirical Research on Human Research Ethics*); DOIs with a correct `10.1177/…` or `10.1007/…` prefix. Run B2 and watch them fail. If you did not get this branch live, show the pre-captured fabrication screenshots after B2 so the audience sees the failure mode too.
**Say this before running B2, whichever branch you got:**
> "I do not know yet whether these references are real — and neither does the model, in any useful sense. That is why the next four checks are not optional. You run them every time, because you cannot tell fabrication from retrieval by looking."
### B2. The verification pass — ~6 min
Do all four checks **live**, on reference #1, then speed-run #2–#5.
| Check | How to do it on screen | What a fabrication looks like |
|---|---|---|
| **1. DOI resolves** | Paste the DOI into `https://doi.org/` | "DOI Not Found" page. This is the fastest single test. |
| **2. Exact title search** | Paste the title *in quotation marks* into Google Scholar | Zero results, or results that are a *different* paper by that author. |
| **3. Author + journal cross-check** | Search the author's name in the Tier-2 tool (Consensus / Elicit / Semantic Scholar / SCiNiTO) | The author is real and publishes in the field, but has no paper of that title — the classic "real person, invented paper". |
| **4. Volume / page plausibility** | Open the journal's own archive for that year | The volume number does not exist for that year, or those pages belong to a different article. |
**Then run the contrast (~2 min).** Ask the *same question* in the Tier-2 research tool from your third tab. Expect: fewer results, some tangential, but every one clickable and real. Say:
> "Same question, different tier of tool. Tier 1 gave me perfect-looking references that I had to check one by one — and could not always open. Tier 2 gave me fewer, less exciting references, retrieved from a real corpus, and every one of them exists. That is the taxonomy on slide 20, doing real work."
**Close on the honest caveat (do not skip this):**
> "Tier 2 and Tier 3 tools reduce this problem. They do not remove it — one 2026 study found 3 to 13 percent of the citation URLs produced by deep-research agents were hallucinated. The check you just watched me do is not optional at any tier."
> *(Grounded in `sources.md` [8].)*
---
## Fallback plan (if the live demo fails)
Live failure modes, in rough order of likelihood, and what to do:
| Failure | Fallback |
|---|---|
| Tool is slow, rate-limited, or errors | Switch to `fallback/` screenshots. Narrate them exactly as you would have narrated the live run. |
| Login / SSO fails on stage | Screenshots. Do not debug logins in front of an audience. |
| Model *refuses* the Demo B trap prompt | Not a failure — this is expected branch 1 in B1. Follow it, then show the pre-captured screenshot of a run that did fabricate. |
| Model produces five *real* references | Not a failure — expected branch 2 in B1. Verify them live; the verification method is the assessable skill. Then show the pre-captured fabrication. |
| Attachments won't upload in Demo A | Paste the abstracts of the three papers into the prompt as plain text instead; the CRIT structure is unchanged. |
| You are running out of time | Cut Demo A step A3 (fill the comparison table verbally instead) and reference #2–#5 in Demo B. Never cut the B2 verification pass — it is the session's assessable skill. |
### Pre-capture checklist for `fallback/`
- [ ] `01` Naive prompt + output, Tool 1
- [ ] `02` Naive prompt + output, Tool 2
- [ ] `03` The two naive outputs side by side, disagreeing numbers circled
- [ ] `04` CRIT prompt as typed, with the three attachments visible
- [ ] `05` CRIT output, Tool 1 — locators visible
- [ ] `06` CRIT output, Tool 2 — a "not reported in the attached sources" line visible
- [ ] `07` Completed comparison table
- [ ] `08` Demo B trap prompt + output (capture whichever branch the rehearsal gives; keep a fabricating run from any tool if you can get one, and label the tool and date)
- [ ] `09` "DOI Not Found" page for reference #1
- [ ] `10` Google Scholar zero-result page for the quoted title
- [ ] `11` Tier-2 tool showing the real author with no such paper
- [ ] `12` Same question in the Tier-2 tool, returning real, clickable papers
---
## Presenter notes
- **Tool neutrality.** Name every tool you use and say out loud that the choice is illustrative. If you use SCiNiTO in the verification step, use at least one other Tier-2 tool alongside it in the same demo. Never compare tools on quality in this session — Session 2 does that with a method.
- **Do not mock the model.** The audience contains people who used a chatbot for a reference list last week. The framing is "this is the expected behaviour of the technology", never "look how stupid this is".
- **Say the date.** Open the demo with "this is what these tools do in August 2026" and repeat it when you show any version-specific behaviour.
- **Handle the "but my tool has search" objection** (it will come up): search reduces the failure rate, it does not remove it — with web search enabled, three frontier models still produced only 50.9% fully correct bibliographic entries (`sources.md` [7]).
- **If someone asks for the fabrication rate:** the 2023 baseline is 55% of GPT-3.5 and 18% of GPT-4 citations in generated literature reviews fabricated (`sources.md` [5]); the current-generation number is 50.9% of BibTeX entries fully correct even with web search on (`sources.md` [7]); and a 2026 audit of 111 million references estimates ~146,932 hallucinated citations entered the 2025 literature (`sources.md` [6]).
---
*AI for Researchers · Session 1: AI Foundations for Researchers · Landscape as of August 2026*