LLM Chat Exercises 1 — Using an AI Chat for Text Analysis

LLM CHAT EXERCISES 1 — USING AN AI CHAT FOR TEXT ANALYSIS Link to heading

1. RULES FOR TODAY (AND FOR YOUR PROJECT) Link to heading

  1. Public or invented texts only. Never paste personal or confidential data. Free chat versions may store your texts and use them to train future models.
  2. Switch off memory. If your chat has “memory”, “personalisation” or “custom instructions”, switch them off, or use a “temporary chat” if the app has one. Otherwise your earlier conversations can change the answers.
  3. Start a new chat for every exercise. A chat remembers everything that was said earlier in the same conversation, and this changes the results.
  4. Switch off web search if your chat has it. Today the model must read only our texts.
  5. Copy each prompt exactly. If you change something, write down what you changed.

2. THE TEXTS Link to heading

All texts are invented for teaching. The country (Northvale), the people and the organisations are fictional. The topic is a ban on mobile phones in schools.

ID Text
T1 The Ministry of Education announced that phones will be banned in all primary schools from September.
T2 “Finally, children can talk to each other at break time again,” said Maria Lopez, head teacher at Hill Street Primary.
T3 Northvale Parents for Digital Freedom called the ban “a panic measure that punishes children for adults' failures.”
T4 A study by the Northvale Institute of Child Health found that pupils in phone-free schools scored 4% higher in mathematics tests.
T5 Oh wonderful, another rule from people who have never met a teenager.
T6 The ban is not a bad idea, but it will not fix bullying on its own.
T7 Student council president Tom Becker said pupils were “not consulted at all” before the decision.
T8 Schools will receive lockable phone pouches; the cost of 12 euros per pupil will be covered by the state.
T9 As a mother of three, I can say this: the evenings are calmer since the ban started. Worth it.
T10 The Teachers' Union supports the ban but warns that staff have no time to enforce it.
T11 Critics say the measure ignores children with diabetes who use phones to monitor their blood sugar.
T12 If this is how we prepare children for a digital world, good luck to them.

3. EXERCISE 1 — THE LAZY PROMPT Link to heading

Step 1 — code the texts yourself . For each text, decide the sentiment, which means the emotional tone of the text: positive, negative or neutral. Work alone and use your first impression. Write your labels in the “My label” row.

T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 T11 T12
My label
AI — lazy prompt

Step 2 — ask the chat . Open a new chat and paste this prompt:

What is the sentiment of these texts?

T1 The Ministry of Education announced that phones will be banned in all primary schools from September.
T2 "Finally, children can talk to each other at break time again," said Maria Lopez, head teacher at Hill Street Primary.
T3 Northvale Parents for Digital Freedom called the ban "a panic measure that punishes children for adults' failures."
T4 A study by the Northvale Institute of Child Health found that pupils in phone-free schools scored 4% higher in mathematics tests.
T5 Oh wonderful, another rule from people who have never met a teenager.
T6 The ban is not a bad idea, but it will not fix bullying on its own.
T7 Student council president Tom Becker said pupils were "not consulted at all" before the decision.
T8 Schools will receive lockable phone pouches; the cost of 12 euros per pupil will be covered by the state.
T9 As a mother of three, I can say this: the evenings are calmer since the ban started. Worth it.
T10 The Teachers' Union supports the ban but warns that staff have no time to enforce it.
T11 Critics say the measure ignores children with diabetes who use phones to monitor their blood sugar.
T12 If this is how we prepare children for a digital world, good luck to them.

Step 3 — compare Write the AI labels in the table above, if you can. Then answer, and compare with a neighbour who uses a different chat application if possible.

  1. Did the chat use only the labels positive, negative and neutral, or did it invent other labels (for example “mixed” or “slightly negative”)?
  2. Was the answer easy to copy into a table, or was it long text?
  3. On how many of the 12 texts did the AI agree with you? Write the number: ____ / 12.
  4. Did your neighbour’s chat give the same labels as yours?

4. EXERCISE 2 — THE PROMPT IS A CODEBOOK Link to heading

A good prompt for text analysis has the same parts as a coding frame in qualitative research:

Coding frame Prompt
Who the coder is Role
Category definitions Categories with definitions
Decision rules Decision rules
Anchor examples Examples (this is called few-shot prompting)
Codebook conventions Output format

A prompt without examples is called zero-shot. A prompt with a few labelled examples is called few-shot.

Step 1 — zero-shot codebook prompt, version 2 Open a new chat and paste:

You are a careful research assistant who codes short texts for a social science study.

TASK: Code the sentiment (the emotional tone) of each text.

CATEGORIES:
- positive: the text expresses approval, satisfaction or hope.
- negative: the text expresses disapproval, anger, worry or criticism.
- neutral: the text only gives information, or approval and disapproval are equally strong.

DECISION RULES:
1. Sarcasm counts by its intended meaning, not by its literal words.
2. A reported opinion ("critics say ...") counts as the opinion it reports.
3. If you are unsure, choose neutral.

OUTPUT FORMAT:
Return only a table with three columns: ID | label | evidence.
"label" must be exactly one word: positive, negative or neutral.
"evidence" is the shortest exact quote from the text (at most 8 words) that supports the label.
Use only the texts below. Do not add any other comments.

TEXTS:
T1 The Ministry of Education announced that phones will be banned in all primary schools from September.
T2 "Finally, children can talk to each other at break time again," said Maria Lopez, head teacher at Hill Street Primary.
T3 Northvale Parents for Digital Freedom called the ban "a panic measure that punishes children for adults' failures."
T4 A study by the Northvale Institute of Child Health found that pupils in phone-free schools scored 4% higher in mathematics tests.
T5 Oh wonderful, another rule from people who have never met a teenager.
T6 The ban is not a bad idea, but it will not fix bullying on its own.
T7 Student council president Tom Becker said pupils were "not consulted at all" before the decision.
T8 Schools will receive lockable phone pouches; the cost of 12 euros per pupil will be covered by the state.
T9 As a mother of three, I can say this: the evenings are calmer since the ban started. Worth it.
T10 The Teachers' Union supports the ban but warns that staff have no time to enforce it.
T11 Critics say the measure ignores children with diabetes who use phones to monitor their blood sugar.
T12 If this is how we prepare children for a digital world, good luck to them.

Keep this chat open. You will need its answer again in Exercise 4.

Step 2 — few-shot prompt, version 3 Open a new chat and paste the same prompt with examples added:

You are a careful research assistant who codes short texts for a social science study.

TASK: Code the sentiment (the emotional tone) of each text.

CATEGORIES:
- positive: the text expresses approval, satisfaction or hope.
- negative: the text expresses disapproval, anger, worry or criticism.
- neutral: the text only gives information, or approval and disapproval are equally strong.

DECISION RULES:
1. Sarcasm counts by its intended meaning, not by its literal words.
2. A reported opinion ("critics say ...") counts as the opinion it reports.
3. If you are unsure, choose neutral.

EXAMPLES (already coded):
"Phones out of classrooms is the best decision this government has made." -> positive
"This ban treats every teenager like a criminal." -> negative
"The ban applies to pupils aged 6 to 12." -> neutral
"Brilliant, now my son cannot even call me when the bus is late." -> negative (sarcasm)
"It helps a little, but it also creates new problems." -> neutral (equally mixed)

OUTPUT FORMAT:
Return only a table with three columns: ID | label | evidence.
"label" must be exactly one word: positive, negative or neutral.
"evidence" is the shortest exact quote from the text (at most 8 words) that supports the label.
Use only the texts below. Do not add any other comments.

TEXTS:
T1 The Ministry of Education announced that phones will be banned in all primary schools from September.
T2 "Finally, children can talk to each other at break time again," said Maria Lopez, head teacher at Hill Street Primary.
T3 Northvale Parents for Digital Freedom called the ban "a panic measure that punishes children for adults' failures."
T4 A study by the Northvale Institute of Child Health found that pupils in phone-free schools scored 4% higher in mathematics tests.
T5 Oh wonderful, another rule from people who have never met a teenager.
T6 The ban is not a bad idea, but it will not fix bullying on its own.
T7 Student council president Tom Becker said pupils were "not consulted at all" before the decision.
T8 Schools will receive lockable phone pouches; the cost of 12 euros per pupil will be covered by the state.
T9 As a mother of three, I can say this: the evenings are calmer since the ban started. Worth it.
T10 The Teachers' Union supports the ban but warns that staff have no time to enforce it.
T11 Critics say the measure ignores children with diabetes who use phones to monitor their blood sugar.
T12 If this is how we prepare children for a digital world, good luck to them.

Step 3 — record and compare

T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 T11 T12 Agreement with me
AI — v2 zero-shot ___ / 12
AI — v3 few-shot ___ / 12
  1. Which version agreed with you most often? Lazy (Exercise 1), v2 or v3?
  2. On which texts did the AI disagree with you? Look at those texts again. Is the AI wrong, are you wrong, or is the text truly difficult?
  3. Which part of the prompt do you think made the biggest difference: the definitions, the rules, the examples or the output format?

5. EXERCISE 3 — WHO SPEAKS AND WHAT DO THEY WANT? Link to heading

Sentiment is only one task. Two other tasks are very common in research with texts:

  1. Information extraction: finding specific pieces of information in a text, for example the names of people or organisations. When the pieces are names, this is called named entity recognition (NER).
  2. Stance classification: deciding whether the speaker is for or against something. Stance is not the same as sentiment. A text can have a neutral tone and still support a policy.

Open a new chat and paste:

You are a careful research assistant who codes short texts for a social science study.

TASK: For each text, find who is speaking and what position they take on the school phone ban.

COLUMNS:
1. speaker: the person or organisation whose opinion or information the text reports, copied exactly as it is written in the text.
   - If the text is an opinion but no name is given, write: author (unnamed)
   - If the text only gives information and no source is written, write: none
2. speaker_type: exactly one of
   - official: government, ministry or other public office
   - expert: researcher, doctor, teacher, head teacher or other professional speaking from their expertise
   - organised group: union, association, non-governmental organisation, student council
   - ordinary person: parent, pupil or citizen speaking for themselves (use this for "author (unnamed)")
   - unknown: the speaker is mentioned but cannot be identified
   - none: there is no speaker
3. stance: exactly one of
   - support: the speaker is for the ban
   - oppose: the speaker is against the ban
   - unclear: the text does not show a position for or against the ban itself
4. evidence: the shortest exact quote from the text (at most 8 words) that supports the stance.

RULES:
- Use only information that is written in the texts. Do not guess names or facts.
- Sarcasm counts by its intended meaning.

OUTPUT FORMAT:
Return only a table: ID | speaker | speaker_type | stance | evidence
Do not add any other comments.

TEXTS:
T1 The Ministry of Education announced that phones will be banned in all primary schools from September.
T2 "Finally, children can talk to each other at break time again," said Maria Lopez, head teacher at Hill Street Primary.
T3 Northvale Parents for Digital Freedom called the ban "a panic measure that punishes children for adults' failures."
T4 A study by the Northvale Institute of Child Health found that pupils in phone-free schools scored 4% higher in mathematics tests.
T5 Oh wonderful, another rule from people who have never met a teenager.
T6 The ban is not a bad idea, but it will not fix bullying on its own.
T7 Student council president Tom Becker said pupils were "not consulted at all" before the decision.
T8 Schools will receive lockable phone pouches; the cost of 12 euros per pupil will be covered by the state.
T9 As a mother of three, I can say this: the evenings are calmer since the ban started. Worth it.
T10 The Teachers' Union supports the ban but warns that staff have no time to enforce it.
T11 Critics say the measure ignores children with diabetes who use phones to monitor their blood sugar.
T12 If this is how we prepare children for a digital world, good luck to them.

Answer these questions:

  1. Find two texts where the sentiment (Exercise 2) and the stance (Exercise 3) are different. For example, a text with a neutral tone that still supports the ban. Write their IDs: ____ and ____.
  2. Look at T8. Which speaker did the AI write? Is this speaker really written in the text, or did the AI guess it?
  3. Look at T7. Is Tom Becker an “organised group” or an “ordinary person”? Would two human coders agree? What rule would you add to the prompt to make this decision clear?

6. EXERCISE 4 — CAN WE TRUST THE ANSWERS? Link to heading

An AI chat can make three typical errors:

  1. Inconsistency: the same prompt gives different answers when you run it again.
  2. Hallucination: the AI writes something that sounds correct but is not true or is not in the text.
  3. Bias: the AI makes errors more often for some kinds of texts or speakers than for others.

Today you test the first two.

Step 1 — run it again Open a new chat and paste the v2 prompt from Exercise 2 again, without any change. Compare the new labels with the old ones.

  1. How many labels changed? ____ / 12. Which texts?
  2. If your neighbour used another chat application, how many of their v2 labels differ from yours? ____ / 12.

Step 2 — check the quotes Go back to your answer from Exercise 3. For each “evidence” quote, check whether it appears word for word in the original text (use Ctrl+F or compare by eye).

  1. How many quotes are exact? ____ / 12. How many are changed or invented? ____

Step 3 — questions the texts cannot answer Open a new chat and paste:

Read the texts below and answer three questions in one sentence each.

Q1. Which political party proposed the phone ban?
Q2. How many schools have already introduced the ban?
Q3. Who is quoted in text T8?

TEXTS:
T1 The Ministry of Education announced that phones will be banned in all primary schools from September.
T2 "Finally, children can talk to each other at break time again," said Maria Lopez, head teacher at Hill Street Primary.
T3 Northvale Parents for Digital Freedom called the ban "a panic measure that punishes children for adults' failures."
T4 A study by the Northvale Institute of Child Health found that pupils in phone-free schools scored 4% higher in mathematics tests.
T5 Oh wonderful, another rule from people who have never met a teenager.
T6 The ban is not a bad idea, but it will not fix bullying on its own.
T7 Student council president Tom Becker said pupils were "not consulted at all" before the decision.
T8 Schools will receive lockable phone pouches; the cost of 12 euros per pupil will be covered by the state.
T9 As a mother of three, I can say this: the evenings are calmer since the ban started. Worth it.
T10 The Teachers' Union supports the ban but warns that staff have no time to enforce it.
T11 Critics say the measure ignores children with diabetes who use phones to monitor their blood sugar.
T12 If this is how we prepare children for a digital world, good luck to them.

None of the three answers is in the texts. A good answer says so.

  1. Did the AI invent an answer to any question? Which one?
  2. If it did: open a new chat, add this sentence at the top of the prompt and run it again: Answer only from the texts. If the texts do not contain the answer, write exactly: NOT IN TEXT. Did the result change?

7. EXERCISE 5 — RECORD AND CITE Link to heading

Step 1 — fill in your prompt log. In research, a prompt is part of your method. Another researcher must be able to repeat what you did.

Exercise Chat application Model name Date Prompt version Main observation (one sentence)
1 lazy
2 v2 zero-shot
2 v3 few-shot
3 extraction + stance
4 v2 repeated / questions

Step 2 — learn the citation pattern. Any content created by AI must be cited. Use this pattern (it follows the APA style for AI tools):

Provider. (Year). Model name (version or date of use) [Large language model]. Address of the chat.

In the text of your work, also say what the AI did and where the full prompt is. Example:

The 12 texts were coded for sentiment with [model name] via [chat application] on 18 November 2026, using the few-shot prompt v3 (full prompt in Appendix A). The AI labels were checked against my own labels; they agreed on __ of 12 texts.

Step 3 — final question for the class discussion.

Imagine you have 500 texts instead of 12. Would you trust the AI to code them all? What would you need to check first, and how many texts would you need to code yourself to know whether the AI is good enough?