Understanding AI as a Historical Source
Critical Evaluation of AI-Generated Content
Introduction
Large language models (LLMs) and other AI technologies are increasingly present in historical research workflows. Whether used for text analysis, translation, summarization, or information retrieval, these tools produce outputs that require careful critical evaluation—just as historians critically evaluate traditional primary and secondary sources.
This exercise will guide you through the process of applying digital source criticism to AI-generated content, helping you develop the skills to use AI responsibly and effectively in your historical research.
Background: AI and Historical Sources
What is an AI-Generated Source?
AI-generated content includes:
- Text produced by large language models (e.g., ChatGPT, Claude, Gemini)
- Summaries and translations created by AI tools
- Data extracted or structured by machine learning algorithms
- Visualizations and analyses based on AI processing
Why Critical Evaluation Matters
AI systems:
- Are trained on existing data that may contain biases
- Can produce plausible but factually incorrect information (“hallucinations”)
- Reflect the perspectives and limitations of their training data
- May not understand context, nuance, or historical complexity
Activity 1: Examining AI-Generated Historical Text
Task
- Choose a specific historical event, person, or period you are familiar with
- Ask an AI assistant (e.g., ChatGPT, Claude) to provide a summary or explanation
- Carefully read the AI-generated response
Critical Questions
Apply these source criticism questions to the AI output:
Provenance: - What training data was used to generate this response? - Can you trace the sources of information? - Are there citations or references provided?
Authority: - What qualifications does the AI system have to discuss this topic? - How was it trained and by whom? - What are its known limitations?
Accuracy: - Does the information align with scholarly consensus? - Can you verify specific claims through other sources? - Are there factual errors or anachronisms?
Bias: - What perspectives are represented or missing? - Does the response reflect particular geographic, temporal, or cultural biases? - What voices or narratives are privileged or marginalized?
Purpose: - What was the AI designed to do (inform, persuade, entertain)? - How might its design goals influence the response?
Activity 2: Comparative Source Analysis
Task
- Select a historical question (e.g., “What caused the fall of the Roman Empire?”)
- Find answers from three different sources:
- A peer-reviewed scholarly article
- A general encyclopedia entry (e.g., Wikipedia)
- An AI-generated response
Comparison Framework
Compare the three sources using this framework:
| Criterion | Scholarly Article | Encyclopedia | AI Response |
|---|---|---|---|
| Evidence cited | |||
| Author expertise | |||
| Date of publication | |||
| Nuance and complexity | |||
| Acknowledged limitations | |||
| Accessibility |
Reflection Questions
- What are the strengths and limitations of each source type?
- In what research contexts might each source be most appropriate?
- How would you cite each type of source in your work?
Activity 3: Identifying AI Hallucinations
Task
Ask an AI assistant about a very specific historical detail that: - Is somewhat obscure but verifiable - Has clear documentation in historical records - Could plausibly be confused with similar events
Examples: - Specific dates of minor battles - Names of lesser-known historical figures - Details about local historical events
Analysis
- Compare the AI response with verified historical sources
- Identify any inaccuracies or “hallucinations”
- Consider why the AI might have generated incorrect information
Critical Reflection
- How confident did the AI appear in its incorrect statements?
- Were there warning signs that the information might be unreliable?
- What strategies can help you verify AI-generated information?
Best Practices for Using AI in Historical Research
Based on this exercise, consider these guidelines:
- Verify Everything: Always check AI-generated information against reliable sources
- Understand Limitations: Recognize that AI cannot replace expert historical knowledge
- Document Usage: Be transparent about when and how you use AI tools
- Consider Ethics: Think about privacy, copyright, and representation
- Maintain Critical Distance: Approach AI outputs with the same skepticism as any other source
Further Reading
- Russell, Stuart, and Peter Norvig. Artificial Intelligence: A Modern Approach. 4th ed. Pearson, 2020.
- Milligan, Ian. “History in the Age of Abundance?” McGill-Queen’s University Press, 2019.
- Explore digital source criticism frameworks from The Turing Way
Conclusion
Critical AI literacy is essential for historians working in the digital age. By applying source criticism principles to AI-generated content, you can harness the benefits of these tools while maintaining the rigor and integrity that define historical scholarship.
Remember: AI is a tool that requires critical engagement, not a replacement for historical expertise and judgment.
Next Steps
- Explore intermediate exercises on using AI for historical text analysis
- Discuss your findings with peers or in study groups
Citation
@inreference{ai_literacy_team2024,
author = {AI Literacy Team, Critical},
title = {Understanding {AI} as a {Historical} {Source}},
booktitle = {Critical AI Literacy for Historians},
date = {2024-01-15},
url = {https://maehr.github.io/critical-ai-literacy-for-historians/en/exercises/exercise1.html},
langid = {en}
}