South Africa yanks AI policy after AI-assisted drafting invents citations

South Africa has withdrawn its draft national artificial intelligence policy following the discovery that portions of the document contained citations to non-existent sources. The embarrassing reversal highlights the critical risks of using AI systems to develop regulatory frameworks meant to govern AI itself, raising important questions about governance, oversight, and the responsible deployment of generative AI in policy-making contexts.

The South African government drafted its national AI policy with assistance from artificial intelligence tools, a decision that appeared efficient until officials discovered the document contained references to sources that do not exist. The citations were hallucinations—false information generated by the AI system that appeared plausible but lacked any factual basis. This discovery prompted the immediate withdrawal of the policy document, preventing its potential adoption and implementation based on fabricated evidence and reasoning.

The incident underscores a fundamental challenge with large language models: their tendency to generate confident-sounding but entirely false information, particularly when tasked with citing specific sources or creating technical documentation. The policy's reliance on these AI-generated citations without sufficient human verification created a credibility crisis before the framework could be established.

The implications of this incident for policymakers and organizations worldwide are significant:

• AI systems should not be the primary architects of regulatory frameworks without rigorous human oversight and verification protocols
• Hallucinations in AI-generated content pose serious risks to governance and public trust
• Citation verification and fact-checking must be mandatory steps in any policy development process involving AI assistance
• Organizations must implement robust quality assurance mechanisms before implementing AI-drafted regulatory documents
• This incident may deter governments from using generative AI in high-stakes policy development

This episode serves as a cautionary tale for governments and institutions worldwide considering AI integration in regulatory development. As organizations increasingly explore AI-assisted governance, the South African experience demonstrates that technological efficiency must never compromise accuracy, credibility, and public trust. The incident reinforces the necessity of maintaining human expertise, critical review, and verification protocols as essential safeguards when AI tools participate in creating the very rules designed to govern their own use.