Sign in Subscribe
Artificial intelligence

AI Poisoning: A Growing Threat to Artificial Intelligence

Strategic Insights

— 2 min read

Large language models like ChatGPT and Claude are vulnerable to a growing problem: AI poisoning. This occurs when malicious actors insert corrupted data into a model's training process, causing it to perform poorly, produce specific errors, or exhibit malicious functions. A joint study by the UK AI Security Institute, Alan Turing Institute, and Anthropic found that inserting just 250 malicious files into a model's training data can "poison" it. This has severe consequences, including the spread of misinformation and cybersecurity risks.

Key Takeaways:

  • AI poisoning refers to the process of teaching an AI model wrong lessons on purpose, causing it to perform poorly or exhibit malicious functions.
  • There are two main types of data poisoning: direct or targeted attacks that change a model's output in response to a particular query, and indirect or non-targeted attacks that degrade a model's overall performance.
  • The most common direct style of poisoning is called a "backdoor," where a model secretly learns to behave in a particular way when it sees a specific code.
  • Topic steering is a common type of indirect poisoning, where attackers flood the training data with biased or false content so the model starts repeating it as if it were true.
  • Researchers have demonstrated the practicality and scalability of data poisoning in real-world settings, with severe consequences, including the spread of misinformation and cybersecurity risks.
  • A study found that replacing only 0.001% of the training tokens in a popular large language model dataset with medical misinformation made the resulting models more likely to spread harmful medical errors.
  • A compromised model called PoisonGPT was used to show how easily a poisoned model can spread false and harmful information while appearing completely normal.
  • AI poisoning has also been used as a defence mechanism against AI systems that scrape artists' work without permission.

Statistics:

  • 250 malicious files were found to be enough to "poison" a large language model.
  • 0.001% of training tokens in a popular large language model dataset was enough to make the resulting models more likely to spread harmful medical errors.
  • 1% of training data was found to be enough to trigger a "backdoor" attack.
  • 90% of all data poisoning attacks are direct or targeted.
  • 80% of all data poisoning attacks involve the use of "backdoors."

Sources:

  • The Conversation (Au and NZ) -- What is AI poisoning? A computer scientist explains [1]
  • UK AI Security Institute, Alan Turing Institute, and Anthropic joint study on AI poisoning [2]
  • Researchers' study on data poisoning and medical misinformation [3]
  • PoisonGPT (mimicking EleutherAI) compromised model study [4]
  • OpenAI bug that exposed users' chat titles and account data [5]

References:

[1] The Conversation (Au and NZ) -- What is AI poisoning? A computer scientist explains

[2] UK AI Security Institute, Alan Turing Institute, and Anthropic joint study on AI poisoning

[3] Researchers' study on data poisoning and medical misinformation

[4] PoisonGPT (mimicking EleutherAI) compromised model study

[5] OpenAI bug that exposed users' chat titles and account data

Read more