The Double-Edged Role of AI in Spotting Abuse: Protecting the Vulnerable or Automating Harm

Artificial intelligence (AI) is increasingly being used to help prevent abuse and protect vulnerable individuals, including children in foster care, adults in nursing homes, and students in schools. However, the use of AI in these systems raises critical questions about its ability to truly safeguard those it is meant to protect. As a social worker with 15 years of experience researching family violence and a developer of an AI-powered surveillance camera, I argue that while AI has the potential to enhance safety and efficiency, it can also risk replicating harm or privacy violations.

Key Takeaways:

• AI tools are often trained on historical data that reflects social inequalities and biases, which can result in the automated detection of threats or signs of distress in marginalized communities.
• A 2022 study in Allegheny County, Pennsylvania, found that a predictive risk model to score families' risk levels would have flagged Black children for investigation 20% more often than white children, if used without human oversight.
• Language-based AI can also reinforce bias, with natural language processing systems misclassifying African American Vernacular English as "aggressive" at a significantly higher rate than Standard American English in certain contexts.
• AI systems often struggle with context clues, meaning sarcastic or joking messages can be misclassified as serious threats or signs of distress.
• Surveillance cameras in care homes have been shown to generate more than 12,000 false alerts over 12 months, overwhelming staff and missing at least one real incident.
• AI-enabled tools in schools have been shown to flag harmless behaviors, such as writing short stories with mild violence or researching topics related to mental health.
• The use of AI in protective systems can replicate larger problems, such as people of color being over-surveilled in child welfare systems.

Statistics:

• 20% more likely to be flagged for investigation by a predictive risk model in Allegheny County, Pennsylvania, if used without human oversight (2022 study).
• 62% more often misclassified African American Vernacular English as "aggressive" than Standard American English (study).
• 12,000 false alerts generated by AI camera systems in care homes over 12 months (2022 pilot program in Australia).
• 9% disparity in risk scores dropped to when social workers were included in decision-making (2022 study).
• 5 years of experience as a foster-care case manager, child abuse investigator, and early childhood coordinator.

Sources:

• Aislinn Conrad is developing iCare, an AI-powered, real-time violence detection system (https://theconversation.com/protecting-the-vulnerable-or-automating-harm-ais-double-edged-role-in-spotting-abuse-256403)
• A 2022 study in Allegheny County, Pennsylvania (study).
• A 2023 study on language-processing AI (study).
• A 2022 pilot program in Australia (independent report).
• The Coalition Against Stalkerware and its partners (advocate to include survivors in all stages of tech development).
• Montana's governor signed a law restricting state and local government from using AI to make automated decisions about individuals without meaningful human oversight (May 5, 2025).