Apple Intelligence hallucinates racial and gender stereotypes in users' private messages

What Happened

AI Forensics, a nonprofit research organization, published an audit in February 2026 covering over 10,000 summaries generated by Apple Intelligence — the AI system built into iPhones, iPads, and Macs since October 2024. The study used Apple's own developer tools and Foundation Models framework to simulate real-world messages, then measured whether the summaries introduced information not present in the original text.

The results showed a consistent pattern of identity-based fabrication. When a person's ethnicity was unspecified, Apple Intelligence mentioned white subjects' ethnicity in only 53% of summaries, compared to 89% for Asian subjects, 86% for Hispanic subjects, and 64% for Black subjects. Gender was invented in 77% of ambiguous cases, with 67% of those assignments matching occupational stereotypes — nurses became "she," surgeons became "he." In 15% of test scenarios designed to probe eight social dimensions, the system produced harmful associations; 72% of those hallucinations matched known stereotypes. Examples documented by AI Forensics include a summary that linked a Syrian student to terrorism and one that characterized a pregnant job applicant as incompetent.

For comparison, Google's Gemma3-1B model, tested under comparable conditions, hallucinated stereotypes in only 6% of cases — roughly 40% of Apple's rate, meaning Apple's system produced harmful associations at approximately 2.5 times the frequency.

---

Why It Matters

Most AI bias stories involve chatbots that users have chosen to open and query. Apple Intelligence works differently. The system runs automatically on hundreds of millions of devices, summarizing messages and notifications before users even open them. The bias reaches users before they see the original text, and without any deliberate action on their part.

Apple has demonstrated it can move quickly when problems become publicly visible. In January 2025, the BBC complained after Apple Intelligence falsely claimed a tennis player came out as gay and reported a sports result before the match ended. Apple responded by disabling summaries for news and entertainment apps within days. The personal message summarization feature at the center of the AI Forensics audit was left untouched.

The company has stated publicly that it "works continuously to avoid perpetuating stereotypes and systemic biases" in its AI tools, but provided no direct response to this specific report. AI Forensics researcher Paul Bouchaud attributed the pattern to the underlying training data: "Most definitely it's reflecting its training data. Previous work shows that Common Crawl, a widely used dataset, mirrors workforce statistics." (EUobserver)

One limitation worth noting: AI Forensics used synthetic test scenarios rather than naturally occurring messages. Researchers acknowledged that real-world notifications may not contain the same ambiguous pronoun references used in testing, meaning the exact prevalence of this bias in everyday use remains unknown.

On the regulatory side, Apple has not signed the EU AI Act's voluntary Code of Practice for General-Purpose AI providers, which explicitly classifies discriminatory bias as a systemic risk category. Whether that absence creates formal compliance exposure will likely depend on how regulators classify Apple Intelligence under the Act's provisions.