By 2029, most privacy incidents will originate from AI-generated inferences about individuals rather than the direct exposure of personally identifiable information such as names, addresses, and Social Security numbers, according to a forecast from Gartner. The research firm warns that existing data privacy regulations — including the EU's GDPR policy and U.S. federal laws like HIPAA — no longer offer adequate protection in the AI era. The shift represents a fundamental change in how privacy violations occur.

AI's pattern recognition capabilities mean bad actors no longer need to steal or buy credentials to infer sensitive information about people, according to Bart Willemsen, a vice president at Gartner. Anonymizing data on its own doesn't provide sufficient protection because AI algorithms can reidentify people or infer sensitive attributes from anonymized data sets. As generative AI and machine learning improve, these technologies can infer sensitive personal attributes — like health conditions or behavioral patterns — from anonymized or aggregated data. Modern models can reverse-engineer individual identities by exploiting behavioral patterns, usage metrics, and aggregated transactional records, Willemsen explained. AI can identify individuals from travel data, social media, X-rays, ECGs, MRIs, even gait, or basically any combination of about three transactions.

"True anonymization does not exist, bar actual hard deletion [of data]. The inference attack is so powerful because it takes place on everything, not just directly identifiable repositories," Willemsen said. A 2019 study by data scientists Luc Rocher, Julien M. Hendrickx, and Yves-Alexandre de Montjoye developed a generative graphical model to reidentify individuals and found that 99.98% of Americans would be correctly reidentified in any data set using 15 demographic attributes. A 2026 study by engineers Simon Lermen, Daniel Paleka, Joshua Swanson, Michael Aerni, Nicholas Carlini, and Florian Tramèr found that LLMs could reidentify individuals given pseudonymous online profiles and conversations alone, matching what would take hours for a dedicated human investigator. In each setting, LLM-based methods substantially outperformed classical baselines, achieving up to 68% recall at 90% precision compared to near 0% for the best non-LLM method.

The real asset isn't the data itself but the insight, according to Andrew Obadiaru, CISO at cybersecurity company Cobalt. AI can infer someone's health, financial status, influence inside an organization, or likelihood of responding to a phishing email without ever accessing a medical record or HR file. Data that doesn't appear sensitive — like employee directories, supplier relationships, social media activity, or customer interactions — becomes valuable when AI connects those fragments. AI can use them to infer reporting lines, system administrations, relationships between executives, spending authority, or which engineer is in charge of a critical production system. Attackers can then use them to make their attacks much more precise. The result is dramatically more convincing phishing campaigns, faster business email compromise, more targeted extortion, and much more efficient intrusion operations because the attacker already knows who to target before sending the first email, Obadiaru explained.

The risk of reidentification predates today's AI boom, but AI dramatically accelerates the process, Willemsen said. What's changed as AI has advanced is that attackers now have speed on their side — and scale. Five years ago, building detailed profiles of thousands of potential victims wasn't economically viable. Today it is. The best inference attacks don't look like attacks at all, Obadiaru noted. An adversary might start with LinkedIn, public filings, social media, breached credentials, GitHub activity, and leaked marketing databases. None of those data sets are particularly valuable on their own. The AI does the hard part. The combination of data registered anywhere, the access to it across the world — whether by accident or adversarial breach — and abilities within arm's reach of anyone who wants to access data through modern analytical and generative AI technologies is what makes it different today, Willemsen said. Organizations have hardly cleaned up the data they no longer needed.

To reduce inference-based privacy risks, CISOs should begin by managing data throughout its lifecycle and discarding data once it no longer provides the business value that would justify the cost and risk of protecting it, Willemsen advised. Data has a lifecycle, and eternity is an ill-advised lifecycle, so organizations should hardcode the end of it. For CISOs, protecting against AI inference-based threats starts with AI governance: setting privacy-by-design guardrails into AI development and assessing regularly for bias or inference risks. Organizations should adopt privacy-enhancing technologies, or PETs, which protect personal data by processing it in a protected state or confidential computing. PETs include differential privacy, synthetic data, privacy-aware machine learning, and homomorphic encryption, which runs calculations on encrypted data without the need to decrypt it first. Organizations should also expand on their traditional approaches to cybersecurity by prioritizing advanced monitoring, anomaly detection, and scenario-planning capabilities to identify inference-based threats. Privacy regulations, which have historically focused on directly identifiable data, should also extend to indirectly identifiable or reidentifiable content as AI technologies advance, Willemsen said. Do not underestimate the risk of different types of AI before using any of it, he warned. The erosion of practical obscurity online forces a rethink of who can be targeted and how cheaply. Organizations that treat inference as a peripheral concern rather than a core security discipline may find their data protection strategies obsolete before they're fully deployed.