For years we talked about data security in the vocabulary of walls. Stronger passwords, tighter access, better encryption, networks built to keep intruders out. None of that has stopped mattering. But artificial intelligence has moved the problem somewhere the walls don’t reach. Personal data now travels through long chains, meaning collection, training, testing, deployment, and the endless back-and-forth of people using a system day to day. Along the way an AI model can infer things nobody ever entered, stitch together data from contexts that were never meant to meet, and produce outputs faster than anyone can check them.
The tempo of the fight is shifting too. AI is compressing both the attack chain and the vulnerability-management cycle, and organisations should prepare for threats moving at machine speed. That is no reason to abandon the basics. If anything, it raises the price of getting them wrong: disciplined risk management, secure design, fast remediation, and human oversight that actually functions.
The numbers arriving on our own desk make the point concretely. This year the Croatian supervisory authority has already recorded twice as many personal data breach notifications under Article 33 of the GDPR as in the whole of 2025, and the trend is visible across the EU. Behind those filings sits a pattern we know well: a large share of organisations still treat information security and the protection of personal data as costs that can be deferred, right up until the moment they cannot be. AI raises the price of that habit year by year. The threats are accelerating; preparedness, in many organisations, is not. That gap can only be closed from the organisations’ side, and closing it can no longer wait.
Even those figures understate the problem. The question is larger than keeping a database from being breached. Data can be used far beyond what anyone reasonably expected, surfaced by accident in a system’s output, kept long after there was any reason to keep it, or turned into a decision that quietly harms someone. A locked database does nothing about any of that. The job is to protect the people the data is about, and that is a wider responsibility than security teams alone have traditionally owned.
You can be well defended and still expose people to serious harm, and the failures often have nothing to do with each other. A model can be walled off perfectly from outside attackers and still have been trained on personal data that was excessive, or gathered without any lawful basis in the first place. An AI assistant can behave exactly as its designers intended and still hand a user confidential records, because permissions were configured carelessly and no output was ever checked against who was allowed to see what. Accuracy doesn’t rescue you either: a system can be right on average and reliably wrong for one particular group of people, which is precisely the group that ends up carrying the cost. Cybersecurity, privacy, and responsible AI are usually treated as three separate meetings with three separate owners. They are three angles on one governance problem.
Article 32 of the GDPR is the right place to anchor this. It asks for security appropriate to the risk, judged against the state of the art, the context of the processing, and what could actually happen to a person if it goes wrong. Confidentiality is only one piece; integrity, availability, resilience, the ability to recover, and regular testing sit at the same level. In an AI system the word “security” has to stretch across the datasets, the model itself, the prompts, the outputs, the logs, the interfaces, the tools it connects to, and the people operating it. Leave any of those out of the assessment and the assessment is fiction.
The most useful work happens before anything is bought or switched on. The first question is whether AI is needed for the purpose at all, and whether the same result is reachable with less data and less machinery. Minimisation tends to get framed as a brake on innovation, which has it backwards: less data means a smaller attack surface, a smaller blast radius when something goes wrong, and often a system that is easier to explain, cheaper to test, and genuinely controllable.
From there the risk work has to run the whole length of the lifecycle rather than living in a launch-day sign-off. At design and procurement you need to know what data goes in, where it came from, who can reach it, how long it stays, and whether it can quietly be reused to train the next version. And the fact that data is sitting in public view online settles nothing about whether collecting and reusing it is lawful — or wise. The EDPB’s recent guidelines on web scraping for generative AI, now out for public consultation, push the conversation back to where it belongs: purpose limitation, transparency, accuracy, minimisation.
The threats worth mapping are specific: unauthorised access, prompt injection, data and model poisoning, sensitive information leaking out through ordinary outputs, insecure integrations, accounts and services running with far more privilege than they need. Agentic systems sharpen every one of these. Once a model can call another application, pull records, send a message, or delete something, a bad output stops being a bad sentence on a screen and becomes an action in the world before a human has had the chance to look.
None of this survives being done once. A control that was proportionate on launch day can be hopelessly inadequate the moment the model is retrained, wired to a new data source, or pointed at a different use. Testing, access reviews, logging, output monitoring, incident rehearsals — recurring obligations, not boxes to tick. Higher-risk processing may call for a data protection impact assessment; some deployments under the AI Act may require a fundamental rights impact assessment. Run them so they actually talk to each other, instead of as two parallel paperwork exercises nobody rereads.
Privacy-enhancing technologies help, within limits. Pseudonymisation, strong anonymisation, differential privacy, synthetic data, privacy-preserving computation , all of them can genuinely reduce a specific risk when it is chosen for a reason and tested against that reason. What they are not is a label you can stick on a system to make a problem disappear. Pseudonymised data is still personal data. Synthetic data is not automatically anonymous. A technique earns its keep only when it is tied to a defined risk and you can show that it works.
Responsibility does not evaporate into the supply chain either. One company builds the model, another integrates it, and the organisation actually deploying it decides what it is used for and on whom. Each of them needs honest information from the others, namely about data flows, controls, limitations, updates, and who owns the response when something goes wrong. Contracts matter, but a clause is not a control, and it cannot stand in for technical verification. An organisation that cannot say what happens to the information typed into a system is in no position to tell its staff, its customers, or the public what they can safely put in.
Human oversight has to mean something a person can actually do. Someone without the time, the authority, the information, or the expertise to overrule the machine is not performing oversight at all, whatever the policy says. Real oversight needs escalation paths that work, the power to stop or reverse an action, and training done on the system as it truly behaves rather than as the brochure describes it. People should understand how an innocent-looking prompt, a pasted document, or a newly connected tool can spill something sensitive.
No single technology, law, or department is going to deliver this. It comes from engineers, security staff, data protection officers, management, and the people using the system, all carrying some of the responsibility for whoever is affected downstream. Strong data protection has never meant eliminating risk, which is impossible anyway. It means seeing risk early, taking it down seriously, and staying answerable for the risk you decide to live with.
Data protection authorities do not guard data for its own sake. We guard the people it describes. Keep that at the centre and the supposed trade-off between innovation and security stops being the real fight. The real work is quieter — naming the risks honestly, reducing them, and owning what you choose to do about the rest.