GDPR, PII and AI: Where Your Customer Data Goes Now

A few years ago, a customer's details lived in a known set of places: the CRM, the billing system, the support desk and the backups behind them. A data protection lead could draw that map on a whiteboard.

AI has quietly redrawn it. The same customer's complaint is now pasted into a chat assistant to draft a reply, indexed so an internal bot can answer questions about it, summarised by a call-recording tool, and logged by an AI vendor in a region nobody checked. None of that is unusual. It is what using AI at work looks like.

GDPR has not changed. What has changed is the number of copies it applies to, and how few of them anyone decided to create. This article sets out where personal data goes once AI is in the business, what GDPR expects, and the controls that keep customer data safe without stopping the business using AI.

Where AI puts personal data

Prompts. Every customer email, CV or spreadsheet pasted into an assistant leaves the business. In 2023 Samsung restricted generative AI tools after engineers pasted internal source code into ChatGPT. That was code rather than personal data, but the route is the same one customer data takes every day.

Retrieval indexes. Assistants that answer from company records store documents in a vector database for search. Most setups keep the original text alongside each vector, so indexing the support inbox creates a second, searchable copy of every conversation in it.

Logs and transcripts. AI vendors and call-recording tools keep records of what went in and what came out, often the least governed copy of all.

Trained models. A model fine-tuned on personal data can reproduce it. The European Data Protection Board's Opinion 28/2024 concluded that such a model cannot automatically be treated as anonymous. Removing one person from a trained model is far harder than deleting a row.

Agents. An agent connected to the CRM, the inbox and the file share can move data between systems in ways nobody designed. Whatever it can read can end up in an answer, a log or a third-party tool.

What GDPR expects of an AI system

There is no separate GDPR for AI. The existing principles apply to the new plumbing:

• A lawful basis and a defined purpose (Article 5). Data collected to fulfil an order is not automatically available to train a model. In December 2024 the Italian regulator fined OpenAI €15 million, finding among other things that training on users' data lacked an adequate legal basis.
• Minimisation. Only the personal data the task needs. For most AI work this is the biggest lever, and the most neglected.
• Processor contracts (Article 28). An AI vendor handling customer data is your processor. You need a data processing agreement, its list of sub-processors, and a clear answer on whether it trains on your inputs.
• International transfers (Chapter V). Many AI services process in the US by default, and a transfer mechanism has to cover it.
• A DPIA before launch (Article 35). New technology processing customer data at scale is the textbook case.
• Rights that reach every copy. An erasure request (Article 17) has to reach the vector index, the logs and the test data, not only the CRM.

Fines under Article 83 reach €20 million or 4% of global turnover (£17.5 million or 4% under UK GDPR). The fine is rarely the largest cost. Customers, insurers and enterprise buyers now ask where their data goes when you use AI, and a business that cannot answer loses the deal before any regulator is involved.

The Commission's Digital Omnibus proposal of November 2025 would recognise AI development as a legitimate interest under GDPR. At the time of writing those provisions are still contested. Plan against GDPR as it stands.

Where the EU AI Act comes in

The EU AI Act does not replace GDPR; both apply to the same data. GDPR asks whether you may process it. The AI Act adds limits on what an AI system may use it for:

• Prohibited outright since February 2025. Article 5 bans social scoring, untargeted scraping of facial images to build recognition databases, emotion recognition in the workplace and in education, and biometric categorisation that infers sensitive traits such as ethnicity, religion or sexual orientation.
• High-risk from 2 December 2027. AI used in recruitment, credit scoring, life and health insurance pricing, education and access to essential services must use training and test data that is documented, relevant and examined for bias, with human oversight and a log of decisions. Sensitive personal data may be used for bias testing only under strict safeguards.

Our EU AI Act high-risk series covers each regulated sector in turn. For every provision in one place, swarmd.ai's EU AI Act guide, built by our core team, sets out each obligation article by article.

The risks that actually catch businesses out

The incidents that cause trouble are rarely sophisticated:

• Shadow AI. Staff using personal accounts on consumer tools because no approved one exists. Some consumer tiers use conversations to improve their models unless the user opts out, and the business has no contract with the provider at all.
• Over-permissioned assistants. An assistant connected to the whole file share will answer a question about salaries from an HR spreadsheet. Retrieval has to respect the permissions on the documents.
• Prompt injection. An inbound email or web page the model reads can contain instructions telling it to send data elsewhere. Limit what the model can reach, and log what it does.
• Copies made for convenience. An export for a fine-tuning experiment, a test set built from real tickets, a demo seeded with production data. Each is personal data with no retention rule.

Data residency, and where the model runs

Where a model runs decides who else can see the data. Most businesses will use more than one of three options.

A public API. The fastest route, often with the most capable models. Check the processing region, the retention period, the training terms, and whether a UK or EU processing option exists and is switched on. A provider subject to US law remains subject to it in a European data centre: residency is not the same as control.

A managed model in your own cloud account. Amazon Bedrock, Azure OpenAI and Google Vertex AI run models inside your existing tenancy, in a region you choose, under contracts you already hold. Check that features such as cross-region inference do not route requests elsewhere.

Open-weight models you host. Llama, Mistral and Qwen can run on servers you own or reserve. Prompts, outputs and logs never leave your environment, and you set the retention. The trade-off is that someone has to run and monitor it, which we covered in AI sovereignty: what it costs to teach an AI your business.

Match the option to the data: regulated and sensitive personal data on infrastructure you control, everyday drafting on a cloud tool under enterprise terms.

Remove the personal data before it reaches the model

The cheapest control is also the most effective: do not send personal data the task does not need. Summarising a contract, classifying a complaint or drafting a reply rarely depends on the customer's name, address or account number. Strip those first and a leak exposes far less, a vendor's retention period matters less, and a transfer question may not arise at all.

That is why we built PII Purge. It removes names, contact details, addresses, employers, education history, ID numbers and financial details from PDFs, Word documents and text. Documents are processed in memory, not retained and not used to train models, and redacted PDFs are rebuilt without the original text layer or metadata, so nothing is left hidden under a black box. Every step, including the language models, runs on our own EU infrastructure, so no document reaches a third-party AI service. It works on the website, over an API, or inside Claude through the Claude Directory, so staff can ask their assistant to strip personal details before working with a document. How we built it covers the pipeline and its accuracy. As with any automated tool, review the output before you share it.

One caution: pseudonymised data, where the business can still link it back to a person, remains personal data under GDPR. Redaction reduces risk; it removes your obligations only when the result is genuinely anonymous.

What to do now

1. Inventory every AI tool in use, including the ones staff adopted themselves, and add the personal data each touches to your record of processing.
2. Give staff an approved tool under enterprise terms, with a one-page rule on what must never be pasted anywhere else. A ban without an alternative drives AI onto personal accounts.
3. Check every AI vendor for a processing agreement, sub-processors, region, retention and training terms. Most businesses find at least one surprise.
4. Make redaction the default for documents going to a model, an index or a supplier.
5. Scope what assistants and agents can reach, with permission-aware retrieval and a log of every tool call.
6. Keep sensitive workloads on infrastructure you control.
7. Test that a deletion request reaches every copy, including the vector index and the logs.

Keeping customer data under control

Data protection engineering maps where personal data lives, including in AI systems, and builds retention, deletion and subject access into them. AI system security hardens assistants and agents against over-permissioned access, prompt injection and leakage. Private AI hosting runs open-weight models on reserved EU capacity for data that should never leave infrastructure you control. And PII Purge is free to try on a document today.

If you want to know where your own customer data has gone, request a proposal or talk to an engineer.

This is our view of the technical and operational side of data protection, not legal advice. Pair it with your data protection officer or legal counsel for formal sign-off.