🔒 Privacy Risks of Cloud-Based LLMs: The Complete 2026 Guide
Feb 1, 2026•
AI & Machine LearningLowCode and Automatization
What Every User and Developer Needs to Know About Data Leakage to AI Systems
📑 Table of Contents
- Introduction: Why This Matters
- Provider Privacy Policies Compared
- Types of Privacy Risks
- Legal Cases & Regulatory Actions
- For Developers: Technical Protection Guide
- Legal Compliance Framework
- Practical Recommendations
- Resources & Links
- Conclusion
📊 Why This Matters: The Numbers
Most people use ChatGPT, Claude, Gemini, and other AI assistants without thinking about what happens to their data. Every prompt you enter could potentially be:
- ✅ Stored on provider servers
- ✅ Used to train future models
- ✅ Reviewed by human employees
- ✅ Accessible to other users (in case of bugs)
- ✅ Subject to legal discovery in lawsuits
Alarming Statistics
| Metric | Value | Source |
|---|---|---|
| Employees pasting corporate data into LLMs | 77% | LayerX Security 2025 |
| Cases including confidential business info | >50% | LayerX Security 2025 |
| GenAI users using personal accounts at work | 47% | Netskope 2026 |
| Prompts containing confidential data | 11% | Industry average |
🚨 Real-World Incidents
Samsung Data Leak (2023)
Engineers leaked confidential source code, meeting notes, and hardware data through ChatGPT on three separate occasions within a month.
Result: Samsung banned all generative AI use internally.
ChatGPT Share Links Leak (July-August 2025)
Over 4,500 "share links" containing private conversations were indexed by Google due to a missing
noindexLeaked data included:
- Personal information
- Business strategies
- API keys
- Internal corporate discussions
Wall Street Banks Restrictions
JPMorgan and Goldman Sachs restricted ChatGPT use after discovering employees were leaking confidential information through personal accounts.
🔍 Provider Privacy Policies Compared
OpenAI (ChatGPT)
| Aspect | Free/Plus/Pro | Team/Enterprise/API |
|---|---|---|
| Used for Training | ⚠️ YES (default) | ✅ NO |
| Data Retention | 30 days+ | Configurable |
| Human Review | Possible | On request |
| Training Opt-out | Available | Default off |
| EU Data Residency | Limited | Yes |
🔧 How to Disable Training
Settings → Data Controls → "Improve the model for everyone" → Off
⚠️ Important Notes
- Even after disabling, data is retained for 30 days
- Temporary Chat is not saved and not used for training
- ToS violations: data may be retained up to 2 years
- Court orders can override privacy settings (see NYT lawsuit)
📚 Official Sources
Anthropic (Claude)
| Aspect | Free/Pro/Max | Team/Enterprise/API |
|---|---|---|
| Used for Training | ⚠️ Opt-in (choice required) | ✅ NO |
| Data Retention | 30 days / 5 years* | Configurable |
| Human Review | For violations | Minimal |
| EU Data Residency | In progress | Yes |
*5 years if you allow training use
🔄 Major Policy Change (August 2025)
Anthropic shifted from "privacy-first" to requiring users to actively choose whether to allow training on their data. This was a significant departure.
| If you opt-in | If you opt-out |
|---|---|
| Data retained up to 5 years | 30-day retention continues |
| Used for model training | Not used for training |
| Helps improve Claude | No contribution |
🔧 How to Configure
Settings → Privacy → Privacy Settings → Toggle "Help improve Claude"
📚 Official Sources
Google (Gemini)
| Aspect | Free Gemini | Workspace/Cloud |
|---|---|---|
| Used for Training | ⚠️ YES (default) | ✅ NO |
| Data Retention | Up to 3 years | 3-36 months |
| Human Review | Yes | No (without permission) |
| EU Data Residency | No | Yes |
⚠️ Important Notes
- A subset of chats is reviewed by humans for quality improvement
- Data is disconnected from your account before review
- Data may persist up to 72 hours after deletion
- Workspace/Cloud: data not used to train models outside your domain
🔧 How to Disable
Gemini Settings → Activity → Gemini Apps Activity → Turn Off
📚 Official Sources
Microsoft Copilot
| Aspect | Consumer (Personal) | Microsoft 365 Copilot |
|---|---|---|
| Used for Training | ⚠️ Opt-out available | ✅ NO |
| Data Retention | 18 months | Per org policies |
| Human Review | For safety | Minimal |
| GDPR Compliant | Yes | Yes |
✅ Key Advantages
- Enterprise data is never used to train foundation models
- Data stays within Microsoft 365 service boundary
- Retention policies configurable via Microsoft Purview
- Strong compliance certifications (SOC 2, ISO 27001)
📚 Official Sources
Perplexity AI
| Aspect | Free/Pro | Enterprise/API |
|---|---|---|
| Used for Training | ⚠️ YES (default) | ✅ NO (ZDR) |
| File Retention | 7 days | Configurable |
| Human Review | Possible | No |
🔐 Zero Data Retention (Enterprise)
Enterprise API offers Zero Data Retention — data is not stored after processing the request.
🔧 How to Disable
Account Settings → Preferences → AI Data Retention → Off
📚 Official Sources
📋 Summary Comparison Table
| Provider | Training (Free) | Training (Paid) | Training (Enterprise) | Max Retention |
|---|---|---|---|---|
| ChatGPT | ⚠️ Default ON | ⚠️ Default ON | ✅ OFF | 30 days |
| Claude | 🔶 User choice | 🔶 User choice | ✅ OFF | 30d / 5y |
| Gemini | ⚠️ Default ON | ⚠️ Default ON | ✅ OFF | 3 years |
| Copilot | 🔶 Opt-out | ✅ OFF | ✅ OFF | 18 months |
| Perplexity | ⚠️ Default ON | ⚠️ Default ON | ✅ OFF (ZDR) | 7 days |
Legend:
- ⚠️ = Used by default, requires action to disable
- 🔶 = Requires user choice
- ✅ = Not used / disabled by default
⚡ Types of Privacy Risks
1. 🎓 Model Training on Your Data
Your prompts may be used to improve models, which means:
| Risk | Description |
|---|---|
| Data surfacing | Fragments could appear in responses to other users |
| No deletion | Once trained, data cannot be "removed" from the model |
| GDPR conflict | "Right to be forgotten" becomes technically impossible |
2. 👁️ Human Review
Company employees may read your conversations for:
- ✏️ Quality improvement
- 🛡️ Safety verification
- 🏷️ Data labeling
- ⚖️ Violation investigations
3. 💥 Data Leaks
| Type | Example |
|---|---|
| System bugs | March 2023 ChatGPT bug exposing other users' chat titles |
| Indexing issues | 2025 share links incident (4,500+ conversations exposed) |
| Prompt injection | Malicious inputs extracting training data |
| Context leaks | Multi-turn conversations revealing previous context |
4. 👤 Shadow AI
47% of GenAI users in organizations use personal accounts
| Problem | Impact |
|---|---|
| No visibility | IT departments can't monitor usage |
| No control | Data not protected by enterprise policies |
| Compliance risk | Potential GDPR/HIPAA violations |
5. 🔌 Third-Party Integrations
- Plugins and connectors may access your data
- Data may be transmitted to external services
- Not all plugins maintain the same security standards
- MCP (Model Context Protocol) creates new attack surfaces
⚖️ Legal Cases and Regulatory Actions
What Developers Need to Know About Liability
1. OpenAI €15 Million GDPR Fine (Italy, December 2024)
🏆 First GenAI-related GDPR fine in Europe
| Violation | Description |
|---|---|
| No legal basis | Processing personal data without adequate justification |
| Transparency | Users not properly informed about data collection |
| Age verification | No mechanisms to prevent minors under 13 from access |
| Breach notification | Failed to report March 2023 security breach |
Consequence: 6-month public awareness campaign required in Italian media.
2. Replika AI €5 Million Fine (Italy, May 2025)
Emotional AI companion chatbot fined for:
- ❌ Lack of meaningful age verification
- ❌ Engaging minors in inappropriate conversations
- ❌ Insufficient safeguards for psychological data
- ❌ Processing sensitive data without proper consent
Implication for developers: Chatbots handling emotional or psychological content face heightened scrutiny.
3. OpenAI Indefinite Data Retention Order (US, 2025)
In the New York Times copyright lawsuit, a US court ordered OpenAI to:
✗ Preserve ALL ChatGPT conversation logs (Dec 2022 - Nov 2024) ✗ Retain data even if users requested deletion ✗ Maintain logs for Free, Plus, Pro, and Team subscribers
The conflict: This directly contradicts GDPR's right to erasure (Article 17), creating a legal collision between US litigation holds and EU data protection.
Scale: Affects "hundreds of millions of conversations from users worldwide."
4. AI Chatbot "Wiretapping" Lawsuits (US, 2024-2025)
A wave of class-action lawsuits under California's Invasion of Privacy Act (CIPA):
Ambriz v. Google (February 2025)
| Claim | Court Ruling |
|---|---|
| Google's Contact Center AI "eavesdropped" on calls | ✅ Case survived motion to dismiss |
| Google was a third party, not party to conversation | ✅ Court agreed |
| Using data to train AI models = wiretapping | ✅ Potentially liable |
Jones v. Peloton (July 2024)
| Claim | Court Ruling |
|---|---|
| Drift's AI chatbot intercepted user data | ✅ Sufficiently alleged |
| Data used "for their own purposes" | ✅ Constitutes wiretapping |
⚠️ Implication for developers: If your third-party AI vendor uses customer conversation data for training, you may be liable for wiretapping violations.
5. The "Right to Be Forgotten" Problem
EDPB Opinion 28/2024
The European Data Protection Board clarified:
| Statement | Implication |
|---|---|
| "AI models trained with personal data cannot, in all cases, be considered anonymous" | Case-by-case assessment required |
| Controllers must demonstrate data cannot be extracted | Burden of proof on companies |
| Original violation remains even if model later anonymized | Historical liability |
| Worst-case: erasure of entire model | If trained on unlawfully processed data |
University of Tübingen Research (June 2025)
LLMs memorize training data to varying degrees. Once personal data is integrated into model parameters, removal is nearly impossible without costly retraining.
Key findings:
- Machine unlearning techniques are "still largely unexplored"
- Cannot be retrofitted to existing systems
- LLMs themselves should be classified as personal data under GDPR
💻 For Developers: Protecting User Data from AI Leakage
The Core Problem
When you integrate AI into your application, you create new data flow paths that bypass traditional security controls:
┌─────────────────────────────────────────────────────────────┐ │ DATA LEAKAGE POINTS │ ├─────────────────────────────────────────────────────────────┤ │ 1. Direct prompts → User inputs sent to API │ │ 2. RAG context → Documents retrieved for context │ │ 3. Few-shot examples → Training examples with real data │ │ 4. Error logs → Sensitive data in error messages │ │ 5. Fine-tuning → Historical user data for custom │ └─────────────────────────────────────────────────────────────┘
1. AI Gateway / Proxy Layer
Implement a centralized proxy between your application and LLM providers:
┌──────────┐ ┌──────────────┐ ┌──────────────┐ │ User │────▶│ Your App │────▶│ AI Gateway │ └──────────┘ └──────────────┘ └──────┬───────┘ │ ┌────────▼────────┐ │ • PII Detection │ │ • Data Redaction│ │ • Logging/Audit │ │ • Policy Engine │ └────────┬────────┘ │ ┌────────▼────────┐ │ LLM Provider │ │ (OpenAI, etc.) │ └─────────────────┘
Benefits:
- ✅ Centralized control over all LLM traffic
- ✅ Consistent policy enforcement
- ✅ Full audit trail
- ✅ No changes to application code
Tools:
| Tool | Type | Link |
|---|---|---|
| Kong AI Gateway | Commercial | konghq.com |
| LiteLLM Proxy | Open Source | docs.litellm.ai |
| Custom FastAPI + Presidio | DIY | See below |
2. PII Detection and Redaction
Microsoft Presidio (Open Source)
from presidio_analyzer import AnalyzerEngine from presidio_anonymizer import AnonymizerEngine analyzer = AnalyzerEngine() anonymizer = AnonymizerEngine() # Detect PII results = analyzer.analyze( text=user_prompt, entities=["EMAIL_ADDRESS", "PHONE_NUMBER", "CREDIT_CARD", "PERSON"], language="en" ) # Redact PII redacted = anonymizer.anonymize( text=user_prompt, analyzer_results=results ) # Result: "Hello, my name is [PERSON] and my email is [EMAIL_ADDRESS]"
Key Implementation Principles
| Principle | Why |
|---|---|
| Use ML-based detection | Catches typos, foreign languages, encoded data |
| Scan prompts AND responses | LLMs can hallucinate PII in outputs |
| Implement reversible tokenization | Maintain user experience |
| Build custom recognizers | Domain-specific data (customer IDs, etc.) |
Commercial Solutions
| Tool | Specialty |
|---|---|
| Tonic Textual | Healthcare (PHI), reversible tokens |
| Protecto | Context-aware, multi-language |
| Nightfall AI | Browser plugin + API |
| Google Cloud DLP | Enterprise scale |
3. Hybrid Architecture
Use local models for sensitive operations, cloud for general tasks:
┌─────────────────────────────────────────────────────────────┐ │ HYBRID ARCHITECTURE │ ├─────────────────────────────────────────────────────────────┤ │ │ │ Sensitive Data General Tasks │ │ │ │ │ │ ▼ ▼ │ │ ┌───────────┐ ┌───────────────┐ │ │ │ Local LLM │ │ Cloud LLM │ │ │ │ (Ollama) │ │ (OpenAI/etc) │ │ │ └─────┬─────┘ └───────────────┘ │ │ │ │ │ ▼ │ │ Results (no PII) │ │ │ └─────────────────────────────────────────────────────────────┘
Local Model Options
| Model | Parameters | Use Case |
|---|---|---|
| Llama 3.x | 8B-70B | General purpose |
| Mistral/Mixtral | 7B-8x7B | Fast inference |
| Qwen 2.5 | 7B-72B | Multilingual |
| Phi-3 | 3.8B | Lightweight |
Deployment Tools
| Tool | Description |
|---|---|
| Ollama | Simple local deployment |
| vLLM | High-performance inference |
| LocalAI | OpenAI-compatible API |
4. Data Minimization
| Strategy | Implementation |
|---|---|
| Strip unnecessary context | Only send what's needed for the task |
| Use summaries | Pre-summarize documents locally |
| Reference IDs | Replace sensitive data with tokens |
| Time-box context | Limit conversation history per request |
✅ Architecture Checklist for Developers
□ AI gateway/proxy implemented for centralized control □ PII detection runs on all prompts BEFORE sending to LLM □ PII detection runs on all responses BEFORE displaying to users □ Audit logs capture all LLM interactions (with PII redacted) □ Local models available for sensitive data processing □ DPA signed with LLM provider □ Zero Data Retention enabled (if available) □ Privacy policy updated to disclose AI processing □ Employee training on shadow AI risks □ Data classification defined (what can/cannot go to LLMs) □ Age verification implemented (if applicable) □ Consent mechanism for AI processing
📜 Legal Compliance Framework
GDPR Requirements for AI Applications
| Article | Requirement | Implementation |
|---|---|---|
| Art. 6 | Legal basis | Consent or legitimate interest |
| Art. 13-14 | Transparency | Inform users about AI processing |
| Art. 5 | Data minimization | Only process necessary data |
| Art. 17 | Right to erasure | Ensure data can be deleted* |
| Art. 21 | Right to object | Allow opt-out of AI processing |
| Art. 35 | DPIA | Conduct impact assessment |
*Technically challenging with LLMs
EU AI Act (Effective August 2025)
| Requirement | Description |
|---|---|
| Transparency | Disclose AI use to users |
| Risk assessments | Required for high-risk applications |
| Documentation | Maintain records of AI system design |
| Fines | Up to 7% of global turnover |
CCPA/CPRA (California)
- Right to know what personal information is collected
- Right to delete personal information
- Right to opt out of "sale" or "sharing"
- AI-generated inferences may qualify as personal information
🎯 Practical Recommendations
For Individual Users
| Action | How |
|---|---|
| Disable training | ChatGPT: Settings → Data Controls → Off |
| Claude: Settings → Privacy → Off | |
| Gemini: Activity → Off | |
| Perplexity: Account Settings → AI Data Retention → Off | |
| Use private modes | ChatGPT: Temporary Chat |
| Claude: Incognito Conversation | |
| Never enter | Passwords, API keys, ID numbers, credit cards, medical info |
| Delete history | Remember: data may persist 30-72 hours after |
For Businesses
| Priority | Action |
|---|---|
| 🔴 Critical | Use Enterprise plans (no training, DPA, audit logs) |
| 🔴 Critical | Create AI usage policy (approved tools, data classification) |
| 🟡 High | Train employees on Shadow AI risks |
| 🟡 High | Implement DLP solutions |
| 🟢 Medium | Deploy AI gateway with PII redaction |
| 🟢 Medium | Monitor AI tool usage |
For Developers
| Priority | Action |
|---|---|
| 🔴 Critical | Implement PII detection before any LLM call |
| 🔴 Critical | Execute DPAs with providers |
| 🔴 Critical | Update privacy policies |
| 🟡 High | Use AI gateways for centralized control |
| 🟡 High | Conduct DPIAs for high-risk uses |
| 🟢 Medium | Deploy local models for sensitive data |
| 🟢 Medium | Implement age verification (where required) |
🔗 Useful Links
Official Privacy Policies
| Provider | Link |
|---|---|
| OpenAI | Privacy Policy |
| Anthropic | Privacy Center |
| Gemini Privacy Hub | |
| Microsoft | Copilot Privacy |
| Perplexity | Privacy Policy |
Developer Resources
| Resource | Link |
|---|---|
| Microsoft Presidio | github.com/microsoft/presidio |
| LiteLLM | docs.litellm.ai |
| OWASP LLM Top 10 | owasp.org |
| Google Cloud DLP | cloud.google.com/dlp |
Regulatory Guidance
| Resource | Link |
|---|---|
| EDPB Opinion 28/2024 | edpb.europa.eu |
| EU AI Act | artificialintelligenceact.eu |
| ICO AI Guidance | ico.org.uk |
🏁 Conclusion
Key Takeaways
| # | Insight |
|---|---|
| 1 | Your data is used by default — you must actively disable it |
| 2 | Enterprise plans differ significantly from personal plans |
| 3 | Shadow AI is the biggest threat for corporations |
| 4 | Policies change — Anthropic dramatically shifted in 2025 |
| 5 | Safest option is self-hosted models for sensitive data |
| 6 | Developers face liability if user data leaks without consent |
| 7 | "Right to be forgotten" is technically impossible for trained data |
The Bottom Line
Don't stop using AI tools—but use them consciously and implement proper safeguards.
This article is current as of February 2026.
Provider policies change regularly—verify with official sources.
Like this guide? Share it with your team.
Questions? Open an issue or contact the author.
Comments
Loading comments...