Navigating Privacy Risks in Generative AI: Concerns, Challenges, and Potential Solutions

Jan 1, 2026•Bangyi Yang•View PDF

TL;DR Highlight

A survey paper covering 4 types of attacks that extract training data from LLMs and defense strategies against them.

Who Should Read

ML engineers, security researchers, and privacy officers responsible for deploying LLMs in production, especially on sensitive or proprietary data.

Core Mechanics

Taxonomy of 4 training data extraction attack types: memorization extraction, model inversion, membership inference, and attribute inference
Memorization rate increases with model size — larger models are more vulnerable to verbatim extraction
Repeated data in training sets dramatically increases extraction risk
Defense strategies: differential privacy, data deduplication, output filtering, and canary detection
No single defense fully mitigates all attack types; layered approaches are needed

Evidence

Survey covers 50+ papers published through 2024
Empirical evidence that GPT-2 and larger models can be prompted to regurgitate training data verbatim
Differential privacy provides formal guarantees but at a significant accuracy cost

How to Apply

Deduplicate your training data before fine-tuning — repeated examples are the primary driver of memorization.
Add output filtering to block responses that match known sensitive patterns (PII, proprietary text).
Use canary tokens in training data to detect if your model has been successfully queried for training data extraction.

Code Example

snippet

# Differential Privacy application example (Hugging Face + Opacus)
from opacus import PrivacyEngine
from torch.optim import AdamW

optimizer = AdamW(model.parameters(), lr=5e-5)

privacy_engine = PrivacyEngine()
model, optimizer, train_loader = privacy_engine.make_private_with_epsilon(
    module=model,
    optimizer=optimizer,
    data_loader=train_loader,
    epochs=3,
    target_epsilon=5.0,   # recommended value from paper
    target_delta=1e-6,    # recommended value from paper
    max_grad_norm=1.0,
)

# Training loop can be used the same way
for batch in train_loader:
    outputs = model(**batch)
    loss = outputs.loss
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

print(f"Privacy budget used: ε={privacy_engine.get_epsilon(delta=1e-6):.2f}")

Terminology

MemorizationWhen an LLM stores and can reproduce training data verbatim, posing privacy and IP risks.

Membership Inference AttackAn attack that determines whether a specific data point was in the model's training set.

Differential PrivacyA mathematical framework that adds calibrated noise to training to limit how much any individual data point influences model outputs.

Canary TokenA unique, fictitious data string inserted into training data to detect if it can be extracted from the trained model.

Original Abstract (Expand)

The rapid advancement of Generative Artificial Intelligence (GenAI) and Large Language Models (LLMs) has revolutionized numerous applications across healthcare, finance, and customer service. However, these technological breakthroughs introduce significant privacy risks as models may inadvertently memorize and expose sensitive information from their training data. This paper provides a comprehensive analysis of current privacy vulnerabilities in GenAI systems, including membership inference attacks, model inversion attacks, data extraction techniques, and data poisoning vulnerabilities. We examine state-of-the-art mitigation strategies including differential privacy (DP), cryptographic methods, anonymization techniques, and perturbation strategies. Through analysis of real-world case studies and empirical evidence, we demonstrate that current privacy-preserving techniques, while promising, face significant utility-privacy trade-offs. Our findings indicate that ε-differential privacy with ε = 5, δ = 10^-6 provides adequate protection for most practical applications, though stronger guarantees may be necessary for highly sensitive data. We conclude by presenting a comprehensive framework for user-centric privacy design and identifying critical areas for future research in privacy-preserving generative AI.