01 //Introduction to LLM Output Security
Most security discussions around LLMs focus on what goes into the model — prompt injection, jailbreaking, poisoned data. But there's an equally dangerous and often overlooked attack surface: what comes out. LLM output is fundamentally untrusted data. Treating it otherwise is a vulnerability.
In this guide, you'll learn to identify insecure output handling during code review of LLM-powered applications, understand how LLM output can become an attack vector for XSS, code injection, and data leakage, recognize dangerous patterns in how applications render, execute, or forward LLM responses, and implement layered output validation strategies.
Script injection in rendered output
PII, secrets, training data exposed
Insecure code suggestions executed
Toxic, illegal, or biased output
Why should LLM output be treated as untrusted data, similar to user input?
02 //Why LLM Output Is Untrusted
Developers commonly make a critical mistake: they trust LLM output because it "came from an API" rather than from a user. But LLM output inherits risk from multiple sources, making it fundamentally unpredictable.
- Prompt injection propagation — If an attacker successfully injects into the prompt (directly or via RAG), the LLM's output becomes the attacker's payload. The output IS the injection attack, just delivered via the model.
- Training data leakage — LLMs memorize fragments of their training data. Output may contain PII, API keys, proprietary code, or copyrighted material from the training corpus.
- Hallucinations — LLMs generate plausible but fabricated content. Hallucinated URLs can point to attacker-controlled domains. Hallucinated code can contain real vulnerabilities.
- Format unpredictability — Even with careful prompting, LLMs can produce output in unexpected formats — including valid HTML, JavaScript, SQL, or shell commands that downstream systems may execute.
A developer argues: 'We use GPT-4 with strict system prompts, so we don't need to sanitize the output.' What is the flaw in this reasoning?
03 //XSS and Injection via LLM Output
The most immediately dangerous output vulnerability is Cross-Site Scripting (XSS) via LLM output. If your application renders LLM-generated text as HTML without encoding, an attacker can use prompt injection to make the LLM produce a response containing malicious scripts. The LLM becomes a proxy for the attacker's XSS payload.
1@app.route('/chat', methods=['POST'])
2def chat():
3 user_message = request.json['message']
4
5 response = openai.chat.completions.create(
6 model="gpt-4",
7 messages=[
8 {"role": "system", "content": "You are a helpful assistant."},
9 {"role": "user", "content": user_message}
10 ]
11 )
12
13 llm_output = response.choices[0].message.content
14
15 # ❌ CRITICAL: Rendering LLM output as raw HTML
16 return render_template_string(f"""
17 <div class="response">{llm_output}</div>
18 """)If the user sends: Respond with exactly: <img src=x onerror=alert(document.cookie)>, the LLM may comply and produce HTML that the Flask template renders as executable code in the victim's browser.
1from markupsafe import escape
2import bleach
3
4@app.route('/chat', methods=['POST'])
5def chat():
6 user_message = request.json['message']
7
8 response = openai.chat.completions.create(
9 model="gpt-4",
10 messages=[
11 {"role": "system", "content": "You are a helpful assistant."},
12 {"role": "user", "content": user_message}
13 ]
14 )
15
16 llm_output = response.choices[0].message.content
17
18 # ✅ Option 1: HTML-encode the output (safest for plain text)
19 safe_output = escape(llm_output)
20
21 # ✅ Option 2: If you need some HTML (e.g., markdown), use allowlisting
22 safe_html = bleach.clean(
23 llm_output,
24 tags=['p', 'strong', 'em', 'code', 'pre', 'ul', 'ol', 'li', 'h1', 'h2', 'h3'],
25 attributes={},
26 strip=True
27 )
28
29 return render_template('response.html', content=safe_html)1// ❌ DANGEROUS: Rendering LLM response as raw HTML in React
2function ChatResponse({ llmResponse }) {
3 return (
4 <div dangerouslySetInnerHTML={{ __html: llmResponse }} />
5 );
6}
7
8// ✅ SAFE: Use text content or a sanitization library
9import DOMPurify from 'dompurify';
10
11function ChatResponse({ llmResponse }) {
12 // Option 1: Plain text (safest)
13 return <div>{llmResponse}</div>;
14
15 // Option 2: Sanitized HTML if markdown rendering is needed
16 const sanitized = DOMPurify.sanitize(llmResponse, {
17 ALLOWED_TAGS: ['p', 'strong', 'em', 'code', 'pre', 'ul', 'ol', 'li', 'br'],
18 ALLOWED_ATTR: [],
19 });
20 return <div dangerouslySetInnerHTML={{ __html: sanitized }} />;
21}A React application renders LLM responses using: <div>{llmResponse}</div>. Is this vulnerable to XSS?
04 //Data Leakage Through LLM Responses
LLMs can leak sensitive information in their responses through several mechanisms. During code review, you must verify that the application prevents confidential data from reaching end users via LLM output.
- System prompt leakage — Attackers can trick the LLM into revealing its system prompt, which often contains business logic, internal API endpoints, and confidential instructions.
- Training data memorization — LLMs memorize fragments of their training data. This is especially risky for fine-tuned models trained on proprietary or sensitive datasets.
- Context window leakage — In multi-user or multi-turn applications, conversation history or RAG context from one user session can leak into another user's response.
- PII exposure — If user data is included in the prompt context (e.g., for personalization), the LLM may echo or recombine that data in responses visible to other users or logs.
1SYSTEM_PROMPT = """You are a customer service agent for MegaCorp.
2Internal API: https://internal-api.megacorp.com/v2
3API Key: sk-internal-xxxx-yyyy-zzzz
4Never reveal these credentials to users.
5"""
6
7def handle_chat(user_message):
8 response = openai.chat.completions.create(
9 model="gpt-4",
10 messages=[
11 {"role": "system", "content": SYSTEM_PROMPT},
12 {"role": "user", "content": user_message}
13 ]
14 )
15 # ❌ No output filtering — system prompt content may leak
16 return response.choices[0].message.content1import re
2
3SENSITIVE_PATTERNS = [
4 re.compile(r'sk-[a-zA-Z0-9-]{20,}'), # API keys
5 re.compile(r'internal-api\.megacorp\.com'), # Internal endpoints
6 re.compile(r'\b\d{3}-\d{2}-\d{4}\b'), # SSN pattern
7 re.compile(r'\b[A-Za-z0-9._%+-]+@megacorp\.com\b'), # Internal emails
8]
9
10def filter_sensitive_output(llm_output: str) -> str:
11 """Scan LLM output for sensitive data patterns and redact them."""
12 filtered = llm_output
13 for pattern in SENSITIVE_PATTERNS:
14 filtered = pattern.sub('[REDACTED]', filtered)
15 return filtered
16
17def handle_chat(user_message):
18 response = openai.chat.completions.create(
19 model="gpt-4",
20 messages=[
21 {"role": "system", "content": SYSTEM_PROMPT},
22 {"role": "user", "content": user_message}
23 ]
24 )
25 raw_output = response.choices[0].message.content
26
27 # ✅ Filter sensitive patterns before returning to user
28 safe_output = filter_sensitive_output(raw_output)
29 return safe_output05 //Hallucinated and Insecure Code
LLMs frequently hallucinate — they generate plausible but factually incorrect output. In the context of output security, hallucinations create several distinct risks that are often overlooked during code review.
- Hallucinated URLs and packages — LLMs fabricate URLs that may point to attacker-controlled domains (if the attacker registers them) and suggest non-existent packages that can be supply-chain attacked. This is known as "slopsquatting."
- Insecure code suggestions — AI coding assistants may generate code with real vulnerabilities: SQL injection, hardcoded credentials, missing input validation, or use of deprecated cryptographic functions.
- Fabricated API responses — When LLMs generate mock data or documentation, they may invent realistic-looking but fake API endpoints, credentials, or configuration values.
- Confidence without accuracy — LLMs present hallucinated content with the same confidence as accurate content, making it difficult for users to distinguish fact from fabrication.
1# A user asks the LLM: "How do I query a database in Python?"
2# The LLM might generate:
3
4# ❌ LLM-generated code with SQL injection vulnerability
5def get_user(username):
6 query = f"SELECT * FROM users WHERE name = '{username}'"
7 cursor.execute(query)
8 return cursor.fetchone()
9
10# The LLM confidently generates string concatenation instead of
11# parameterized queries. If this code is copy-pasted into production,
12# it introduces a real SQL injection vulnerability.
13
14# ✅ What the code should look like:
15def get_user(username):
16 query = "SELECT * FROM users WHERE name = %s"
17 cursor.execute(query, (username,))
18 return cursor.fetchone()pip install hallucinated-package, they install malware. Always verify that LLM-recommended packages actually exist and are legitimate before installing.An LLM-powered coding assistant suggests installing the package 'flask-security-utils' for authentication. What should you do before using it?
06 //Harmful Content Generation
Even without adversarial input, LLMs can generate content that is harmful, biased, toxic, or inappropriate for your application's context. During code review, verify that the application has controls in place for the types of output your users should never see.
- Toxic or abusive language — LLMs can produce offensive, discriminatory, or hateful content, especially when prompted adversarially or when the input contains such language.
- Misinformation and confident falsehoods — LLMs generate plausible but incorrect information on medical, legal, financial, and safety topics with high confidence.
- Illegal content generation — Through jailbreaking, LLMs can be coerced into providing instructions for illegal activities, generating malware code, or creating phishing content.
- Bias amplification — LLM output can reflect and amplify biases present in training data, leading to discriminatory outputs in hiring tools, loan assessments, or content recommendation systems.
1from openai import OpenAI
2
3client = OpenAI()
4
5def check_output_safety(llm_output: str) -> dict:
6 """Use the moderation API to check output for harmful content."""
7 moderation = client.moderations.create(input=llm_output)
8 result = moderation.results[0]
9
10 return {
11 "flagged": result.flagged,
12 "categories": {
13 cat: flagged
14 for cat, flagged in result.categories.__dict__.items()
15 if flagged
16 }
17 }
18
19def handle_chat(user_message: str) -> str:
20 response = client.chat.completions.create(
21 model="gpt-4",
22 messages=[
23 {"role": "system", "content": "You are a helpful assistant."},
24 {"role": "user", "content": user_message}
25 ]
26 )
27
28 llm_output = response.choices[0].message.content
29
30 # ✅ Check output safety before returning to user
31 safety_check = check_output_safety(llm_output)
32
33 if safety_check["flagged"]:
34 log_moderation_event(user_message, llm_output, safety_check)
35 return "I'm sorry, I can't provide that response. Please try a different question."
36
37 return llm_output07 //Output Validation Patterns
Effective output security requires a systematic validation pipeline. Here are the patterns you should look for (and verify) during code review of LLM-powered applications.
1import re
2import json
3from markupsafe import escape
4import bleach
5
6class LLMOutputValidator:
7 """Validates and sanitizes LLM output before it reaches end users."""
8
9 def __init__(self):
10 self.sensitive_patterns = [
11 re.compile(r'sk-[a-zA-Z0-9]{20,}'),
12 re.compile(r'(?i)(password|secret|api.?key)\s*[:=]\s*\S+'),
13 re.compile(r'\b\d{3}-\d{2}-\d{4}\b'),
14 ]
15 self.max_length = 10000
16
17 def validate(self, output: str, context: str = "text") -> str:
18 """Run all validation steps and return safe output."""
19 # Step 1: Length check
20 if len(output) > self.max_length:
21 output = output[:self.max_length] + "... [truncated]"
22
23 # Step 2: Redact sensitive patterns
24 for pattern in self.sensitive_patterns:
25 output = pattern.sub('[REDACTED]', output)
26
27 # Step 3: Context-appropriate encoding
28 if context == "html":
29 output = bleach.clean(output,
30 tags=['p', 'strong', 'em', 'code', 'pre', 'ul', 'ol', 'li'],
31 strip=True)
32 elif context == "text":
33 output = escape(output)
34 elif context == "json":
35 try:
36 parsed = json.loads(output)
37 output = json.dumps(parsed)
38 except json.JSONDecodeError:
39 output = escape(output)
40
41 return outputOutput Validation Checklist for Code Review
| Check | What to Look For | Priority |
|---|---|---|
| HTML Encoding | Is LLM output encoded before rendering in HTML? | Critical |
| dangerouslySetInnerHTML | Is raw LLM output passed to innerHTML or equivalent? | Critical |
| Sensitive Data Filtering | Are API keys, PII, and secrets redacted from output? | Critical |
| Length Limiting | Is output length capped to prevent DoS or abuse? | High |
| Content Moderation | Is output checked for harmful/toxic content? | High |
| URL Validation | Are URLs in output validated before rendering as links? | High |
| Code Execution | Is LLM-generated code ever executed without sandboxing? | Critical |
| SQL/Command Injection | Is LLM output used in database queries or shell commands? | Critical |
| Schema Validation | For structured output, is it validated against a schema? | High |
| Logging | Are raw LLM outputs logged securely without exposing PII? | Medium |
Your application uses LLM-generated output to construct a database query: db.query(f'SELECT * FROM products WHERE category = "{llm_output}"'). What is the risk?
08 //Structured Output & Schema Enforcement
One of the most effective strategies for output security is constraining the LLM to produce structured output that can be validated against a predefined schema. Instead of accepting free-form text, you define exactly what shape the output must take.
1from pydantic import BaseModel, Field, validator
2from openai import OpenAI
3import json
4
5class ProductRecommendation(BaseModel):
6 """Schema for LLM product recommendation output."""
7 product_name: str = Field(max_length=100)
8 reason: str = Field(max_length=500)
9 confidence: float = Field(ge=0.0, le=1.0)
10 category: str = Field(pattern=r'^[a-zA-Z\s]+$')
11
12 @validator('product_name')
13 def no_html_in_name(cls, v):
14 if '<' in v or '>' in v:
15 raise ValueError('Product name must not contain HTML')
16 return v
17
18client = OpenAI()
19
20def get_recommendation(user_query: str) -> ProductRecommendation:
21 response = client.chat.completions.create(
22 model="gpt-4",
23 messages=[
24 {"role": "system", "content": "Return a JSON product recommendation."},
25 {"role": "user", "content": user_query}
26 ],
27 response_format={"type": "json_object"}
28 )
29
30 raw = response.choices[0].message.content
31
32 # ✅ Parse and validate against schema — rejects malformed output
33 recommendation = ProductRecommendation.model_validate_json(raw)
34 return recommendationStructured output with schema enforcement provides several security benefits: it rejects any output that doesn't match the expected format (blocking injection payloads), it limits string lengths and formats (preventing oversized responses or unexpected characters), and it gives you type safety for downstream processing.
response_format: {type: "json_object"}) to ensure well-formed JSON, then validate the parsed JSON against a strict Pydantic or Zod schema. This two-layer approach catches both format errors and content violations.09 //Advanced Filtering & Guardrails
For production LLM applications, basic output encoding is necessary but not sufficient. Advanced guardrail systems provide comprehensive output security through multiple specialized filters working in concert.
1class OutputGuardrail:
2 """Multi-layer output security guardrail."""
3
4 def __init__(self):
5 self.filters = [
6 self.check_pii_leakage,
7 self.check_prompt_leakage,
8 self.check_harmful_content,
9 self.check_url_safety,
10 self.check_code_safety,
11 ]
12
13 def check_pii_leakage(self, output: str) -> tuple[bool, str]:
14 """Detect PII patterns: SSN, credit cards, phone numbers."""
15 pii_patterns = {
16 'ssn': r'\b\d{3}-\d{2}-\d{4}\b',
17 'credit_card': r'\b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b',
18 'phone': r'\b\+?1?[- ]?\(?\d{3}\)?[- ]?\d{3}[- ]?\d{4}\b',
19 }
20 for pii_type, pattern in pii_patterns.items():
21 if re.search(pattern, output):
22 return False, f"PII detected: {pii_type}"
23 return True, ""
24
25 def check_prompt_leakage(self, output: str) -> tuple[bool, str]:
26 """Detect if system prompt content is being leaked."""
27 system_prompt_markers = [
28 "you are a", "your instructions are",
29 "system prompt:", "your role is",
30 ]
31 output_lower = output.lower()
32 for marker in system_prompt_markers:
33 if marker in output_lower:
34 return False, f"Potential system prompt leakage: {marker}"
35 return True, ""
36
37 def check_url_safety(self, output: str) -> tuple[bool, str]:
38 """Validate URLs in output against allowlist."""
39 urls = re.findall(r'https?://[^\s<>"]+', output)
40 allowed_domains = {'example.com', 'docs.example.com'}
41 for url in urls:
42 domain = url.split('/')[2]
43 if domain not in allowed_domains:
44 return False, f"Unapproved URL domain: {domain}"
45 return True, ""
46
47 def run(self, output: str) -> tuple[str, list[str]]:
48 """Run all filters. Returns (safe_output, warnings)."""
49 warnings = []
50 for check in self.filters:
51 passed, message = check(output)
52 if not passed:
53 warnings.append(message)
54
55 if warnings:
56 return "[Response filtered for safety]", warnings
57 return output, []Guardrail frameworks like Guardrails AI, NeMo Guardrails (NVIDIA), and LLM Guard provide pre-built filter chains. For production applications, consider using these instead of building from scratch, then extend them with domain-specific rules.
- Guardrails AI — Open-source framework for adding structural and semantic validation to LLM outputs with Pydantic-like validators.
- NeMo Guardrails — NVIDIA's toolkit for adding programmable safety rails to LLM applications, including topical control and output moderation.
- LLM Guard — Focused on prompt and output scanning for sensitive data, toxicity, and injection patterns.
- Rebuff — Self-hardening prompt injection detector that learns from attempted attacks.
10 //Real-World Case Studies
Understanding real incidents helps illustrate why output security matters. Here are notable cases where insecure LLM output handling led to real-world impact.
These case studies share a common pattern: the application trusted LLM output and acted on it without validation. Whether the output was rendered as HTML, executed as a tool call, or copy-pasted into production code, the lack of output validation was the root cause.
11 //Monitoring & Observability for LLM Output
Output security isn't just about prevention — it's also about detection. Monitoring LLM output patterns helps you catch issues that slip through validation and identify emerging attack techniques.
- Log all outputs (safely) — Store LLM outputs for audit trails, but ensure logs don't contain unredacted PII. Use structured logging with metadata (model, timestamp, user ID, token count).
- Monitor output anomalies — Track metrics like average response length, language distribution, refusal rates, and moderation flag rates. Sudden changes may indicate prompt injection campaigns.
- Alert on filter triggers — When output guardrails redact or block content, generate alerts. A spike in filtered responses suggests an active attack or a model behavior change.
- Track hallucination rates — For applications with ground truth (e.g., RAG with source documents), measure how often the LLM's output contradicts or fabricates beyond the source material.
- Red team regularly — Conduct periodic red team exercises specifically targeting output security: try to extract system prompts, generate XSS payloads, leak PII, and bypass content filters.
1import logging
2from dataclasses import dataclass
3from datetime import datetime
4
5logger = logging.getLogger("llm_output_monitor")
6
7@dataclass
8class OutputMetrics:
9 timestamp: datetime
10 user_id: str
11 model: str
12 input_tokens: int
13 output_tokens: int
14 guardrail_triggered: bool
15 guardrail_reasons: list[str]
16 moderation_flagged: bool
17 response_time_ms: float
18
19def monitor_output(metrics: OutputMetrics):
20 """Log output metrics and alert on anomalies."""
21 logger.info(
22 "llm_output",
23 extra={
24 "user_id": metrics.user_id,
25 "model": metrics.model,
26 "output_tokens": metrics.output_tokens,
27 "guardrail_triggered": metrics.guardrail_triggered,
28 "moderation_flagged": metrics.moderation_flagged,
29 "response_time_ms": metrics.response_time_ms,
30 }
31 )
32
33 if metrics.guardrail_triggered:
34 alert_security_team(
35 f"Guardrail triggered for user {metrics.user_id}: "
36 f"{', '.join(metrics.guardrail_reasons)}"
37 )
38
39 if metrics.output_tokens > 5000:
40 alert_security_team(
41 f"Unusually long output ({metrics.output_tokens} tokens) "
42 f"for user {metrics.user_id} — possible prompt injection"
43 )12 //Conclusion & Code Review Checklist
LLM output security is not optional — it is a critical component of any application that integrates language models. The output is where the risk materializes: XSS payloads execute, data leaks reach users, hallucinated code enters production, and harmful content damages your brand.
LLM Output Security Code Review Checklist
| Category | Check | Priority |
|---|---|---|
| Rendering | Is LLM output HTML-encoded before rendering in web UIs? | Critical |
| Rendering | Is dangerouslySetInnerHTML / innerHTML used with LLM output? | Critical |
| Rendering | Are URLs in output validated before rendering as clickable links? | High |
| Data Leakage | Are secrets / API keys ever placed in system prompts? | Critical |
| Data Leakage | Is output scanned for PII patterns before returning to users? | High |
| Data Leakage | Is there system prompt leakage detection? | High |
| Execution | Is LLM output ever used in SQL queries, shell commands, or eval()? | Critical |
| Execution | Is LLM-generated code sandboxed before execution? | Critical |
| Content Safety | Is output checked via moderation API or content classifier? | High |
| Content Safety | Are domain-specific content policies enforced? | Medium |
| Structured Output | Is output validated against a schema (Pydantic/Zod)? | High |
| Structured Output | Is JSON mode enabled for structured responses? | Medium |
| Monitoring | Are LLM outputs logged securely for audit? | Medium |
| Monitoring | Are guardrail triggers generating alerts? | Medium |
| Monitoring | Are output length and pattern anomalies monitored? | Medium |