Output handling (LLM02)

LLM02 is the risk that downstream code trusts an LLM output the same way it would trust output from a vetted internal service. It does not. Every SQL statement, every snippet of HTML, every command-line argument an LLM produces is content of unknown provenance. Any code path that executes that content without validation is an injection vulnerability waiting for the right prompt. The rule is uniform across Bizzi. LLM output is untrusted input, and is handled as such.

The principle

Never trust LLM output absolutely. Especially when it generates code or SQL.

LLM02 covers all the classic injection categories at once. SQL injection through text-to-SQL, XSS through markdown rendering, command injection through shell tools, prototype pollution through generated JSON. The class is broad, and the defense is correspondingly layered.

Principle: The output handling rule

Every byte produced by an LLM is treated as untrusted input. SQL goes through a read-only sandbox. HTML goes through escaping. Commands go through human approval. There is no fast path that skips validation.

Text-to-SQL. The central case

The AI assistant lets a user ask an analytical question. The agent generates SQL. The SQL runs on the OLAP database. This is the most likely place for a high-impact injection in our platform, so the defense is explicit and end-to-end.

1. The agent generates SQL against MCP (Pillar IV §12), which exposes only the tables and columns the feature is authorised to use.
2. MCP parses the SQL and validates. No DDL or DML (DROP, UPDATE, DELETE), a mandatory WHERE tenant_id = ? clause injected at the MCP layer, no UNION against tables outside the whitelist, and an automatic LIMIT when one is missing.
3. The validated SQL executes against a read-only OLAP user with the minimum privileges required.
4. The result is returned to the agent for formatting.

This is Defense-in-depth. If a prompt-injected agent generates malicious SQL, the MCP parser rejects it before it touches the database. If the parser is bypassed, the read-only user blocks anything destructive. If the user’s tenant filter is missing, MCP rejects the query.

Code generation

Bizzi currently exposes limited AI code generation. Where it does:

• Generated code is scanned by static analysis before any execution path reaches it.
• Execution happens in a sandboxed environment with no network egress and no privileged file system access.
• Privileged operations require explicit human approval. The HITL pattern from Pillar III §3.

The output validation pipeline

Every LLM response passes through a five-step validator before it is served.

1. Type check. Output conforms to the expected schema (JSON shape, structured fields).
2. PII scan. No leakage of redacted values (see §7).
3. Cross-tenant scan. No references to data outside the current tenant.
4. Citation validation. Any citation points to a chunk or document that exists in the corpus.
5. Tone and safety check. No content that violates platform policy.

A response failing any step is not served to the user. It is logged for investigation and the user receives a graceful degradation.

Render-time XSS prevention

When AI output is rendered in the UI, the rendering layer treats it as untrusted markdown.

• HTML escaping is on by default, even though we do not expect HTML in responses.
• Markdown is sanitised through a fixed allowlist of nodes.
• dangerouslySetInnerHTML is never used for AI content. Code review rejects it.
• A strict Content Security Policy blocks inline scripts from executing.

Email and notification content

When AI participates in outbound email or in-app notifications, three rules apply. Content passes the same validation pipeline as in-app responses. The template structure is fixed and the AI only fills placeholders rather than generating raw content. Bulk sends require admin preview before they go out.

Standards mapping

Standards Mapping

OWASP Top 10 for LLMs. LLM02 (Insecure Output Handling). OWASP Top 10 for Web Applications. A03 (Injection). ISO/IEC 42001. A.8.2 (Performance). ISO/IEC 27001. A.14.2 (Security in development).