Security, Privacy and Residency

The idea in one minute#

Security review is where AI projects wait longest, and the wait is usually caused by a design that cannot answer three questions. Where does the data go? — which provider, which region, kept for how long, used for what. What can the system see? — whose permissions apply when it reads. What can the system do? — which actions it can take, and what stops a wrong one.

AI adds one genuinely new problem to ordinary application security: a model cannot reliably tell instructions from data. Text inside a document, an email or a web page can steer it. This is prompt injection, and no filter removes it completely. The architect’s answer is structural: assume the model can be fooled, and design so that a fooled model cannot do much harm.

You do not need to be a security engineer. You need to arrive at the review with a data-flow diagram, a short list of controls that do not depend on the model behaving, and honest answers to twelve questions.

A picture#

flowchart LR
  subgraph EXT["Untrusted"]
    direction TB
    U[":i-user: <b>User input</b>"]
    DOC[":i-file-text: <b>Documents, email, web</b><br/><small>may contain instructions</small>"]
  end
  subgraph ORG["Your boundary"]
    direction TB
    APP[":i-server: <b>Application</b><br/><small>authenticates the user</small>"]
    RET[":i-search: <b>Retrieval</b><br/><small>filters by the user's permissions</small>"]
    POL[":opa: <b>Policy and approval</b><br/><small>code, not prompt</small>"]
    TOOLS[":i-wrench: <b>Tools</b><br/><small>least privilege</small>"]
    LOG[(":i-scroll-text: <b>Audit log</b>")]
  end
  subgraph PROV["Model provider"]
    M[":i-brain: <b>Model</b><br/><small>region, retention, no training</small>"]
  end
  U --> APP
  DOC --> RET
  APP --> RET
  RET --> M
  APP --> M
  M -->|"proposed action"| POL
  POL -->|"allowed"| TOOLS
  POL -.->|"needs a person"| H[":i-hand: <b>Approval</b>"]
  APP -.-> LOG
  POL -.-> LOG
  class U,DOC warn
  class APP,RET compute
  class POL queue
  class TOOLS io
  class LOG memory
  class M neutral
  class H warn

How it really works#

Question one: where does the data go?#

Draw the path of one request and write beside every hop what data crosses it. Then answer, per provider:

The last row is the one teams forget. A trace store full of customer prompts is a sensitive database you created yourself. It needs access control, a retention period and a deletion process like any other.

Data residency and sovereignty#

Residency is where data is stored and processed. Sovereignty is whose law can compel access to it. They are different: data held in one country by a company headquartered in another may still be reachable under the second country’s law.

Each step down the table narrows model choice and raises cost. Establish the real requirement — from legal, in writing — before designing, because teams routinely assume a stricter rule than the one that applies.

Privacy#

When personal data is involved, a handful of principles do most of the work.

• Purpose — is using this data for this feature compatible with why it was collected?
• Minimisation — send the model only what the task needs. Strip or mask identifiers before the call where the task allows.
• Retention — set and enforce how long prompts, outputs and logs live.
• Rights — can you find and delete one person’s data across indexes, logs and caches?
• Automated decisions — if the output affects a person significantly, a human review and a way to contest are commonly required.
• Assessment — a data-protection impact assessment before launch for anything high-risk. Start it at design time; it shapes the design.

Involve the privacy function in the first design review. Their answer to “may we?” decides the architecture more often than any technical constraint.

Question two: what can it see?#

The rule is simple and absolute: the system must not show a user anything that user could not see directly. Three ways to enforce it, in order of preference.

Never rely on the prompt — “do not reveal confidential documents” — to enforce access. If the text reaches the context window, assume the user can get it out.

Question three: what can it do?#

Every tool is a capability granted to whatever text manages to steer the model. Apply ordinary least privilege, strictly.

• Read before write. Start with read-only tools. Add writes one at a time.
• Narrow tools. A specific action with validated parameters, not a general query or shell.
• Act as the user, within limits. The agent’s reach is the intersection of what the user may do and what the agent is allowed to do.
• Approve the irreversible. Payments, deletions, external messages and permission changes wait for a person.
• Budgets and an off switch. Enforced by code outside the model.
• Log every action with who asked, what the model proposed, and what was executed.

Prompt injection, for architects#

A model reads its instructions and its data as one stream of text. If an attacker can place text where the model will read it — a web page, an uploaded PDF, an email, a support ticket — that text can act as an instruction. Indirect injection, arriving through content rather than from the user, is the dangerous form because the user never sees it.

Filters and guardrail models catch the obvious attempts and should be used. They do not make the problem go away. The design position that holds is: assume injection succeeds, and limit the consequences.

A system is in real danger when it combines three things at once:

access to private data   +   exposure to untrusted content   +   a way to send data out

Remove any one and the worst outcomes are gone. This gives a practical review technique: for each workload, check which of the three it has, and break the combination — no external sending for an agent that reads untrusted email; no private data for an agent that browses the web; human approval on every outbound action where all three must coexist.

The attack mechanics and the defensive architectures are covered in depth in Prompt Injection and Architectural Defenses.

Controls that do not depend on the model#

Sort every control in the design into one of two columns.

Probabilistic controls reduce how often something goes wrong. Only deterministic controls put a ceiling on how bad it can be. A design whose safety case rests on the right-hand column will not pass a competent review, and should not.

The twelve questions security will ask#

Arrive with written answers.

1. What data is sent to the model, and what is its classification?
2. Which provider and region process it? Is it retained or used for training?
3. Who can use the system, and how are they authenticated?
4. How is each user limited to data they are entitled to?
5. What untrusted content can reach the model?
6. What tools can the model call, with whose permissions?
7. Which actions require human approval?
8. What stops data leaving — in outputs, in tool calls, in rendered links or images?
9. What is logged, where, for how long, and who can read the logs?
10. How are prompts, models and dependencies sourced and updated?
11. How was it tested — including adversarial testing?
12. How is misuse or an incident detected, and how is the system switched off?

If you can answer all twelve on two pages with a data-flow diagram, the review takes days. If you cannot, it takes months — correctly.

Frameworks to recognise#

Frameworks and Maps explains when to use which.

Remember this#

• Three questions: where does the data go, what can it see, what can it do.
• Your own logs, traces and caches hold the same sensitive data as the source.
• Enforce access in retrieval, by the user’s identity. Never in the prompt.
• Assume prompt injection succeeds. Break the combination of private data, untrusted content and a way out.
• Safety rests on deterministic controls. Guardrail models reduce frequency, not severity.

Try it#

1. For a design of your own, draw the data flow and label every hop with the data crossing it, the region, and the retention period.
2. Sort its controls into the deterministic and probabilistic columns. Is there any serious harm that only the right-hand column prevents?
3. Check the design for the dangerous combination of three. If all three are present, which one can be removed with the least loss of function?

Check yourself#

1. What is the difference between data residency and data sovereignty?
2. Why must document permissions be enforced at retrieval rather than in the prompt?
3. What three capabilities together make an AI system dangerous, and what follows from that?
4. Why can a guardrail model not be the basis of a safety case?