Human-Centered AI Privacy Design Framework

• Who uses the AI system?
• Who may be affected even if they never interact with it?
• Which people or groups are represented in the data?
• What relationship exists between the organization and the person?
• Can the person realistically refuse or avoid the AI system?
• Are any people particularly dependent on the organization or service?
• What would people reasonably expect the AI to know?
• What would they expect it to remember?
• What would they not expect it to infer, disclose or use?

An internal employee AI assistant may appear low risk because it is only used inside the company. But Human Context may reveal:

• employer and employee power imbalance;
• people mentioned in documents who never use the assistant;
• sensitive HR relationships;
• employees who cannot realistically opt out;
• expectations that private work conversations will not be used for monitoring.

• affected people and groups;
• important relationships;
• vulnerability and dependency considerations;
• privacy expectations that should influence design.

• Where does the data come from?
• Is it user-provided, internal, observed, scraped or third-party?
• What types of personal data are involved?
• Is sensitive data involved?
• Is the information accurate, current and relevant?
• What is each category of data used for?
• Is data reused for additional purposes?
• Is it shared with vendors, model providers, APIs or other systems?
• What is stored?
• Where is it stored?
• How long is it retained?
• Does deletion propagate to logs, memory, backups and downstream systems?

A RAG assistant may connect to:

• public internal policies;
• confidential project files;
• HR documents;
• user prompts;
• generated responses;
• logs;
• vector databases;
• external model providers.

Those data sources and uses cannot automatically be treated the same way.

• AI-specific data map;
• sensitive data categories;
• data-use boundaries;
• sharing restrictions;
• retention expectations;
• data-quality requirements.

• Is personal data used for training or fine-tuning?
• What information can the system retrieve?
• What can the system infer?
• Can it generate new personal information?
• Does it remember information across interactions?
• Is memory temporary or persistent?
• Does the system make or influence decisions about people?
• Can it trigger actions outside the AI interface?
• Can it communicate with other systems or agents?
• Can its capabilities change over time?

A chatbot becomes materially different once it can:

• access email;
• remember preferences;
• infer intent;
• make decisions;
• send messages;
• modify records;
• perform external actions.

• privacy-relevant AI capabilities;
• important processing behaviours;
• memory boundaries;
• inference boundaries;
• decision and action boundaries.

• Is the AI collecting or accessing more data than necessary?
• Could information be exposed to the wrong person or system?
• Could supposedly anonymous information become identifiable?
• Could the system enable monitoring or tracking?
• Could it infer information a person never provided?
• Could data be reused for an unexpected purpose?
• Could the system generate false or misleading personal information?
• Could information remain available longer than expected?
• Can people understand or influence what happens to their information?
• Could personal data or inferred vulnerabilities be used to influence behaviour?

An AI system may have legitimate access to employee data but still create privacy risk if it can infer:

• pregnancy;
• health problems;
• intention to resign;
• financial difficulty;
• personal relationships.

The privacy issue is not only what data the system sees. It is also what the AI can derive from it.

• priority privacy risk scenarios;
• areas requiring mitigation;
• risks requiring engineering controls;
• risks requiring governance action.

• Can the system operate with less data?
• Can access be narrowed?
• How is personal data protected?
• Could privacy-enhancing technologies reduce exposure?
• Can users understand what the AI knows, remembers or shares?
• What controls should users have?
• What should the AI not access?
• What should it not infer?
• What should it never disclose automatically?
• Which actions should require confirmation?
• How will privacy controls be tested?
• What evidence shows that those controls work?

For an AI agent connected to email and calendar, privacy engineering may include:

• scoped permissions;
• minimized access;
• memory restrictions;
• privacy guardrails;
• confirmation before sensitive external actions;
• leakage testing;
• deletion controls.

• privacy requirements;
• technical safeguards;
• privacy UX requirements;
• privacy guardrails;
• test cases;
• validation evidence.

• What privacy principles and internal policies apply?
• Which legal and regulatory requirements are relevant?
• Who owns privacy decisions for the AI system?
• Who approves high-risk uses or exceptions?
• Has the AI use case undergone an appropriate privacy assessment?
• How can individuals exercise applicable privacy rights?
• What information should be communicated to users?
• Where are consent, choice or opt-out relevant?
• Which external providers are involved?
• How are third-party privacy obligations managed?
• How will privacy controls be monitored after launch?
• What evidence demonstrates effectiveness?
• What happens if the AI system creates a privacy incident?

A privacy-safe AI system at launch may become unsafe later if:

• a new model provider is introduced;
• persistent memory is enabled;
• new datasets are connected;
• an agent gains new permissions;
• logs are reused for another purpose;
• model behaviour changes.

• governance requirements;
• ownership and accountability;
• privacy assessment needs;
• rights and transparency requirements;
• third-party requirements;
• monitoring and assurance expectations;
• incident responsibilities.