AI Data Security: What Happens to Your Business Data?

Whether an AI service uses your business data to improve its models depends on the product, plan, settings and contract—not simply on whether the logo says Microsoft, OpenAI, Google or Anthropic.

Microsoft says prompts, responses and Microsoft Graph data used by Microsoft 365 Copilot are not used to train its foundation models. OpenAI and Anthropic make similar default commitments for their business products. Consumer AI products can work differently: activity settings, feedback and temporary-chat modes can change what is retained or used.

The practical rule is simple: never approve “AI” as one category. Approve a named service and plan for named data, under named controls.

Fact-checked against Microsoft Learn and the vendors’ current privacy documentation on 7 August 2026.

The short answer

• A paid account is not automatically private, and a free account is not automatically unsafe.
• Microsoft 365 Copilot inherits Microsoft 365 identity, permissions, sensitivity labels, retention, audit and admin controls—but it can still surface information a user already has permission to access.
• Prompts and responses may be stored for history, compliance, audit or abuse monitoring even when they are not used to train a foundation model.
• Connectors, agents, plug-ins and uploaded files widen the data path. Their permissions and terms need a separate review.
• Your strongest control is data minimisation: give the service only what the task needs.

Start by identifying the exact product

“We use ChatGPT” or “we use Copilot” is not enough for a security decision. Record:

1. the exact product and plan;
2. whether the account is personal or organisation-managed;
3. the tenant or workspace owner;
4. the data classes allowed;
5. retention and model-training settings;
6. connected apps, agents and data sources;
7. region or processing-location commitments; and
8. the contract and privacy terms that apply.

This matters because consumer and business versions from the same vendor can have materially different defaults. It also stops a policy written for one approved enterprise workspace being treated as permission to paste the same data into a personal account.

What Microsoft 365 Copilot does with data

Microsoft’s enterprise data-protection documentation says prompts, responses and data accessed through Microsoft Graph are not used to train foundation models. Data is encrypted at rest and in transit, and Microsoft 365’s existing tenant boundary, identity, permissions, sensitivity labels, retention, audit and compliance controls continue to apply.

That is a meaningful enterprise commitment. It is not a promise that no data is stored or that Copilot can never reveal sensitive information.

Permissions still decide what Copilot can find

Copilot works within the signed-in user’s existing permissions. If someone can already open an overshared SharePoint site, document or email, Copilot can make that authorised-but-inappropriate access easier to use.

Before broad deployment, fix oversharing, stale access and weak information architecture. Our Microsoft 365 Search guide explains the same security-trimming principle from the search side.

Prompts and responses can be compliance records

Microsoft documents prompts and responses as data that may be logged and stored for audit and eDiscovery. Retention policies can apply. “Not used for training” therefore does not mean “never retained.”

Decide whether prompts may contain personal data, privileged advice, security details or regulated records. Then configure retention and access accordingly rather than relying on users to remember what a chat window stores.

Agents and connectors create new boundaries

An agent can call other services, use connectors or process data outside the core Microsoft 365 experience. Microsoft explicitly tells administrators to review the privacy statement and terms for each agent.

For every agent, record:

• the data sources it can read;
• the actions it can take;
• whose identity and permissions it uses;
• which service receives the prompt or result;
• what that service retains; and
• who reviews permissions when the agent changes.

An enterprise-protected chat does not magically extend its contract to an unrelated service invoked by a connector.

Azure OpenAI and Azure AI Foundry need their own review

Microsoft says prompts, completions, embeddings and training data submitted to Azure Direct Models are not made available to model providers or other customers, and are not used to train foundation models without permission.

But application design still matters:

• base inference models are stateless, while stateful features can store conversation history or files;
• Global and DataZone deployments can process prompts in different geographies within the documented boundary;
• abuse monitoring can retain flagged prompts and completions and permit authorised human review, unless a customer has been approved for modified abuse monitoring; and
• your own application may log requests independently of the model service.

Do not advertise an Azure OpenAI solution as “zero retention” unless the exact feature, deployment, monitoring approval and application logging have all been verified.

How major AI products differ

This is a policy-level comparison, not a complete contract review.

Service context	Current high-level position	What to verify
Microsoft 365 Copilot and Copilot Chat with enterprise data protection	Prompts, responses and Graph data are not used to train foundation models	Licence/account context, permissions, retention, agents and connectors
Azure OpenAI / Azure AI Foundry	Customer inputs and outputs are not used to train foundation models without permission	Stateful features, region, abuse monitoring and application logs
OpenAI business products and API	Business data is not used for model training by default	Workspace controls, retention, connected apps and any opt-in
Individual ChatGPT	Content may be used to improve models unless the user opts out; Temporary Chat is excluded from training	“Improve the model” setting, history, feedback and account ownership
Anthropic commercial products	Commercial inputs and outputs are not used to train generative models by default	Feedback/opt-in, retention and connected services
Consumer Gemini Apps	Activity and feedback settings affect retention, review and model improvement	Keep Activity, Temporary Chat, human review and account type

Policies change. Link the approval record to the applicable vendor page and review it on a schedule.

A practical risk assessment

1. Classify the data before choosing the tool

Use categories people can apply consistently, for example:

• Public — approved for public release.
• Internal — routine business information with limited harm if exposed.
• Confidential — customer, employee, commercial or security information.
• Restricted — regulated, privileged, highly sensitive or contractually controlled data.

Then define which approved product can receive each class and under what conditions. “Do not paste sensitive data into AI” is too vague to enforce.

2. Map the complete data path

Follow the information from input to output:

user -> AI product -> model endpoint -> storage/logging -> connected service -> output destination

Include browser extensions, meeting bots, custom agents and automation. The model provider is only one part of the path.

3. Check identity and administration

Ask:

• Must users sign in with managed work accounts?
• Can administrators disable model-improvement settings?
• Are sharing and connector permissions controlled?
• Can the organisation audit activity and remove access?
• What happens when an employee leaves?

A strong contract is undermined if staff can quietly use personal workspaces with no administrative control.

4. Set a minimum-data rule

Remove names, account numbers, secrets and unnecessary context. Use representative or synthetic data for prompt experiments. Retrieve only the records needed for the current answer rather than sending an entire library.

For custom integrations, grant the narrowest supported permissions. The difference between delegated access to one user’s mailbox and application-wide access to every mailbox is enormous. Our Claude and Microsoft 365 email integration guide shows why permission design is part of the product decision.

5. Plan for bad output as well as data loss

Security is not only confidentiality. AI output can be inaccurate, discriminatory, insecure or inappropriately authoritative.

Define where a human must verify the result, which sources must be cited, what actions require confirmation and how a harmful answer is reported. A model that keeps data private can still produce a dangerous recommendation.

Red flags that should stop a rollout

Pause when:

• nobody can identify the exact product plan or account owner;
• the sales claim says “your data is private” but provides no applicable terms;
• staff are asked to use personal accounts for business data;
• an agent requests broad permissions without a task-level reason;
• “not used for training” is being presented as “not stored”;
• the organisation cannot say where prompts, outputs or files are logged;
• restricted data is needed for a proof of concept that synthetic data could test; or
• no owner is responsible for reviewing service changes.

A safe adoption checklist

Before enabling a service:

• Name the product, plan, tenant/workspace and owner.
• Read the applicable enterprise or consumer privacy terms.
• Record the model-training, history and retention settings.
• Classify the data the use case needs.
• Review permissions, oversharing, agents and connectors.
• Confirm processing location and contractual requirements.
• Test with synthetic or low-risk data first.
• Define human review and prohibited actions.
• Enable audit, incident and offboarding processes.
• Set a date to review the decision again.

Frequently asked questions

Does Microsoft use Microsoft 365 Copilot prompts to train its foundation models?

Microsoft says it does not use prompts, responses or Microsoft Graph data from Microsoft 365 Copilot to train foundation models. Prompts and responses can still be retained under Microsoft 365 compliance controls.

Is business data safe in a paid AI account?

Payment alone proves very little. Check the exact business plan, contract, settings, retention, administrators, permissions and connected services.

Is turning off chat history the same as zero retention?

No. A history control can change what the user sees or what is used for model improvement, while a provider may still retain data temporarily for safety, compliance or service operation. Read the product-specific documentation.

Keep the boundary clear

This guide explains documented product data-handling commitments and a practical review method. It is not legal advice, a certification of any service or permission to process a particular regulated dataset. Your contract, configuration, jurisdiction and use case decide the final position.

Join the Microsoft Copilot Adopters Space for practical adoption guidance that connects Copilot’s features to permissions, governance and real work.

Sources

• Microsoft: Enterprise data protection in Microsoft 365 Copilot and Microsoft 365 Copilot Chat
• Microsoft: Data, privacy and security for Azure Direct Models in Azure AI Foundry
• OpenAI: How your data is used to improve model performance
• OpenAI: Enterprise privacy
• Anthropic: Is my data used for model training?
• Google: Gemini Apps Privacy Hub