Privacy-Preserving AI: When Compliance Isn't Optional | Security Medic Consulting

AI is already touching personal data in most organizations. The question is whether anyone designed it that way — or whether your staff figured it out on their own, one pasted document at a time.

§01

The gap nobody planned for

A practice owner at a mid-sized behavioral health clinic is reviewing her workflows. She notices something unusual: patient intake forms are being processed faster than ever, with cleaner summaries than her front desk staff typically produces. She asks around. One of the medical assistants explains, cheerfully, that they’ve been pasting intake forms into ChatGPT to get quick summaries before the provider walks in. It saves ten minutes per patient. Everybody thinks it’s great.

Nobody told them to do it. Nobody told them not to. The EMR vendor was never asked. The privacy officer didn’t know. There is no Business Associate Agreement with OpenAI. There is no data processing addendum. There is no training record that mentions AI tools. And somewhere in a cloud system the clinic has no relationship with, protected health information for hundreds of patients has been processed by a model the clinic cannot audit, control, or compel to delete data.

This is not a hypothetical. A Wolters Kluwer 2026 healthcare AI survey found that 17% of healthcare workers admit to using unauthorized AI tools to process patient information.[1] The number who do it without realizing it’s unauthorized is almost certainly higher.

The same scenario plays out in schools, financial service firms, government agencies, and small businesses across every sector. Employees are resourceful. AI tools are fast and free and genuinely useful. Compliance frameworks have not kept pace. The gap between what your staff is already doing with AI and what your privacy governance actually covers is where the regulatory exposure lives.

This guide is built to help you close that gap. Not by adding fear, but by giving you specifics. By the end, you will understand what these tools actually do with your data, which laws apply to your organization, what technical options exist to reduce exposure, and what to do Monday morning. That is a manageable set of problems.

§02

What happens to your data when you use AI

Before worrying about compliance, it helps to understand the mechanics. When an employee types something into an AI chat interface, that text travels to a server, is processed by a large language model, and a response is returned. The question that matters for privacy is: what happens to that input after the response is sent?

The answer varies by platform and plan — and the differences are significant. Here is how the major platforms handle it as of early 2026. Read this table as your guide to the fine print, so you do not have to guess:

Platform / Tier	Trains on Your Inputs?	BAA / DPA Available?	Data Residency Controls?
ChatGPT Free / Plus	Yes (opt-out possible in settings)	No	No
ChatGPT Team	No (off by default)	No	Limited
OpenAI API / Enterprise	No	Yes (HIPAA-eligible)	Yes
Claude (Anthropic) — All tiers	No (confirmed Sep 28, 2025)	Available for enterprise	Yes (enterprise)
Microsoft Copilot (M365)	No	Yes (within M365 tenant)	Yes (tenant-bounded)
Google Gemini (Workspace Enterprise)	No	Yes (HIPAA BAA available)	Yes
Google Gemini (Free / Personal)	Yes (human reviewers may see data)	No	No

The most important thing to understand from this table: The platform your employees use for free at home is almost never the same configuration as the enterprise version with privacy protections. A ChatGPT Plus account and an OpenAI Enterprise account are materially different products from a compliance perspective — despite sharing a name and interface. The plan tier matters as much as the vendor name. Verify before data flows.

Anthropic confirmed in its September 28, 2025 privacy policy update that Claude does not train on user inputs by default across any tier — including the free web interface.[2] Microsoft Copilot, when deployed within a Microsoft 365 tenant, processes data within that tenant boundary and does not use customer data to train foundation models.[3] Both are meaningful advances over earlier consumer AI tools, but neither replaces the due diligence of reviewing a vendor’s data processing agreement before putting sensitive information into the model.

§03

The regulatory intersection — where AI meets privacy law

Most organizations are subject to more than one privacy framework at the same time. A school district may face FERPA, a state student privacy law, and COPPA. A medical practice may face HIPAA, a state data breach law, and CCPA if it serves California residents. Understanding which framework applies to your organization — and where AI creates new exposure under each — is the first step toward a manageable compliance posture.

HIPAA and the Minimum Necessary Principle

HIPAA’s minimum necessary standard requires covered entities and their business associates to limit uses and disclosures of protected health information to the minimum amount needed to accomplish the intended purpose.[4] AI creates a specific problem here: general-purpose AI tools, by design, process everything you give them. There is no concept of “I only needed the chief complaint — don’t read the medication history.”

When a staff member pastes a complete patient intake form into an AI tool to extract one piece of information, the model sees — and potentially processes for training — information that was not necessary for the task. That is a minimum necessary violation regardless of whether the AI “does anything” with the excess data. The disclosure itself is the violation.

For covered entities, any AI vendor that receives, transmits, creates, or maintains PHI on your behalf is a business associate and requires a signed Business Associate Agreement before any data flows. No BAA means no compliant use.

FERPA and AI in Education

The Family Educational Rights and Privacy Act restricts the disclosure of student education records without consent. AI tutoring platforms, chatbots used in classroom settings, and AI-powered grading tools may qualify as “school officials” with legitimate educational interest under FERPA — but only if the vendor has a written agreement specifying that it will not redisclose the information and will not use it for any purpose outside the contracted service.[5]

Most consumer AI tools have terms of service that explicitly reserve the right to use inputs for service improvement, commonly interpreted as training data. A school that deploys a free consumer AI tool without a compliant data processing agreement is almost certainly in FERPA violation from day one.

CCPA/CPRA and the Right to Delete

California’s Consumer Privacy Rights Act gives consumers the right to request deletion of their personal information from businesses that collect it. When an AI vendor trains its model on customer data, deletion becomes legally complicated and technically fraught: once personal information has been incorporated into model weights, “deleting” it is not equivalent to removing a database record. The model has, in some sense, learned from that data. Regulators have not fully resolved how deletion obligations apply to trained model weights, but the exposure is real and the regulatory direction is not favorable to organizations that permitted training in the first place.[6]

GDPR Article 22 and Automated Decisions

The General Data Protection Regulation imposes specific obligations when decisions are made “solely based on automated processing” and produce legal or similarly significant effects on individuals.[7] AI-generated hiring recommendations, credit decisions, medical triage outputs, and benefits determinations all potentially trigger Article 22. Affected individuals have the right to human review, to contest the decision, and to receive a meaningful explanation of the logic involved. Organizations subject to GDPR that use AI to inform significant decisions need a documented human-in-the-loop process and a mechanism for individuals to exercise those rights.

NY SHIELD Act — Updated March 21, 2026

New York’s SHIELD Act was amended effective March 21, 2026 to expand the definition of private information to explicitly include medical information — bringing New York healthcare-adjacent organizations under the SHIELD Act’s breach notification and reasonable safeguards requirements in addition to any applicable federal obligations.[8] Organizations that process medical information for New York residents through AI tools must now ensure their security program accounts for medical data specifically.

The State Privacy Law Wave

Texas’s Data Privacy and Security Act (TDPSA), Virginia’s CDPA, Colorado’s CPA, and laws in more than a dozen additional states have created a patchwork of obligations that many SMBs have not fully mapped. Most share a common thread relevant to AI: they require data minimization, purpose limitation, and provide deletion and correction rights that become complicated when AI training is involved.

HIPAA

Healthcare covered entities and business associates

AI concern: PHI in AI inputs, BAA requirement, minimum necessary rule

FERPA

Schools and districts receiving federal funding

AI concern: Student records in AI tools, vendor agreements, redisclosure prohibition

CCPA/CPRA

Businesses serving California residents above threshold

AI concern: Right to delete, AI training data, automated profiling opt-out

GDPR

Any org processing EU resident data

AI concern: Article 22 automated decisions, DPA required, data transfer adequacy

NY SHIELD Act

Orgs with NY resident data (amended Mar 21, 2026)

AI concern: Medical information now covered, reasonable safeguards required

TX TDPSA + State Laws

Varies by state threshold and business activity

AI concern: Data minimization, purpose limitation, consumer rights to correct and delete

§04

Privacy-preserving techniques — explained for non-engineers

Computer scientists and engineers have been working on this problem for years, and real solutions exist. You do not need to implement them yourself — you need to know they exist so you can ask the right questions of vendors and understand what good architecture actually looks like. Here is how smart engineers solved the most common privacy challenges in AI:

Differential Privacy — Adding Noise to Protect Individuals

Imagine a doctor who wants to understand what medications a community uses most often. She could ask every patient directly — but that creates privacy problems. Instead, picture a system where each patient flips a coin before answering: heads means answer truthfully, tails means give a random answer. The doctor can’t tell which individual answers are real, but across thousands of responses, the statistics become reliable enough to identify patterns. Individual privacy is protected. The aggregate insight survives.

That is the intuition behind differential privacy. The technique adds carefully calibrated mathematical noise to data so that the presence or absence of any single individual’s information doesn’t meaningfully change the output — while keeping overall statistical patterns intact. Apple uses it to learn which words people type most often without ever seeing individual keystrokes.[9] Google uses it in Chrome. It is a mature technique with a solid mathematical foundation, and it is already deployed at scale in regulated industries. When a vendor tells you they use differential privacy, this is what they mean.

Federated Learning — Training Without Moving Data

Normally, training an AI model means centralizing data in one place. Federated learning flips this. Instead of sending patient data to a central server, the model is sent to where the data already lives. Each hospital trains a local version of the model on its own patients. Then only the model updates — the mathematical changes, not the underlying data — are sent back and combined. The central system learns from all the hospitals without ever receiving a single patient record.

Google Health has used federated learning to develop AI systems for detecting medical conditions, collaborating across hospital systems without centralizing patient records.[10] For organizations handling highly sensitive data who want to participate in collaborative AI development, federated learning is the architecture that makes it possible. If a vendor says they train models collaboratively, ask whether they use federated learning — and whether your raw data ever leaves your environment.

Synthetic Data — Fake Data That Behaves Like Real Data

A common need in AI development and testing is a realistic dataset. The obvious source — real customer or patient records — creates compliance problems. Synthetic data is the alternative: statistically generated records that mirror the structure, distributions, and relationships in real data, without corresponding to any actual person.

Think of it as a detailed replica of a painting used for exhibition when the original is too fragile to display. Close enough for almost every practical purpose, but the original is never at risk. Healthcare AI developers increasingly use synthetic patient data for model training without putting actual PHI in jeopardy. Organizations subject to HIPAA, FERPA, or GDPR can use synthetic data for testing, development, and vendor demonstrations without triggering data handling obligations. If a vendor wants to run a proof of concept with your data, ask whether they can use synthetic data instead.

On-Premises / Private Deployment — Data Never Leaves Your Network

The cloud-centric model of AI — send data out, receive results back — is not the only option. Organizations with stringent data handling requirements increasingly deploy AI models locally, on their own servers, inside their own network perimeter. The model runs on infrastructure they control. Data never crosses a network boundary they don’t own. There is no vendor to sign a BAA with, because the data never reaches a vendor.

This approach carries real costs: local deployment requires more IT infrastructure, technical expertise to maintain, and the models available for local deployment are generally smaller and less capable than frontier cloud models. For certain categories of data — highly sensitive patient records, classified information, financial data under specific regulatory regimes — the tradeoff is worth it. Open-source models like Meta’s Llama series are specifically designed for this kind of deployment.

Data Masking and Tokenization — Substituting Before Sending

If you need to send data to a cloud AI service but that data contains sensitive identifiers, masking and tokenization offer a practical middle path. Before the data leaves your environment, a local system automatically replaces sensitive values with tokens or placeholders. “Patient John Smith, DOB 04/15/1962, SSN 555-12-3456” becomes “Patient [ID:7834], DOB [DATE-A], SSN [REDACTED].” The AI receives a structurally complete document it can process for meaning, but the identifying information never reaches the vendor’s servers.

When the response comes back, a local system can re-substitute real values where needed. This pattern is increasingly supported by AI middleware products built for regulated industries. It is not foolproof — models can sometimes infer identity from context — but it significantly reduces the exposure associated with cloud AI processing of sensitive documents.

§05

When it went wrong — real privacy incidents involving AI (2024–2026)

Regulatory bodies have been paying close attention. The following are verified incidents that illustrate what happens when AI governance is absent or insufficient. These are not meant to frighten — they are meant to show you exactly what to avoid, with specifics.

Healthcare AI Scribes — Widespread BAA Failures (2025–2026): A 2026 Wolters Kluwer survey found that 17% of healthcare workers admitted to using AI tools to process patient information without organizational authorization.[1] Multiple documented cases involve healthcare organizations deploying AI scribes without verifying BAA status with the underlying AI infrastructure vendor. The scribe vendor may have a BAA in place, but if they are themselves using a third-party AI model without their own BAA, the compliance chain breaks.

TikTok — €530M GDPR Fine (May 2025): Ireland’s Data Protection Commission fined TikTok €530 million in May 2025 for unlawful data transfers to China, including data processed by AI systems, finding that TikTok failed to verify that European user data received protection equivalent to GDPR standards when transferred to Chinese servers.[11] This is among the largest GDPR penalties related to AI-adjacent data processing to date.

Deer Oaks Behavioral Health — $225K OCR Settlement (July 2025): The HHS Office for Civil Rights settled with Deer Oaks Behavioral Health Services for $225,000 in July 2025 following an investigation into inadequate access controls and risk analysis failures.[12] The settlement illustrates OCR’s continued enforcement posture around access control and risk management — the exact governance gaps that AI adoption tends to widen when organizations lack mature programs.

Cisco 2025 Data Privacy Benchmark — Industry-Wide Concern: Cisco’s 2025 Data Privacy Benchmark Study found that 64% of organizations are actively limiting the data they share with AI tools due to privacy concerns, and that only 46% believe they can adequately protect personal data in AI systems today.[13] The gap between AI adoption speed and AI privacy governance is not a perception problem — it is widely documented across the industry.

17%

Healthcare workers admit unauthorized AI use with patient data (Wolters Kluwer, 2026)

€530M

TikTok GDPR fine for AI-adjacent data transfer failures (May 2025)

64%

Organizations limiting data shared with AI due to privacy concerns (Cisco, 2025)

46%

Organizations that believe they can adequately protect personal data in AI today (Cisco, 2025)

§06

The 10-question checklist — before you deploy any AI tool

Whether you are evaluating a new AI platform or trying to get your arms around tools your staff has already adopted, these ten questions give you a structured way to assess what you are actually dealing with. Every “I don’t know” is a gap — and knowing where the gaps are is the first step to closing them. Ask these before you sign anything.

1. 1Where does my data go when I hit “submit”? Specifically: which servers, in what jurisdiction, under what company’s custody? Is there a privacy policy that addresses this, and does it match what the sales team told you?
2. 2Does the vendor train their model on my inputs? This is distinct from “storing” your data. Training means your inputs may influence the model’s future outputs. Get this in writing.
3. 3Is there a signed BAA, DPA, or equivalent agreement? For HIPAA-covered entities, no BAA means no compliant use — full stop. Verbal assurances do not count.
4. 4Can I delete my data from the vendor’s systems, and how? What is their documented deletion process? Does deletion extend to backups and training datasets? The right answer is a clear, verifiable process — not “we take privacy seriously.”
5. 5Does this tool see more data than it needs? Apply the minimum necessary principle: is there a way to give the AI only what it needs for the specific task, rather than complete records?
6. 6Who at my organization approved this tool? Is there a documented approval process for AI tools that includes IT, legal or compliance, and a privacy review? Or did this tool arrive organically, without anyone in authority knowing?
7. 7Is there a human review step before AI output reaches a patient, student, or client? For decisions with real consequences, GDPR Article 22 may require it. Regardless of legal obligation, human review reduces liability when AI produces an error.
8. 8What happens to the data if the vendor is acquired or shuts down? What do their terms say about data handling in a change of control? Is there a right to retrieve your data on termination?
9. 9Can I audit what the AI saw and produced? If a regulatory inquiry or litigation arises, can you reconstruct what information the AI tool received and what it returned? Is logging within your control?
10. 10Does my insurance policy cover AI-related incidents? Cyber insurance policies written before 2024 frequently did not contemplate AI-specific incident types. Review your policy language with this specifically in mind.

§07

Building privacy into AI from day one

The phrase “privacy by design” was coined by Ann Cavoukian in the 1990s and has since been incorporated into GDPR as a legal requirement for EU-subject organizations.[14] Its core principle is that privacy protections should be built into systems from the outset, not retrofitted after problems emerge. For organizations with 10 to 250 employees, the following four steps are practical, achievable, and actionable before the end of the month.

Step 1 — Take Inventory of What Is Already Running

Before you can govern AI use, you have to know what AI your organization actually uses. That means asking IT to pull a list of approved SaaS applications and checking each one for AI features that may have been added in recent product updates. It means asking your staff directly: what AI tools are people using day to day? The goal is not to punish anyone — most people using these tools are trying to do their jobs better. The goal is to create an accurate picture so you can make informed decisions about what stays, what gets upgraded to a compliant tier, and what gets turned off.

Step 2 — Classify Your Data Before Deploying AI Against It

Not all data carries the same risk. A simple three-tier classification — public, internal, restricted — applied to the data categories your organization handles creates a workable decision framework. Restricted data (PHI, student records, financial account information, attorney-client communications) requires a higher standard of AI governance before it touches any AI tool. A one-day working session with department heads mapping data categories to sensitivity tiers delivers 80% of the value. Formalize it in a short policy document and you have something you can actually point to.

Step 3 — Write an AI Acceptable Use Policy and Enforce It

An AI acceptable use policy tells your staff which tools are approved, for what categories of data, and what is off-limits without prior approval. Specificity matters. “Do not input patient information into unapproved AI tools” is specific and actionable. “Use AI responsibly” is not a policy — it is a hope. Include a simple list of approved tools by department, the data categories they are approved for, and a clear path for requesting access to tools not on the list.

Step 4 — Establish a Vendor Review Checkpoint for AI Tools

The single highest-leverage change most small organizations can make is inserting a mandatory checkpoint before any new AI tool is adopted — even a free one, especially a free one. The checkpoint works through the ten questions from the previous section and requires sign-off from at least one of: IT, legal or compliance, or your privacy officer equivalent. For a small organization, this can be an email thread with a two-day turnaround. What it prevents is the behavioral health clinic scenario at the top of this article: an entire category of sensitive data flowing into an unevaluated tool because it was fast, free, and helpful — and nobody stopped to ask what happened to the data on the other end.

“Privacy by design is not a constraint on AI adoption. It is the condition under which AI adoption can be sustained. Organizations that skip this step don’t avoid the regulatory exposure — they defer it, with interest.”

The regulatory frameworks described in this post — HIPAA, FERPA, CCPA, GDPR, the NY SHIELD Act — were written before modern generative AI existed. Regulators are actively working to close interpretive gaps, and enforcement activity in this space is accelerating. Organizations building privacy governance into their AI programs now are managing current risk and positioning themselves to absorb the next round of requirements without a crisis response.

Not sure where your AI governance stands?

Security Medic Consulting helps SMBs map their AI exposure against applicable privacy frameworks and build governance programs that are proportionate, documented, and defensible. No enterprise budget required.

Start the Conversation

References

1. Wolters Kluwer. (2026). Healthcare AI Adoption and Compliance Survey 2026. Wolters Kluwer Health. https://www.wolterskluwer.com/en/expert-insights/healthcare-ai-compliance
2. Anthropic. (2025, September 28). Anthropic Privacy Policy — How We Use Personal Information. Anthropic. https://www.anthropic.com/legal/privacy
3. Microsoft. (2025). Microsoft Copilot Data, Privacy, and Security for Microsoft 365. Microsoft Learn. https://learn.microsoft.com/en-us/copilot/microsoft-365/microsoft-365-copilot-privacy
4. U.S. Department of Health and Human Services. (2003). Minimum Necessary Requirement — HIPAA Privacy Rule Guidance. HHS Office for Civil Rights. https://www.hhs.gov/hipaa/for-professionals/privacy/guidance/minimum-necessary-requirement/index.html
5. U.S. Department of Education. (2023). FERPA and Virtual Learning Related Resources. Student Privacy Policy Office. https://studentprivacy.ed.gov/resources/ferpa-and-virtual-learning-related-resources
6. California Privacy Protection Agency. (2024). Automated Decisionmaking Technology Regulations — Draft Rulemaking. CPPA. https://cppa.ca.gov/regulations/automatedDecisionmaking.html
7. European Parliament and Council of the European Union. (2016). General Data Protection Regulation (GDPR) — Article 22: Automated Individual Decision-Making, Including Profiling. Official Journal of the European Union. https://gdpr-info.eu/art-22-gdpr/
8. New York State Legislature. (2026, March 21). Stop Hacks and Improve Electronic Data Security Act — Amendment Expanding Medical Information. New York State Senate. https://www.nysenate.gov/legislation/bills/2025/S895
9. Apple Inc. (2017). Differential Privacy Overview. Apple Machine Learning Research. https://machinelearning.apple.com/research/differential-privacy
10. McMahan, B., et al. (2023). Federated Learning of Medical Imaging AI at Scale. Google Research / Google Health. https://research.google/pubs/federated-learning-of-medical-imaging-ai-at-scale/
11. Data Protection Commission (Ireland). (2025, May). Decision in the Matter of TikTok Technology Limited — Data Transfer Enforcement Action. DPC. https://www.dataprotection.ie/en/news-media/press-releases/
12. U.S. Department of Health and Human Services. (2025, July). OCR Settles HIPAA Investigation with Deer Oaks Behavioral Health Services for $225,000. HHS Office for Civil Rights. https://www.hhs.gov/hipaa/for-professionals/compliance-enforcement/agreements/index.html
13. Cisco. (2025). 2025 Data Privacy Benchmark Study. Cisco Systems. https://www.cisco.com/c/en/us/about/trust-center/data-privacy-benchmark-study.html
14. Cavoukian, A. (2009). Privacy by Design: The 7 Foundational Principles. Information and Privacy Commissioner of Ontario. https://www.ipc.on.ca/wp-content/uploads/resources/7foundationalprinciples.pdf