Malicious AI Prompt Injection Attacks Increasing, but Sophistication Still Low: Google

Threat Overview

Google's research reveals a growing pattern of prompt injection attacks targeting AI systems — a critical development for healthcare practices increasingly relying on AI-powered clinical documentation, patient communication tools, and administrative automation. While current attack sophistication remains relatively low, the volume of attempts is climbing. For independent practices, this matters because AI systems often handle protected health information (ePHI) through chatbots, voice assistants, documentation tools, and patient portals. Even unsophisticated attacks can manipulate AI systems into exposing data, generating incorrect clinical documentation, or bypassing security controls. Google's findings distinguish between benign probing and genuine malicious exploits — but healthcare practices face unique risks when any AI manipulation could compromise patient privacy or clinical accuracy.

Attack Vector & Tactics

Prompt injection attacks work by feeding malicious instructions to AI systems that override intended behavior. In indirect attacks, threat actors embed hostile commands in documents, web pages, or data sources the AI processes. For healthcare applications, this could manifest as:

• Clinical documentation AI ingesting manipulated patient notes that instruct the system to ignore privacy rules or alter medical records
• Patient-facing chatbots receiving crafted inputs that extract ePHI or generate misleading medical guidance
• Administrative automation tools processing invoices or forms containing hidden commands that redirect payments or expose vendor data

The "low sophistication" finding suggests these attacks rely on social engineering rather than technical exploits — making them accessible to basic threat actors but also easier to defend against with proper controls. Healthcare practices using AI for ambient clinical documentation, patient scheduling, or insurance verification are potentially vulnerable if those systems lack input validation and security boundaries.

Defense Measures

Practices deploying AI systems must treat them as high-risk ePHI processing environments requiring explicit security controls:

• Input validation and sanitization — establish strict rules for what data AI systems can process and from what sources
• Business Associate Agreements (BAAs) — verify AI vendors sign BAAs and maintain HIPAA-compliant infrastructure
• Audit logging — track all AI system interactions with ePHI, including prompts, outputs, and data sources accessed
• Access controls — limit which staff roles can interact with AI systems and what data those systems can access
• Vendor security assessments — evaluate AI vendors' security practices, including how they handle adversarial inputs and model safety

The low-sophistication nature of current attacks means basic security hygiene works — but only if implemented. Practices cannot assume AI vendors handle these controls by default.

What This Means for Your Practice

If your practice uses AI-powered tools for clinical documentation, patient communication, or administrative tasks, you are potentially exposed. The HIPAA Security Rule's Technical Safeguards (§164.312) require access controls and audit controls for systems handling ePHI — and AI systems are not exempt. Regulators increasingly scrutinize how practices secure emerging technologies, and a breach caused by AI manipulation carries the same penalties as any other HIPAA violation: up to $1.8M per violation category annually, plus state attorney general enforcement and patient lawsuits.

Action steps:

• Inventory AI systems currently accessing or processing ePHI
• Verify BAAs are in place with AI vendors
• Document security controls specific to AI systems in your policies
• Train staff on AI-specific risks and safe usage practices