New Architecture, Underestimated Security Requirements
The security controls that protect a conventional web application are well understood. Authentication and authorisation at the API boundary, input validation, secrets management, vulnerability scanning in the build pipeline, runtime protection. These controls reflect decades of accumulated AppSec practice, and most enterprise security programmes implement them with reasonable competence.
AI workloads introduce attack surfaces that these controls were not designed to address. The threat model for a machine learning system includes attacks that do not have equivalents in conventional software security, and they require security controls that most enterprise security programmes have not yet built. The gap between the security controls that AI workloads require and the controls that most enterprise security programmes currently provide is wider than most CISOs appreciate, and it is growing as AI deployment accelerates faster than the security programmes that are supposed to govern it.
This is not a theoretical concern. AI workloads that handle sensitive data, that are integrated into customer-facing processes, or that provide capabilities that could be misused if manipulated are security risks that materialise in production, not in academic research.
The Attack Surfaces That AI Workloads Introduce
Training data poisoning is the attack that is most specific to machine learning systems. An attacker who can influence the data used to train a model can embed systematic biases, backdoors, or incorrect behaviour patterns into the model before it is deployed. The model then produces incorrect outputs in predictable ways that the attacker can exploit. Protecting against training data poisoning requires data provenance controls that ensure training data integrity from collection through to training use, and anomaly detection in training data that identifies manipulated inputs.
Model inversion and membership inference attacks exploit the model’s learned behaviour to extract information about the training data. A model trained on sensitive personal data may inadvertently encode information that can be extracted by an attacker through carefully crafted queries, even if the training data itself is appropriately protected. The security implication is that access to a trained model is access to some information about its training data, and this access must be governed accordingly.
Prompt injection is the attack surface most directly relevant to large language model deployments. An attacker who can inject instructions into the input processed by an AI model can potentially redirect the model’s behaviour to produce attacker-controlled outputs, bypass safety controls, or access information the model has been given access to but should not be sharing. Prompt injection is particularly concerning for agentic AI deployments where the model’s outputs trigger downstream actions in other systems: a prompt injection that causes the model to approve a transaction, send an email, or modify a record is an attack with immediate operational consequences.
Adversarial inputs are crafted inputs designed to cause the AI system to produce incorrect outputs. For vision models, adversarial image perturbations can cause misclassification that is invisible to human reviewers. For language models, adversarial inputs can cause misclassification or harmful output generation. The relevance to enterprise security depends on the context: an adversarial attack on a security-relevant AI system, such as one used for fraud detection or access control, has direct operational security implications.
The Architecture Gaps That Need Addressing
Model provenance and integrity is the first architectural gap. Most enterprise security programmes have robust controls for software artefact integrity: code signing, image signing, build provenance attestation. These controls are not being applied to AI model artefacts with the same rigour. A model file downloaded from an external repository, or transferred between environments, without integrity verification is a supply chain vulnerability that the current security architecture does not adequately address.
Training data governance is the second gap. The access controls, audit logging, and data quality requirements that apply to training data used in production AI systems should be equivalent to the controls applied to the sensitive data the model will subsequently process. In practice, training data management in most enterprise AI programmes is handled as a data engineering concern rather than a security concern, and the security controls applied are correspondingly less rigorous.
Inference endpoint security is the third gap. The API endpoints through which AI models are accessed in production are application security surfaces that need the same controls as any other API: authentication, authorisation, rate limiting, input validation, and monitoring. The specific threat modelling for AI inference endpoints needs to include prompt injection and adversarial input as threat categories alongside the conventional web application attacks that the standard API security controls address.
Access control for AI systems that process sensitive data requires controls that are often absent in early AI deployments. A model that has been given access to a company’s internal documentation, customer records, or financial data needs to have that access governed by the same principles that govern any privileged access: least privilege, just-in-time access, and comprehensive audit logging of what the model accessed and when.
The Process Gaps That Compound the Architecture Gaps
The architecture gaps would be more tractable if they existed in isolation. The process gaps that compound them are in some ways more significant because they are the reason the architecture gaps exist.
AI security risk assessment is absent from most enterprise AI project delivery processes. Security review is a standard gate in software delivery pipelines; it is not a standard gate in AI project delivery. AI projects that proceed from ideation through model development to deployment without a structured security risk assessment that addresses the AI-specific attack surfaces described above are deploying AI systems with unassessed security risk.
AI-specific threat modelling is the process that should address this gap, and it requires explicit inclusion in the AI project delivery methodology. A threat model for an AI workload needs to address training data integrity, model deployment security, inference endpoint access control, and the specific attack surfaces relevant to the use case, alongside the conventional application threat model for the serving infrastructure.
Vulnerability disclosure for AI systems is a process gap that will become visible through incidents before most organisations address it proactively. Conventional vulnerability disclosure processes address software vulnerabilities identified in deployed code. AI systems have vulnerabilities that do not correspond to software bugs: model vulnerabilities, bias vulnerabilities, and behavioural vulnerabilities that may be identified by external researchers or disclosed through security research. Having a defined process for receiving, assessing, and responding to AI-specific vulnerability disclosures before the first one arrives is the process investment that prevents the improvised response.
The Investment Case
The security investment required to address AI workload security gaps is not primarily a tooling investment. It is a process and architecture investment: adding security requirements to AI project delivery methodology, training security architects and engineers on AI-specific threat modelling, and building the data governance and model provenance controls that prevent the most consequential attack categories.
The business case is straightforward: the cost of a security incident resulting from an unsecured AI system, including the reputational and regulatory consequences, is substantially higher than the cost of the security controls that would have prevented it. The AI workloads currently in production at most enterprises are generating that exposure with every deployment cycle. The question for CISOs is whether to address it proactively or to wait for the incident.