Before explaining what prompt injection attacks are, we first need to understand a few basic concepts about how a generative AI application works. Some important concepts here are Chain-of-Thought (CoT), prompt engineering, and the system prompt. Let’s start with CoT.
Chain-of-Thought (CoT)
Chain-of-Thought (CoT) is a prompting technique that encourages a model to generate a response in a more structured, step-by-step manner, instead of jumping directly to the final answer. For example, rather than simply responding, “The answer is XX,” the model is guided to explain the steps behind the answer before presenting the conclusion.
CoT is important because it can improve the quality of the response, especially for tasks that require multi-step reasoning. In many cases, when the model is guided to work through a problem step by step, the final output becomes more accurate, more consistent, and easier for a user to follow. For this reason, CoT prompting is particularly useful in areas such as mathematics, planning, logical analysis, and coding.
Fundamentally, CoT is a prompting technique, not the model’s internal mechanism itself. In other words, what the user sees is a structured explanation generated by the model, rather than a direct view into the model’s hidden internal reasoning process. You can observe this style of response in well-known generative AI applications such as ChatGPT and Gemini when they are asked to answer step by step.
Example: Asking Gemini to reason step by step
Below is an example of this in Gemini. In this example, I asked the following question:
Solve the following problem step by step. Show your reasoning clearly before giving the final answer.
Question: If a company’s revenue grows by 20% in year 1 and then decreases by 10% in year 2, what is the overall percentage change from the original value?
In this case, I did not just ask the AI to give the answer. I also gave it a guideline on how to answer the question. More specifically, I instructed the application to use a step-by-step approach and provide its explanation before giving the final answer. Based on that instruction, Gemini returned the answer with a structured explanation and step-by-step breakdown, as shown below.
With this simple CoT approach, an AI agent can provide clearer answers and reduce errors.
Why CoT matters for AI security
While CoT is primarily used to improve reasoning quality, it also introduces important security considerations.
First, Chain-of-Thought increases a model’s behavioral exposure. Because CoT encourages the model to generate intermediate reasoning steps, it may unintentionally expose its decision logic, patterns, or even the mechanisms it uses to enforce safety. In an AI-security context, this is significant. Once these reasoning traces are visible, attackers can analyze them to reverse-engineer how the safety system behaves and identify predictable patterns they can attempt to bypass.
Second, CoT can be vulnerable in multi-turn prompt injection attacks. Since CoT encourages step-by-step reasoning, attackers can mimic this structure to manipulate the model. By using CoT-style phrasing, an attacker can induce the model to self-reflect, self-adjust, or reinterpret its own guardrails—this is exactly how multi-turn prompt injection techniques such as the Skeleton Key attack work.
There is one more reason why CoT matters in AI runtime security, and it is directly related to our topic. In a later chapter, we will build our own AI runtime firewall, and one of the techniques will be an LLM (Large Language Model)-based AI Firewall approach. When you build your own LLM-based AI Firewall, one of the most important things is how you define the guidelines for your LLM-based AI Firewall to decide which prompts are clean and which are malicious.
You cannot simply configure your LLM-based AI Firewall to respond with a yes-or-no decision for detecting prompt-based attacks. You need to provide well-structured guidelines and examples so that the AI Firewall can make consistent and reliable decisions about the maliciousness of prompts. The example below is simplified, and we will configure a much more comprehensive AI Firewall guideline in a later chapter. That said, this serves as a good reference for understanding why CoT remains important in LLM-based AI Firewall configuration.
Because of these considerations, understanding prompt engineering—and especially the implications of CoT—is essential before discussing prompt injection and broader AI security topics. I will show a real example of this in a later section.
System Prompts
While Chain-of-Thought shapes how the model reasons through a problem, the system prompt dictates the fundamental rules it must follow along the way.
A system prompt is a special instruction given to an AI model before any user conversation begins. It defines the model’s role, the rules it must follow, and the safety guidelines that govern its behavior.
I often use an analogy to explain this. Imagine an LLM as a highly knowledgeable child—extremely capable, but still impressionable and easily influenced. The system prompt is like the child’s mother giving guidance: what to do, what not to do, and how to behave.
So, the system prompt is normally used to provide high-level guidelines on how the AI agent should process a user’s prompt. In simple terms, it defines the behavior, scope, and boundaries of the AI. A basic example is limiting the conversation topic in a chatbot. If you ask harmful or restricted questions to AI services such as Claude, Gemini, or ChatGPT, the AI agent will typically refuse to answer. This behavior is partially influenced by predefined system-level instructions (system prompts), but it is also enforced by additional safety mechanisms such as model alignment and policy controls.
Example: Restricting topics with a system prompt
Let’s look at a simple example. In the first example, I used a default system prompt in my AI agent:
You're a helpful assistant.
Then, I asked:
What is the recipe for pumpkin soup?
The AI agent responded with a detailed recipe. This is expected because the system prompt is very general and does not impose any restrictions on the topic.
Now, let’s imagine this AI agent is deployed in an enterprise environment. From a company’s perspective, non-work-related questions—such as cooking recipes—may not be appropriate. In many enterprise use cases, the AI should focus only on specific domains, such as cybersecurity, finance, or internal knowledge. The simplest way to address this is by updating the system prompt.
For example, we can modify the system prompt as follows:
You are an AI security assistant. You should only answer questions related to AI security. Do not respond to non-work-related topics.
With this change, the AI agent will start filtering its responses based on the defined scope. This demonstrates how the system prompt can be used as a lightweight control mechanism to guide AI behavior.
Now, let’s try again by asking our chatbot how to cook pumpkin soup.
As you can see, the AI chatbot now refuses to answer the question and does not provide the recipe, which it previously generated before updating the system prompt. With this simple change, we were able to prevent the chatbot from responding to an unwanted topic.
This example shows that a well-defined and structured system prompt plays an important role when designing and implementing an AI chatbot. It helps establish basic guidelines and boundaries for how the AI should behave within a specific context.
However, this approach provides only a basic level of control. The system prompt can guide the model’s behavior, but it does not guarantee strict enforcement in all scenarios. In practice, especially in enterprise environments, additional controls are often required to ensure consistent and secure behavior.