How Great AI Agent Engineers Think About Security
AI agents are quickly becoming part of our daily software engineering stack. They read repositories, search documentation, generate code, call APIs, modify infrastructure, and much more. Entire development workflows are now partially autonomous.
For engineers alike, the question is: how much faster can we build with agents?
But the engineers building the most reliable systems are asking a different question.
What happens when the agent is wrong? Or worse, manipulated.
The AI agent engineers do not start by asking how powerful their agents can become. They ask what those agents should never be allowed to do.
Because once an agent can read external data, call tools, and execute actions, the system stops being a coding assistant. It becomes a security-sensitive automation layer.
Understanding that shift is what separates experimental agent workflows from production-grade systems.
The First Rule: Treat Model Output as Untrusted
The most important mental model: don’t trust the output from the start.
Large language models do not enforce a strict boundary between instructions and data. When an agent reads external content such as documentation, websites, comments, or messages, that content may contain instructions that influence the model’s reasoning.
This is why prompt injection exists.
Security guidance from organizations such as OWASP describes prompt injection as one of the most critical risks in systems built around language models.
Unlike traditional injection vulnerabilities, the problem is structural. The model interprets text rather than executing deterministic instructions.
The engineers therefore assume that any output generated by the model may have been influenced by untrusted data.
The system must remain safe even if the model is partially manipulated.
The Second Rule: Reduce Agent Authority
Most failures in agentic systems are caused by what the model is allowed to do.
In the beginning of this article, we highlighted how agents are frequently granted capabilities such as: writing to repositories, executing commands, calling external APIs, deploying infrastructure, and more.
When these permissions are too broad, a compromised reasoning process can lead directly to unintended actions.
Security frameworks increasingly refer to this problem as excessive agency. Great agent engineers apply the principle of least privilege. An agent that reads documentation should not have permission to modify infrastructure. An agent generating code should not be able to deploy it.
Limiting authority reduces the potential impact of manipulated behavior.
The Third Rule: Separate Reasoning From Execution
One of the most effective architectural patterns in secure agentic systems is separating reasoning from execution.
In this design the agent first produces a plan. That plan is then evaluated by deterministic controls before any actions are executed.
This approach introduces a verification layer between model reasoning and system behavior.
For example:
- The agent proposes code changes.
- Automated tests verify the change.
- Security checks analyze dependencies.
- Policy rules confirm the action is allowed.
- Only then does execution occur.
This structure prevents a single model response from directly triggering critical operations.
The Fourth Rule: Security Gates Are Non-Negotiable
Agentic development can accelerate engineering workflows dramatically. However, speed cannot replace verification.
Great AI agent engineers treat security gates as mandatory components of the pipeline.
Generated code should pass through the same controls as human-written code:
- unit and integration testing
- static analysis
- dependency scanning
- secret detection
- policy validation
These checks ensure that autonomous systems cannot bypass the safeguards that protect production environments.
Research from the NIST AI Risk Management Framework emphasizes the importance of continuous evaluation and governance for AI systems throughout their lifecycle.
Agentic pipelines are no exception.
The Fifth Rule: Observe Everything
Autonomous systems require visibility.
Agents can perform many actions rapidly and across multiple systems. Without detailed logging and monitoring, unexpected behavior may go unnoticed.
Great engineers instrument their agent workflows the same way they instrument infrastructure.
They track: tool usage, command execution, repository changes, network requests, system modifications
Observability allows organizations to detect anomalies early and respond quickly if something goes wrong.
The Threat Model Behind These Principles
These engineering practices exist because agentic systems introduce new attack surfaces.
- Prompt injection allows external content to influence reasoning.
- Tool misuse can trigger unintended operations.
- Persistent memory layers can be manipulated to bias future decisions.
- Model outputs can introduce vulnerabilities if downstream systems treat them as trusted input.
Traditional application security tools focus on static code. Agentic systems require a broader view that includes reasoning, context, and automation behavior.
Understanding this expanded threat model is essential for building secure AI-driven systems.
The Next Phase of Application Security
Agentic development is not a temporary trend. It represents a structural shift in how software is built. Engineers are increasingly collaborating with autonomous systems that interpret context, generate solutions, and execute tasks.
This transformation creates enormous productivity gains, but it also creates new responsibilities. The engineers building the most reliable systems understand that autonomy requires stronger security design:
- They assume models will make mistakes.
- They assume inputs will be adversarial.
- They assume outputs may be unsafe.
And they design their systems so that none of those assumptions can compromise the environment.
As AI agents become more capable, the organizations that adopt this mindset will be the ones that build faster without sacrificing security.