New Threats Demand New Approaches: How Superhuman Security Engineers Use AI

By Igor Tarpan, Security Intelligence Engineer, Superhuman

With a thin, in-house-built harness, we can build and deploy agents in under an hour—all without replicating the underlying agent infrastructure

Today, sophisticated cyberattacks have become as cheap and consumable as tokens, and the Superhuman Security Engineering team takes this growing threat seriously. We’ve seen an increasing number of AI-assisted attacks, a direct result of AI making it easier for anyone to hack (similar to the shift in code generation). But as the controversy about Claude Mythos rages on, with Anthropic and the US government going back and forth over whether the model can even be released, we’re convinced we’re reaching a fresh inflection point, one defined by models becoming more and more capable, and more and more dangerous.  

Our team, like many teams, is in the process of making the transition from automated approaches to agentic ones. We used to employ standard, rigid security automations that made our workflows more efficient but to a very limited degree. With the advent of AI agents, we started experimenting. Our first agent-based system saved time lost to operations but created new bottlenecks. This is where we suspect many other teams are today. 

Now, we’ve built a new infrastructure that supports the continuous shipping of agents, turning a bottleneck into a virtuous feedback loop. In this article, we’ll provide an overview of the new system, and in further articles, we’ll do deep dives on each component. 

Upgrading rigid automation to intelligent (but limited) agents

Agents are not magic, and adding them to your processes won’t magically save you time and energy. Without the right infrastructure, agents can become bottlenecks themselves, turning potentially significant gains into merely incremental ones. The key is to find the right infrastructure to support the agents, allowing your engineers to truly delegate tasks to them and free up time for more creative, strategic work. But getting to this point requires a lot of iteration, testing, and running into the limits to see what we need to design around.

Before AI, our Security Engineering team employed rigid, deterministic Python-based automations that ran on specific events, performed specific enrichments, and correlated against rules or opened tickets. This helped the team delegate repeatable, well-bounded work, but all other work fell outside the scope. Triaging novel alerts, correlating sparse signals, and determining whether an alert warranted a deeper look—all these required human involvement. 

As LLMs matured, we began experimenting and eventually replaced the most tedious of these processes. For instance, we built automations using an LLM, integrating a collection of MCP servers with a security SaaS platform and a Jira trigger, reducing tier-one triage time from 30–45 minutes per ticket to about four minutes. This early experiment resulted in two main outcomes: It gave back some of the time we were losing to operations, a measurable benefit, and it proved that AI automation is the right path to invest in.

So we started as any team would: by identifying use cases, building more agents, and tracking outcomes. But we quickly ran into a problem: The infrastructure became burdensome. Each agent needed to duplicate the underlying infrastructure, including a queue, session store, continuous integration pipeline, and compute cluster. That meant we were reshipping the platform every time we shipped an agent, which just wasn’t scalable. We needed to take a step back to determine the right infrastructure to ship agents more efficiently and maintain—or improve—our security posture along the way.

Building a shared harness platform

To ship agents faster and more securely, we built a shared harness that provides monitoring and guardrails by default. The harness comes first and provides credential isolation, a read-only mission, signed plugins, a trigger envelope, and an agent-shaped data layer, all of which work together to form a platform to ship new agents for security operations. 

Three elements behind each agent

On our platform, agents are configurations of skills with a mission, not bodies of code. They comprise three elements, each sitting on top of the shared harness:

• Agent configuration: Two YAML files that set the agent’s security and operational profile, including triggers (the conditions that cause an agent to activate), session mode (the setting that defines how a single run is conducted), turn limits (a cap on how many steps the agent may take before it must stop), an egress tier (a classification controlling the agent’s outbound network access), which MCP servers it can call, which tools it can invoke per server, and behavioral guardrails
• Main system prompt: A markdown mission file read by the model every time it runs to help it understand how to carry out its mission, written the way a human would write a runbook for a junior analyst
• Plugins: Signed bundles from a separate plugin repository that equip the agent with capabilities, such as skills, MCP integrations, subagent definitions, and in-process hooks

The harness reads the agent config, the model reads the system prompt, and the plugins provide the capabilities allowed by the configuration. Engineers can build an agent with a single command, allowing them to focus on writing the prompt rather than scaffolding. 

This approach has its trade-offs: Markdown missions aren’t unit-testable like code, which increases the risk of regression, and skills can drift across teams as they update plugins on different cadences. Model upgrades can also shift behavior in ways tests won’t catch, but we can mitigate this risk by pinning model versions, reverifying upgrades, and running evaluations. 

Still, especially because we can mitigate these issues, the harness-based approach has proven ultimately effective: It pushes isolation, observability, and coordination into the platform around the model rather than treating these variables as constraints.

Agents on the platform today

Today, the team has built five agents on our platform, with responsibilities ranging from triage to adversarial simulation, triggered by Slack chats and Jira tickets. Here’s an overview of the agents and their use cases: 

Agent

Trigger

Role

Treat Intel triage agent

Queue

Autonomous triage of threat-intel alerts; fans out to specialized subagents on critical alerts; publishes structured reports

Conversational CTI assistant

Chat

Interactive answers on common vulnerabilities and exposures, indicators, threats, and posture questions

Detection-engineering agent

Ticket

Implements or tunes detection rules; validates and opens merge requests

Red-team agent

Manual

Adversarial simulation; the one agent on the open-internet tier

On-call assistant

Manual

Local on-call assistant that we interact with manually and work hand in hand with during incidents, custom requests, etc.

The four defense principles that inform our security observability layer

Since we’re working with AI agents, we wanted to protect against two main threats: prompt injection (when adversarial content is introduced into the data that the agent processes, instructing the model to take actions outside its mission) and supply chain risk (when upstream vulnerabilities cascade into our agents, which have broad access to security tools, investigation data, and internal identities). 

To catch and debug these threats, we built a single observability layer that sits above everything. It includes:

• Network traffic logging and monitoring to understand and detect anomalies at the network level
• A scrubbing pipeline that catches secrets and PII during the execution and log/trace steps and before delivery to a central location to prevent data leakage
• Tracing that spans every model and tool call to capture prompts, responses, latency, and cost (with scrub rules applied on write)

Additionally, to maintain true defense in depth, we’ve layered in four defense principles that further strengthen our security posture, primarily focused on limiting the types of access agents have and on creating guardrails to enforce these policies. Let’s take a closer look at them.  

1. Credentials never reach the agent process.

To prevent accidental credential leaks from prompt-injection attacks, we designed the agents to store no credentials or secrets. A man-in-the-middle forward proxy runs alongside the agent in every pod, and we route all outbound HTTPS routes through it. Agents call upstream APIs without credentials in the request, inject the authentication header, re-encrypt, and forward. 

2. Agents have no direct internet access.

By design, every tool the model can call is one that the platform supplies. There is no native internet, file system, or shell access without a wrapper; we disable the model’s native browser tool. Instead, we use a platform fetch tool that we route through a credential proxy. We ensure it respects the egress tier, enforces a per-agent rate cap, applies content-level filtering, and, for agents that need it, passes large pages through an in-tool summarization step that returns a structured digest instead of raw HTML. If a page has prompt injection text, the web fetch tool can disarm it and prevent raw text from being delivered directly to the agent, preventing other prompt injection attacks.

3. The mission is read-only, and dangerous defaults are pinned at the image layer.

We ensure plugins are read-only and load only at start-up. Agents run with restricted permissions, and we lock down critical settings to prevent unauthorized changes. Any special exceptions require formal approval from the platform team.

4. Custom AI guardrails enforce behavioral policy on top of the model.

AI agents are surrounded by guardrails that enforce agent-specific rules, prevent runaway behavior, and intercept tool use. This keeps the agent from drifting outside its intended mission, even if the underlying model makes a bad decision.

Guardrails include:

• Mission-level rules with behavioral invariants. Layered with hooks and budgets, these rules catch most model drift.
• Anti-loop budgets cap per-tool retries and per-investigation tool-call counts.
• Pre- and post-tool-call hooks can block or modify tool calls based on platform policy.

The last limitation can, for example, refuse a write_artifact call without a valid session ID, downgrade a destructive operation to a dry run, or require an explicit confirmation phrase before a high-impact action. 

Surprises along the way

The harness has been live for about five months, and as we’ve launched agents, we’ve learned some lessons from operating the harness in the wild.

Plugins are an attack surface in a nondeterministic environment

The first time someone proposed editing a plugin via a merge request, we approved it in five minutes. The diff looked like a simple documentation change with clearer wording in the section on duplicate detection. The second time the reviewer looked at it, though, they noticed that the change affected how the alert-triage agent determined whether something was critical. 

That was when we realized that plugins are software and that we needed to iterate on our design. A malicious or careless plugin update is effectively changing the agent’s decision-making process. To fix this, we updated our code approval process for plugin changes. Now, plugin updates require a signature from the platform team’s signing key, which is the same approval gate as a code change to the harness.

Tasks need idempotency keys

The harness exposes a small set of trigger types, including queue, chat, ticket, webhook, and schedule, that are all normalized into a common envelope, which includes source, source ID, payload, and idempotency key. By the time the mission code runs, every trigger looks the same, and agents trigger each other through the same envelope. 

But that’s our design today. The first version didn’t include idempotency keys, and we ended up with a lot of duplicated work. At one point, the alert-triage process dispatched an agent to investigate an alert, but the mission sent the agent back to triage for a related entity. The triage agent then saw a new entity and dispatched the agent again. 

We caught the problem when the queue depth spiked overnight. Now, every dispatched agent carries an idempotency key derived from the upstream event ID and writes it to a database with a seven-day TTL. 

Data needs to be shaped for agents

Our first autonomous agent pointed at the SIEM’s query API, but the bill was steep—just to run a single investigation, the agent read the same vendor threat report in its entirety five times, since the API returned it in full. We realized then that the SIEM API was designed for large batch calls, not for a model paying a token tax on every byte. 

The deeper problem, however, was the agent itself. Agents are good at investigating and bad at generalizing. You can hand a model a hypothesis, and it will pursue it, but if you hand it 20,000 log rows and ask for the anomaly, it often gets stuck on the first 500. 

To address both issues, we began shaping the data for the agent by precomputing aggregates and structuring each entity around a fixed set of dimensions. Now, we return a single ~400-token packet instead of six 200-token tool calls. 

Results so far

The throughline of our new approach to security operations is the harness, which we treat as a central, foundational element. We expect further iteration, but already, security operations at Superhuman have been transformed. 

• Time to new agent: Once an engineer is familiar with the platform, they can deploy a new agent—from thinking of it to running it in staging—in about one hour.
• Expanded monitoring scope: In the twelve months since the harness went live, our detection library (fed by over a dozen custom data pipelines we built in-house) roughly doubled — from about 360 to about 720 production detections — across 51 distinct log sources. Over the same period, we tuned over 670 detections and continuously retired stale ones, keeping coverage sharp.
• Reduced alarm fatigue: Our agents monitor flags automatically, allowing us to expand the scope of what we’re watching while ensuring humans don’t experience alarm fatigue from false positives. Handling alert flow at machine speed enabled us to widen the aperture without overwhelming the team.

In the long term, the biggest benefit might be the hardest to measure: increased experimentation. As AI reduces manual, repetitive work, we can experiment and take on more ambitious projects.  

As AI-armed attacks proliferate, we’re confident that this approach will enable us to stay ahead of the evolving threat landscape. If you are building something similar, or if any of these design choices would change your approach, we’d like to hear about it. Join us by applying for a role on Superhuman’s Security Engineering team.