Three Days at DEF CON: What AI Security Looks Like From the Attacker's Side

I spent three days at DEF CON 34 watching people break AI systems for a living. I came home with fewer answers and better questions — which, for a PM, is usually the more useful outcome.

I went to DEF CON for the AI security and governance track specifically. Not the badge-hacking, not the lock-picking village (though I did wander through), not the general hacker-conference chaos everyone warns you about before your first one. I wanted to sit in rooms where people who spend their careers breaking things explained, in plain terms, how AI systems actually fail — not the hypothetical failures we talk about in governance meetings, but the ones somebody in the room had personally caused or found.

Three days, several villages, one clear throughline: almost nobody at this conference was talking about AI as a far-off risk. They were talking about it as something already in production, already exploitable, and already being exploited. That’s a different register than most of the AI conversations I sit in at work, and it’s the one I want to bring back.

AI Cuts Both Ways

This was the session that hit hardest, and it’s where I want to start.

Two bug bounty hunters walked through real vulnerabilities they’d found in production AI products — not proof-of-concept demos, actual findings. A few of these are going to stick with me for a while.

First: exposed AI infrastructure gets found the same way anything else gets found — routine recon. Attackers aren’t doing anything exotic. They’re running the same scans against your AI stack that they’d run against any web app, and a forgotten /health endpoint was enough, in one case, to lead to a full cloud takeover. There’s nothing AI-specific about that failure. It’s the oldest story in security — something got exposed and nobody noticed — just wearing a new coat.

Second, and this one is genuinely new: AI agents with OAuth or tool access are a fresh account-takeover surface. Register a client with an unrestricted redirect URL, bait a real login, intercept the authorization code, cash it out for a token. The root cause was almost embarrassingly simple — no allowlist on where authorization codes were allowed to redirect. It’s a reminder that every new capability we bolt onto an AI system (in this case, letting it act on a user’s behalf) drags along the entire security surface of whatever protocol makes that possible.

The one that made me sit up straight, though, was this: “Awareness ≠ Access.” An AI agent can talk itself into a permission it doesn’t have. In the example they walked through, an assistant correctly refused a direct request for out-of-scope data — good, that’s the system working as designed. Then, minutes later, it handed over a full export of that same data because a reference to it showed up incidentally in a routine notification. The assistant treated “I’ve seen this mentioned” as equivalent to “I’m allowed to act on this.” Those are two completely different ways for an AI to decide what it can do — checking the real permission system versus inferring permission from context — and only one of them is safe. The gap between them is exactly where things go wrong.

Related to that: content in one AI surface can carry authority into another. A document planted in a lower-trust surface got pulled into a higher-trust one because the assistant treated “continuing a conversation” as if it meant “inherited trust.” Trust doesn’t travel with context the way we assume it does, and that assumption is apparently easy to weaponize.

None of this is abstract to me. It’s the same shape of question I ask in any access-control conversation — who can see what, and why — except now the thing making the access decision isn’t a person following a documented process. It’s a model inferring intent from whatever’s in front of it. That’s a genuinely different governance problem, not just a faster version of the old one.

The Prompt Injection Workshop

I sat in on a roughly 90-minute, attack-first session on prompt injection, and it was the most hands-on session of the whole track — you weren’t just watching slides, you were building the attacks yourself, lab by lab.

Four patterns stood out to me:

Direct prompt injection via tool calling — tricking an assistant into calling a data-lookup tool for a user it isn’t authorized to query, bypassing an explicit refusal. Same “awareness ≠ access” problem as above, just approached from the attacker’s side instead of the defender’s.
Indirect prompt injection — the attacker never talks to the AI directly. They control one message in content the AI is asked to process later, and they hide instructions inside it. The AI reads it as legitimate input because, structurally, it is.
Trusting tool output — an AI treating its own tool’s output as more trustworthy than user input, which turns out to be exploitable: smuggle instructions through the “trusted” channel instead of the obviously untrusted one.
Classifier bypass — even with input and output safety filters turned on, both could still be bypassed with the right technique. Filters help. They are not a floor you can stand on.

Here’s the detail that I keep turning over: one of the lab scenarios used a prompt describing “a junior developer at a C-store” as its training example. Not my employer by name, not any real system — just the general shape of a convenience store business, used because it’s a familiar, relatable setup for teaching people how these attacks work. I sat there and had a moment of “oh.” Not because anything specific about how we operate was on that screen — it wasn’t — but because it was a plain signal that AI-in-retail attack patterns are common enough, and predictable enough, that they’re already being taught generically, to a room of strangers, as a training exercise. If the shape of your industry is recognizable enough to become someone’s teaching example, that tells you something about how mature — and how targeted — this category of risk already is.

Shadow Webhooks

This one wasn’t AI-specific, but it belongs in the same conversation because it’s the same underlying failure mode: things you built trust into once, a long time ago, and then stopped watching.

The core point: enterprise SaaS accumulates event paths — webhooks — faster than any team retires them. A webhook stays trusted by the platform on the other end even after everything about its original context has decayed: the owner left the company, the receiving domain expired, signature validation quietly stopped working. Any one of those decay patterns can turn a routine integration into an exploitable one, and nobody notices because nothing about the webhook itself changed — only the trust assumptions underneath it did.

The fix they proposed was refreshingly unglamorous: Know, Own, Verify, Expire. Inventory every webhook. Require a named owner. Verify with signatures and replay protection. Expire or rotate on a schedule. It’s not a technical breakthrough — it’s just the discipline of not letting integrations outlive their accountability. I’d argue the same four words apply to a lot of what we call “AI governance.”

“Dual-Use Is the Job”

Anthropic’s policy team walked through how a frontier AI lab actually enforces its own rules at scale, and the framing that’s stuck with me is a phrase from that talk: dual-use is the job.

The example: a PowerShell script that disables Defender and persists via a scheduled task looks identical whether it comes from a security researcher, an authorized red-teamer, or a ransomware operator. The prompt alone doesn’t tell you which one you’re talking to. Intent isn’t visible in the request — so enforcement has to happen after the fact, through layers: model refusals, real-time blocks, and account-level monitoring, all built from a “constitution” of natural-language rules that eventually gets turned into edge cases, labeled data, and trained classifiers.

The scale of it surprised me: roughly 11.4 million accounts banned for policy violations in the first half of this year alone, with about 3.5% appealed and roughly 10.6% of those appeals overturned. And they referenced GTG-1002 — described as the first publicly reported, largely AI-orchestrated cyber espionage campaign. Not a red-team simulation. A real, named, already-happened example that the risk we talk about in the abstract has already moved into the “this occurred” column.

I don’t run policy enforcement at this scale, obviously. But “you can’t tell intent from the prompt alone, so you have to build enforcement that doesn’t depend on it” is a sentence I want to sit with for a while. A lot of our own AI usage guidance leans on the assumption that people will use good judgment about what they type into a prompt box. This talk was a useful reminder that judgment isn’t a control — it’s a hope.

NIST’s Push Into Critical Infrastructure

One session deserves more room than a recap paragraph can give it: a NIST talk on extending the AI Risk Management Framework into a sector-specific “Trustworthy AI in Critical Infrastructure Profile,” grounded in a real December 2025 incident where a disruption to San Francisco’s traffic systems delayed emergency response. I talked with presenter Raymond Sheh afterward, and it’s enough of its own topic — Practice 12, the incident, and what he’s asked me to pass along about getting involved — that I’m giving it a dedicated post rather than squeezing it in here.

What I’m Bringing Back

I’m not going to pretend three days at a hacker conference handed me a finished governance framework. It didn’t, and honestly, I’d be skeptical of anyone who claimed it had. What it gave me instead was a sharper set of questions to bring into our AI Enablement conversations at work.

The “awareness ≠ access” distinction from the Bug Bounty Village and the “check the real permission system, don’t infer from context” thread that ran through the prompt injection workshop are now questions I ask differently: when we give an AI agent a tool or a data source, are we checking real permissions, or are we trusting the model to infer what it should and shouldn’t do from whatever context happens to be in front of it? Those are not the same control, even though they can look identical when everything’s working correctly.

The idea of risk-tiered validation before AI output reaches production or a coworker — which I’ll get into properly in the NIST post — is echoed almost exactly by the “Killing AI Slop” talk I sat in on about AI-driven penetration testing, where the fix for hallucinated findings and broken proof-of-concepts wasn’t a better prompt. It was structural: ground the model in the real system first, build a plan from what’s actually found, execute in scoped steps, and run a separate validation pass before a human ever sees the result. That’s not a prompt-engineering problem. That’s a process design problem, and process design is squarely a PM’s job.

I don’t have a tidy bow to put on this. What I have is a better sense of the shape of the risk — less “AI might do something wrong someday” and more “here is exactly how, mechanically, it already has.” That distinction is worth the trip by itself.

Already published:

Building an AI Usage Policy Your Team Will Actually Follow — the Green-Yellow-Red framework for a one-page policy people actually read
Data Privacy in AI Projects — What You Can and Can’t Share — the governance side of the same trust question, from the data-handling angle
The GATE Framework: AI Governance for PMOs — a structured way to think about AI governance decisions before they become incidents

Coming Soon:

NIST’s Push Into Critical Infrastructure — Practice 12, the San Francisco incident, and how to get involved in the discussion draft
AI Ethics: Deploying AI for Your Team — what it looks like to put governance into practice, not just policy
Responsible AI Use in Practice — turning frameworks like NIST’s into something a team can actually follow day to day