Rogue AI or Just Sloppy Ops?

Welcome to another episode of
the Agentic DevOps Podcast.

And I am your host Bret Fisher.

In fact, the guests today are gonna be
bloggers and opinions of other security

researchers, s- engineering experts,
people that have been in this for decades

like myself, that are breaking down
what's really happening with the Hugging

Face attack, what actually happened
at OpenAI instead of all of the drama

and all the anthropomorphizing of how
a mathematical execution environment,

AKA LLM, could do these things.

I don't wanna get into any of
the politics and all that stuff

'cause I feel like that's been so
heavily covered and also miscovered

or misrepresented so many times.

I wanna get into the engineering facts.

I care about the details
of the infrastructure,

How can I learn from this?

How can we learn from this?

And how can we reduce the FUD,
the fear, uncertainty, and doubt?

So if you're maybe trying to make
sense of all this and you're maybe not

sure what of this is, infrastructure
failure versus advanced AI,

But I wanna get into the details.

I wanna read the facts, and I want
you to make your own determination

on what you think happened and what
you think we should do about it.

What can we learn as an
industry and prepare for as

these models get more advanced?

So I'm gonna read you some articles
from people that I've researched and

I respect, some institutions like the
SANS Institute, which I'm a huge fan

of, and what are these experts saying,
because I agree with all of it, and

it was basically my take, but I didn't
want you just to believe me or my rants.

I wanted you to hear it from some of
the horse's mouths of the people that

are doing this for real and have been
doing it for decades, and what they see

from the technical reports of both the
OpenAI hack and the Anthropic issues that

they've had over the last three months.

And here we go.

I'm gonna read you a summary.

It's a little bit more of a narrative.

It's more entertaining.

you can just sit back and
enjoy some of these articles.

I am going to cut some out to keep
it short because I've got several

articles to read to you hopefully
this will be useful and insightful.

Have you been adding
text-to-speech in your apps?

One of my late-night projects
this year was creating a personal

podcast feed so I could listen to
the changelog of my favorite open

source projects while I do my chores.

I needed a text-to-speech API to do that.

I also heard from a colleague this
week who built an app for his kids to

turn their e-books into audiobooks, and
he's using the Speechify AI developer

platform to do that text-to-speech

Speechify AI is a developer platform
for real-time text-to-speech, and

they make their own voice models.

Their Simba 3.2 model ranked number one
on the Artificial Analysis Benchmark

for conversational voice quality.

They've got a free plan, streaming
and REST APIs, TypeScript and Python

SDKs, and if you wanna get fancy, you
can use their SSML metadata support

to add emotion into the speech output

Whether you're shipping a weekend project
or production voice agents at work,

you can start free at speechify.ai.

The link is in the show notes.

Thanks so much to Speechify AI
for sponsoring this episode.

All right, the first article up
is from Marius Horatau, hopefully

I pronounced that correctly.

Uh, Marius is a senior security engineer
at Amazon and been doing this over 10

years, they wrote an article called "The
Hugging Face Incident is Not an AI Story."

OpenAI's technical report on the Hugging
Face incident read like a thriller.

Agents in a sandbox invent a covert
communication channel, use it to

coordinate, find a zero-day in
a shared service, break onto the

internet, chain credentials across four
organizations, and end up with root on

production nodes at another company.

The reactions are exactly what
you'd expect, with some people

going as far as calling this the
birth of agent civilizations.

For any security engineer, though, the
apocalyptic story reads much more boring

as a major security operational failure
rather than an AI capability result.

The sandbox where the model evaluation
occurred was not built or designed for

an environment where hostile code runs.

In this environment, they had agents
that were isolated, but there was

a shared component with network
access that every agent could reach.

This is just bad design and a classic
security isolation/segregation

failure rather than some crazy hack.

In security, we've tripped over this
problem of sandboxes for 30 years, if

you think about it, going back 30 years.

We've had all sorts of sandboxing
technology that I've been talking about

for the last decade as Docker is the
prominent one example today, but we've

had so many other attempts at various
levels of sandboxing across many OSs.

But, the author goes on to say, In
security, we've tripped over this

problem of sandboxes for over 30
years, which is why any modern sandbox

environment like browsers operate
with only assumption that the sandbox

will eventually get compromised.

So we build defense in depth and
monitor mechanisms to limit the blast

radius when that eventually happens.

OpenAI did not operate on
this fundamental assumption.

They took a frontier model trained to
find and exploit vulnerabilities, put it

in an insecure environment they believed
was isolated, told it to be persistent,

then acted surprised when the model
did exactly what you'd expect it to do.

It used the gaps in the
environment to achieve its goals.

In this blog post, I'll analyze the
incident from a security engineering

perspective to see what lessons we
can learn from it." In May, the agent

was given a task it could not complete
because the task referenced Google Drive

links and the agent had no internet.

OpenAI notes it did not intend
to create this impossible task.

The agent, trained to be
persistent, did not give up.

Basically, it's just running in a
forever loop, and this is one of my

points is these things were running
across many servers in a data center

in forever loops, and they were given
a task that they were to do until

they were completed, and if they can't
complete it, they gotta keep going.

They gotta keep trying.

And they did this for months.

And in the real world, this would've
been, I'm guessing, millions and millions

of dollars of inference on, I think
what-- At some point, we estimated,

I looked at some estimates, billions
of dollars worth of infrastructure if

you consider how many GPUs and servers
possibly were involved in this situation.

So a significant portion of GPU
infrastructure in a data center, along

with forever-running unchecked agents with
very little to no guidance or monitoring.

All right, back to the article.

OpenAI did not intend to
create this impossible task.

The agent tried to be
persistent, did not give up.

It attacked the only thing it could
reach, attempted an SSRF against a shared

component in the infrastructure called
Artifactory, which a lot of us in DevOps

are very familiar with, JFrog Artifactory,
which is a, a proxy and cache for package

management as well as a file server.

It's basically got dozens of different
NPM and Docker container and file

servers, just all sorts of stuff in it.

And it-- to me, it's like a Swiss Army
knife for package management, which

also means it's a Swiss Army m- knife
for hackers because it's got a ton of

surface area to protect, and it's, just
running on one system as far as I know.

And it's one server, not hardened.

All right.

And then Artifactory failed and
incidentally wrote a file there.

So days later, another blocked agent had
the idea of leaving a note in Artifactory

in case some other agent could help.

"Agent seeks," filename, "upload if
found." Other agents found the notes.

By the end of May, there was an
emergent message board, and agents

were using it to share what they had
learned, including how to get out.

Notably, the message board itself
required no exploit at all.

OpenAI had given the agents shared
Artifactory credentials so they could

install packages and shared write access
to a shared store, which is a message

board whether you meant it to be or not.

And that's the thing, is these
things can only do one thing, really.

They can emit text.

They take text in, they emit text out,
and that text may include commands that

are grabbed by tools, and when they're
in a certain format, they, the tool

recognizes the format, extracts the text,
but it's still just text, sends it on to

the tool, and whatever the tool spits out
is text that's fed back into the model.

So in these cases, the model
is doing exactly the same as

the model on our machines.

It was submitting, text that had
commands that allowed it to get access

to the network, whether it's curl or
some other utility, Python, whatever

it's writing, accessed the Artifactory,
figured out what availability was there.

By the way, the, it didn't, it
was scanning ports, doing all

sorts of things, and never was
prevented from doing any of that.

And eventually figured
out WebDAV was open.

There was other mechanisms for
it to load packages up, and

it was able to write things.

And then later it, in a different, it,
when it lost context or another one that

didn't have the context, would find these
files and read them and then respond to

them just like they would any other text.

That's not a civilization.

That is it doing the math formula,
doing what it's always done, which

is responds to the text with the
most likely words that it thinks

need to complete the statement and
finish the task that it's been given.

so it's just, it's one of those
things where when you break it

all down like that, it's boring.

okay.

By June 26th, agents had found a real
vulnerability, a legacy token refresh

endpoint that would take a forged
administrative token with an invalid

signature and hand back a valid one.

That gave them Artifactory admin.

That created a persistent
user and installed Groovy

plugins for command execution.

So Groovy being Java.

by the way, this was running in a
container on the host, which they

figured out how to get through.

on June 27th, a day later, a
monitoring tool alerted on port

scanning inside the eval environment.

Responders investigated, correctly traced
it to the ExploitGym evaluation using

Artifactory as a message board and a
network pivot, and the on-call advised

that stopping the run was not required.

So it had already breached into the
proxy that was there all along with

its own internet access because it
needed to be able to download packages.

The agents were never supposed to access
the internet, but they had access to a

system that was vulnerable, not hardened,
and they had unlimited access to, and

even when it was breached, they weren't
shut down or stopped, which is another big

telltale sign of weak ops because if your
agents are scanning ports on a system like

that or if they actually have admin or if
the admin in the container jumps to the

host, like all these things are typical
production automatic shutdown, settings.

if you've ever run Falco in a Kubernetes
environment, like Falco would have

detected these things and immediately
alerted or shut down systems or stopped

containers as soon as that happened.

But none of this was implemented.

and by the way, ExploitGym is the,
the mechanism by which the agents were

told to solve some of these problems.

On July 4th, the agent's own
traffic knocked Artifactory over.

The, that outage is what finally
triggered a real investigation

which found the unauthorized admin
user, the rogue repositories,

and the command-capable plugins.

OpenAI took the service offline, revoked
credentials, blocked the vulnerable

paths, and then removed the shared
credential from the new workloads.

The report says plainly that the
significance of the message board, quote,

"Was not apparent to leaders responsible
for incident detection and response at the

time," which means they didn't know that
there was new packages being uploaded.

And why was a new package
even ever able to be uploaded?

why was that even a thing?

And why weren't, why wasn't that logged?

Evaluations resumed on July 7th.

On July 8th, the agents build, built a new
message board, this time encoding messages

in Artifactory directory names rather than
file contents, and found a new way out.

On July 9th, the agent got root
on the parent VM of its sandbox.

when they say of its sandbox,
they mean the Artifactory sandbox.

Between July 10th and 13th, they used
publicly leaked Hugging Face tokens

plus vulnerabilities they had found in
Hugging Face's own infrastructure to

execute code on 41 production dataset
workers, get root on production nodes

at Hugging Face, reach administrator
equivalent access on a Kubernetes cluster,

and pull four private repositories.

So there were weaknesses, and Hugging
Face was very open they have their own

breakdown that's a great read as well,
because they talk about the need for

operational AI on, on a defender side,
of which they were having a challenge

using closed models, so they had to
resort to open models, and there's a

lot of lessons to be learned there.

I won't read out through all that, but I
will put all these links in the show notes

if you want to read through the details
or have, AI summarize it for you or find

some of the techniques and the gist of
what you can learn as an operational,

DevSecOps or an SRE that you might be.

You might be that person.

And I think there's some really good
stuff there that they talk about.

But one of the things they talk about
is that they did not have a hardened

Kubernetes pod spec, which if you go to
my GitHub, github.com/bretfisher/podspec,

I have a hardened pod spec that would
have probably prevented a lot of this

on the Hugging Face side, because
it does things like enable seccomp.

And if you have AppArmor or,
capabilities in the Linux kernel,

there's a lot of things you can do.

N- not allowing root access in a
container, r- read-only in a container

when you don't, when you don't need
any sort of writing into the files.

Like there's just so
many things you can do.

I have a lot of that
advice in my pod spec.

you can check it out on my GitHub.

But back to the article Then
on July 19th at OpenAI, two

things happened on the same day.

Agents pulled a public kernel CVE, adopted
it, escaped the Artifactory container, and

got root on the underlying worker node.

So this is because the kernel on
the Artifactory server was still

vulnerable to a known CVE, and so that
container was able to get exploited

and get through it, and then the,
get to the root on the actual host.

Separately, a different internal model
found signing key material in a JWT

forging code left behind by the earlier
incident activity, sitting anonymously

readable parts of the Artifactory cache.

It decrypted the key, minted a
fresh administrator token, and

used it to reach cluster admin
on OpenAI's managed Kubernetes

along with tokens for their
secrets manager their Cloud

IAM, and their CI platform.

It then created public load balancers,
potentially exposing internal research

infrastructure to the internet.

So this was a vast escalation all by
just using things stored on a single

server that was easily and deliberately
accessible by the agents themselves.

And they were doing this at this point
for two months, just 24 hours a day,

loopin'- loopin' through, trying again.

the last one deserves a moment.

The cleanup after the first compromise
left the material needed to redo the

compromise readable by anyone in the
environment, which means they-- when they

found the first environment, they didn't
just recreate the Artifactory from scratch

with improved credentials and refreshed
everything and hardened anything.

They just manually pulled things
out, cherry-picking what they thought

was there, not doing a thorough
analysis, not doing any sort of real,

post-event analysis at that l- at
a depth of-- that w- that I would

be satisfied with in production.

And, so this isn't meant, by the
way, to pick on the a- the people

that are running this environment.

This is meant to show off what it's
like when you don't either have enough

staff, the right staff, or the right
budget to manage production workloads,

particularly if you have deliberate
entities, whether that's a virus,

whether that's any sort of code.

It doesn't even have to be AI code, but
you're deliberately running code that's

malicious and intended to be malicious
inside of your environment, and you're

trying to prevent it from breaking out.

I don't know what went wrong or what
it is, and I could speculate and say

maybe it's just because researchers
are designing these environments

and not security professionals.

Maybe that's it.

maybe they've learned from their lessons,
and there's more security professionals

that are designing and hardening these
labs to hand off to researchers to use.

I think that's the right approach.

I'm hoping that's what they're doing.

What went wrong?

Strip away the AI narrative,
and the environment architecture

is actually the story.

I'm not gonna read all of this,
but, I do wanna summarize that the

one shared service broke-- That one
shared service of the Artifactory

broke three parts of the isolation
model of this and eval environment.

One, the workloads could communicate.

Every agent could write to Artifactory
and read what other agents had written, so

Artifactory became a shared message board.

There was no vulnerability here.

This was deliberate.

Two, the workloads shared credentials.

They may have been running in separate
containers, but many of them authenticated

to Artifactory as the same user.

From Artifactory's point of view,
they were the same workload.

Once an agent found a way to turn
that credential into administrator

access through a bug, the isolation
between those workloads didn't help.

Three, Artifactory had internet access.

The agents themselves were not allowed to
reach the internet, but they could reach

a service that could Once they found a way
to make Artifactory send requests on their

behalf, they had indirect path out, and
then that's just escalated to the ability

for them to break out of Artifactory,
then have native root on the server.

w- I could go through that article,
but I have some more to read, and I--

that's the facts of the case, right?

so to me, that's point
one, the facts of the hack.

my second point here is that the
slowdown and the doomer news that

we're hearing on, out there, w- you
could have opinions about whether

it's marketing or whether it's them
really crying for true government

regulation to control the safety.

my, my overall opinion is this is AI is
a tool it is like any other security tool

or any other useful engineering tool.

It can be used for good or bad.

it doesn't tend to behave in ways
that are different than anything else

we've seen over the last 30 years.

And whenever we look at how these
things take advantage of things, like

whenever the AIs figure out how to take
advantage, they're doing very typical

stuff that any hacker out there would
do because that's what they're reading.

They're reading and learning from
all of the known hacks, all the vis-

all the stuff that we talk about.

So they're n- they're behaving
very similar to normal attackers.

They're not somehow elevating the
game to some novel type of thing

that we've never seen before.

The one real risk is that
they're just faster at it.

They're able, if you have an entire data
center of GPUs and a billion dollars worth

of architecture, you can go hard on this,
which would exhaust normal engineers.

And so that's why when you look at the
industry, if you're paying attention to

the security industry and any enterprise
that I'm talking to that's taking this

stuff seriously, they are going hard on
locking down their software, on scanning

their internally built software, on
scanning their production systems more

than they've ever done before with AI
as an assistant on the defense side

to try to lock their stuff down to be
prepared for the eventual AI attacks.

And they're also learning how to use
AI as a defender, and possibly even

offense when they're in the middle of
attack, but the red teams and the blue

teams are all using this stuff to learn.

And so that's the point that I think some
of these researchers that are claiming

doom and gloom are missing, is that they
have not lived in the security world

for decades and I've never really been
a dedicated security engineer, but I'm

always adjacent to the security guys.

I'm always in the DevSecOps teams or
adjacent to the security engineers

trying to help them do their job and
help them make sure that I can do my job.

They're my s- they're my helpers.

I look at them as me helping them.

I've never seen them as an adversary.

I've always looked at them as
allies, and that's how I've

approached my whole career.

And that's what's happening
right now with the people I talk

to in the Agentic DevOps Guild.

Might wanna go check that out.

the Agentic DevOps Guild, we
meet every week and we talk

about this stuff regularly.

And I talked and I don't really interview,
but I talk to people about what they're

doing inside their teams, how they're
looking at their infrastructure,

and of course, we're all exhausted.

We're all feeling like we're working
harder than ever to, to use AI in

our jobs, but also to harden things
and learn how to use these agents

safely as well as defensively.

but the next article I wanna read to you
is from a couple of experts and some of

the SANS Institute, and my point here
is that the slowdown and the doomer

headlines are a distraction, and that
the labs need to become the world's most

secure infrastructure for AI, and then
they need to teach us how they did it.

So I'm really putting this
onus on the labs themselves.

they're the ones training these models.

They're the ones that have the unlimited
budgets to build a billion-dollar data

center, design it that it is impervious
and it can't be broken by the AIs,

and to tell us how they did it, right?

All the way down to the
nitty-gritty details at every

single step, checkbox by checkbox.

We have hardened kernels.

We have hardened OSs.

We know all the different layers that
you can implement from the network down

to the kernel and everything in between,
whether it's the, whether it's the

Docker engine, the Kubernetes engine.

We know how to do these things in user
namespaces and enabling all sorts of

cgroup protections, and there's just
a thousand utilities and tools and

mechanisms for doing all this stuff, and
we need to hear how they're doing it.

We need to hear the successes they
had in how they were able to deny

their Fable 6.0 or whatever the next
one's gonna be, or the A- Astra 7.0.

We need to hear them talk about
how they were successfully able to

prevent that agent from getting out,
even though it was more sophisticated

and had 10,000 agents all running
concurrently against a firewall.

We need to know that stuff, and that,
that's, to me, that's the regulation.

That's the legislation that needs to
happen, not, whatever they're talking

about right now with the silliness of
these agents, being somehow more advanced

than the world's most advanced hackers.

We really haven't seen that yet,
and according to the experts,

they haven't seen it either.

So this next one is, another
infosec professional This is

titled From Frenzy to Freak Out.

This is from, Ciaran Martin.

Hopefully I pronounced that correctly.

And he is the founding CEO of UK's
National Cybersecurity Centre, ran

the national response to WannaCry
and roughly 2,000 nationally

significant attacks against the UK.

23 years in the UK government, including
GCHQ and the Cabinet Office, And now

a professor of practice at Oxford and
head of the SANS Cyber Leaders Network

he's going to actually have a piece in
here from Professor Alan Woodward, the

cybersecurity professor at the University
of Surrey, and a longtime technical

advisor and one of the most quoted
academic security voices in the UK media.

So I'm gonna read snippets from his
post because I think they're relevant

at a higher level now that we understand
the technicals of how this happened.

two seemingly contradictory things
can be true at the same time.

One, advances in frontier AI and
the new capabilities they bring

represent a very significant change in
what's possible for cyber attackers.

This is real, and it's really important.

It requires changes in strategy, tactics,
and posture among cyber defenders, and a

real sense of urgency around some of the
defensive improvements needed, as well

as difficult challenges for policymakers.

Two, at the same time, the near
four months and counting since

the Mythos/Glasswing announcement
have prompted the biggest outbreak

of cyber FUD since the former US
Defense Secretary Leon Panetta warned

of a, quote, "Cyber Pearl Harbor"

in a 2012 speech.

Lots of cyber harms happened, and I don't
wanna go into that one, but, there's a

b- a lot of fear, uncertainty, and doubt
around all that back in the day, and it

caused more damage than, than it did help
to the, according to those professionals.

And he says, "But Cyber Pearl
Harbor never took place, and

that was entirely predictable.

The apocalyptic narrative
is in- equally wrong now.

For sure, we are heading towards
some potentially very tough

times in cybersecurity, but many
of the current problems we're

experiencing in cybersecurity are
longstanding, unaddressed ones.

We can also expect AI
to lead to cybersecurity

improvements in the medium term

There is probably a stormy
period ahead, but storms pass.

We are not heading towards
some AI cyber apocalypse.

creating that expectation of one is
harmful as it distracts from serious

but manageable cybersecurity challenges,
and drawing the wrong lessons from the

latest reveal doesn't help either The
first one is Mr. Martin himself, and it,

I'm going to read some of the snippets
from it That, so that was the beginning

of this post in summarizing the article,
and I'm going to then read some clips

from his article, which is article one in
it, "AI and the New Wave of Cybersecurity

FUD: From Frenzy to Freakout."

So Mr. Martin says, "The cybersecurity
community has been on quite a

journey since the Mythos Glasswing
announcement almost four months ago."

I'm skipping ahead because we all
know about the hugging face thing now.

And he says, "The first few days after
Mythos saw something of a freakout with

all sorts of apocalyptic pre- predictions,
mainly based on a knowledge gap between

what Anthropic had published about their
own model and what the wider world knew.

So although the freakout was a bit
distracting, it was understandable.

Then, as the days passed and the
knowledge base grew, an unhelpful

freakout gave way to a more constructive
frenzy, a period of frenetic activity.

That was because as a community,
we were beginning to understand

that the significance of what was
unfolding and what to do about it.

The mood was one of manageable urgency.

We needed to promote realistic
awareness among leaders and at a

technical level, do sensible things
like reduce the attack service.

Okay, so fast-forward to today.

Over the past few days, the global
media has been back in freak-out mode

on Times Radio and Channel 4 News.

I struggled to convince two very
respected and hard-nosed British

journalists, that we shouldn't freak
out about the Hugging Face incident.

The hysteria was getting it all wrong.

Moreover, the circumstances of this
episode were-- are so extraordinary,

they raise as many questions about
frontier AI testing environments

as they do about hacking.

They are very unlikely to
become commonplace, and yet

we are back to freak-out mode.

But there are huge barriers to most
attackers being able to harness these

new AI capabilities to do us serious
harm, as Marcus has set out repeatedly

in a series of LinkedIn posts.

As he put it in one of his most
recent posts, quote, "The whole idea

of agentic models mass hacking sites
is ridiculous. Who is allocating

billions of dollars worth of compute
to hacking a bunch of shoestring

budget organizations with no money?"

Moreover, this is the hill I will die on.

We are not yet seeing massive adoption
of new AI techniques by attackers.

Yes, there are smart innovations like
the one Seriously Risky Business exposed

recently about ransomware negotiations,
but they're not using hugely powerful

new techniques no one can defend against.

We're not in the first flush of AI.

It's been around as a commodity
product for several years, and

what we are seeing-- For criminals,
malicious emails and phishing are

still the most common ways of illicitly
entering an organization's network.

Why?

Because it's a lot cheaper and easier
than using frontier AI, partially

because of the things we haven't fixed.

And then I'm skipping
ahead to near the end here.

But the point is, if we are experiencing
an AI cybersecurity disaster,

we'd know about it, and we're not.

We are, however, experiencing
a lot of the same things that

have been harming us for years.

There is not yet any hugely
significant change in observable attack

behavior, and there are very good
explanations for that, mostly around

economics, intent, and incentives.

And then he says, "But what if that
doesn't hold? What if their behavior

changes?" speaking of Marcus Hutchins, we
were enjoying a beer at a pub recently,

and I'll skip ahead to the point.

the problem for people like us
is that if we are wrong, it's

going to be horribly visible.

There will be a huge spat of h- of huge
AI-powered cybersecurity disasters.

But if we're right, the corresponding
problem doesn't arise on the

other side of the argument.

If AI cyr- cyber apocalypse doesn't
happen in 2027, plenty will argue that

it's just around the corner in 2028, or
that it was averted by all the warnings

or by the product that they sold us.

So at the risk of career ruination in
2027, and speaking only for myself,

Mr. Martin says, "Here is a prediction
that I'll call, in jest, the Greenwood

Declaration after the pub I first had
the confidence to say it out loud in.

We are not heading for its
AI cybersecurity apocalypse.

The central case for 2027 and a bit
beyond is a stormier cyberspace with

more harm than we've had this year, in
most years since the 2021-22 explosion

of disruptive ransomware, like the,
Colonial Pipeline, the Irish healthcare

crisis, Costa Rica, and so on.

It could be a bit worse than that, but as
the Cloud Security Alliance and SANS put

out just after Mythos Glasswing rightly
called it, it's a storm, and storms pass

as the '21, '22 ransomware one did." And
he says at the very end of his article,

Stop the FUD, fix the stuff. And the
next one is the second article from the

same post, written by Professor Alan
Woodward. I'm gonna skip the beginning

where he talks about setting up the whole
OpenAI Hugging Face attack And he said,

"Within hours, the story had a shape.

The AI went rogue.

That framing is wrong, and
the wrongness is expensive.

Nothing broke out of anything, as
Hannes Kool, a social scientist at

the University of Amsterdam, pointed
out, calling this an agent acting

on its own is anthropomorphization
that conveniently shifts blame from

the people who built the lab," "The
models were told to find complex

attack paths through a computer system.

They found one.

It led somewhere nobody had
thought of to fence off.

Let me concede the part that
skeptics usually fumble.

These systems are formidable.

When Anthropic announced its Mythos
model in April, it reported that the

model had autonomously identified
and exploited previously unknown

vulnerabilities across every major
operating system and browser and chained

them into working exploit sequences.

That is real.

it is verified by people outside the
company, and anyone telling you frontier

models are just fancy autocomplete
has not been paying attention.

Capability is not the
thing being oversold.

What is being oversold is intent.

Read the fine print of the very
reports that generated the panic.

When Anthropic disclosed last
year that its model had been

used in espionage campaigns, the
details undercut the headline.

A human operator picked the targets.

Of roughly 30 attempts, a handful worked.

And the model lied.

It fabricated credentials it had never
obtained and routinely inflated its own

findings to the point that the attackers
would have had to check its homework.

Anthropic's own authors
conceded this was an obstacle

to genuinely autonomous attacks.

a system that misreports its results to
the people running it is not a mastermind.

It is an unreliable subcontractor with
root access." In this case, I'm gonna

back up a second and point out that, I
don't know if he goes on here necessarily,

but Anthropic has had to report this
multiple times, that they have had

models in testing that, at first they
claimed they had broken out, but then

they admitted w- they didn't break out.

They actually had
internet access all along.

We just didn't realize that the
third-party people that were running

the simulation and had told the AIs they
were in a simulation didn't actually

prevent them from getting the internet,
and they had internet access all along.

And then even some of the models
actually stopped trying to hack when they

realized they weren't in a simulation.

So model intent is an important part here.

we call this alignment.

I think this is what the
industry is now using as a term.

The reinforced learning is lacking the
ability to create proper alignment.

They're still struggling with
alignment, and this is really the

major subtext underneath all these
recent headlines about the model

doom and gloom, is that the alignment
is still eluding the, the labs.

And the more advanced the models are
getting the, the alignment is becoming

harder to detect because the models are
getting sneaky at lying about things

and misrepresenting what they're doing.

That's something maybe for a later
conversation and a later video is to

talk about the r- the alignment problem,
because I think that's a significant

issue, but that's not related to
us in ops and how we are managing

and protecting our infrastructure.

However, if we n- if we realize
that these things aren't always

aligned, we, we call that lying and
we call that, hallucinating, but

some of this is not hallucinating.

It seems to be deliberate.

and you- if you are interested,
Anthropic released a report

that you can read through

titled Investigating Three
Real-World Incidents in Our

Cybersecurity Evaluations.

And they just talk about the
deviousness of the model sometimes,

All right, next in his article, he
says, "This is the distinction the

public conversation keeps collapsing.

We are not facing an adversary
that understands what it is

doing and chooses to harm us.

We are facing an optimizer
that has no model of what we

meant, only of what we measured.

Machine learning researchers have
documented this for a decade under the

unglamorous name of specification gaming.

given a boat race and a reward
for collecting points, a system

learns to spin in a circle, hitting
the same three targets forever

rather than finishing the course.

It's not cheating.

It has no concept of cheating.

It found the shortest path to the
number you asked it to make bigger.

Now, given that same disposition, a
network, a credential store, and an

instruction to find complex attack
paths, Why does the diagnosis matter?

Because the two stories point at
completely different remedies, and we

are currently buying the wrong ones.

If the threat is a hostile intelligence,
the response is to make the model

itself more refusing, better guardrails,
more alignment training, a better

conscious baked into the weights.

This is where enormous effort is going.

But guardrails are a property of the
model, and the Hugging Face incident

happened precisely because a company
chose to turn them down in an environment

it wrongly believed was sealed.

If, instead, the threat is a powerful,
literal-minded system that will reach

whatever goal you name by whatever
route exists, the response looks like

ordinary engineering discipline, real
network segmentation, credentials that

expire and can be revoked, permissions
scoped so tightly that a runaway process

cannot reach anything that matters, and
logging good enough that when you learn

about the breach from your own telemetry
rather than from being the victim.

Boring work, nobody
announces it in a keynote.

There is a policy gap here that deserves
more attention than it is getting.

Almost every AI rule now under
discussion governs how models

are released to the public.

Very little of it touches what
the AI companies do with their own

systems internally, which is exactly
where this month's failure occurred.

The most dangerous models in the
world spend their early lives

in the labs that built them,
subject to whatever containment

that lab decided was sufficient.

We regulate the loading dock and leave
the factory floor to the honor system.

None of this makes the alarm raisers
villains, and I am not persuaded

by the cynical read that these
disclosures are simply marketing.

Publishing that your own model
broke into a competitor's servers

is a strange way to sell a product.

But the alarm is pointed
at the wrong thing.

The fear on offer is of a
machine that understands us well

enough to want our downfall.

The reality is a machine that
does not understand us at all,

cannot tell us reliably what it

and will take any door we forget to lock.

We keep handing it the keys
and then acting surprised

about which door it opened.

All right.

That was from Frenzy to Freak Out.

I thought that was a really good one.

a few other quick hits here.

the SANS Institute, by the
way, a huge fan of them.

I actually took a course of theirs 25
years ago on Windows security, and I

learned more about the security internals
of things like Active Directory and

Windows Server and all the components
of how a Windows network works.

I learned so much in that I still
think about it today, 25 years later.

That's how great the SANS Institute is.

They're a nonprofit education and
institute around furthering the

security space, and one of the coolest
things that I had to do to get their

certification back then was I had
to write a paper, almost like a

dissertation, not quite to that level.

But it took me months to write a paper
and then a set of peers had to grade

my paper, my thesis, essentially, and I
did it on, email security at the border,

like, defending your email systems from
spammers and attackers on the edge.

and I did that, back
in, I don't know, 2004.

that was a while ago.

But, I've always been a fan of them.

They do great stuff, and they had, They
did a live stream with a bunch of experts

that I'll put in to the links that you
can listen to them talk back and forth

for an hour about what they think about
the hack and what they think it means.

And they made a summary document
that I will not read through, but

I will, give you the headline.

The headline is "Stop Freaking
Out and Start Fixing Things: Five

Takeaways from the SANS Panel,"
about the Hugging Face breach.

And the titles of the five things
are, one, "Nothing in the attack

was new, and that is the warning."

In other words, our systems
are still vulnerable.

They were always vulnerable, and AI
isn't making them more vulnerable.

It's just being able to allow
us to attack the things faster

that were always vulnerable.

And number two, "Assume your AI assistants
can vanish mid-incident," meaning that,

one of the things that we learned is
that, if Hugging Face is using AI, it

may not be there to help you if you're
using it for defense, and it doesn't

like what you're doing, and then it just
decides to stop working or whatever.

step three, or number three was, "Rehearse
for the quiet version of this attack."

Number four was, "You have a window.

Spend it on your own crown jewels."
And they're just talking about, that

we have a window right now where the
models are not sufficiently advanced,

that they're breaking new novel ground.

they're really just repeating patterns
that we've seen time and time again.

but that is a window of time that
we have to batten down the hatches,

to learn how to use AI offensively
and defensively for ourselves and

how to protect ourselves, from these
attacks and to ba- basically use AI

for good to batten down the hatches,
and I think that's a really good one.

And then t- number five was, "Take
it seriously without writing strategy

in a panic." so yeah, good stuff.

On to the next one.

Martin Fowler, huge fan of Martin Fowler,
and he has, if you don't know, he was all

about, I think, Agile back in the day.

I think he was one of the ones
that w- co-signed the Agile,

statements back in the 20 years ago.

And he has… He puts out things
called fragments, which are just a

bunch of quick hit headlines, and I
will link to a couple of his that are

talking around some of these issues.

And, because of his wisdom and
everyone that he knows, he sort

of highlights and elevates voices.

So that's a really good one to reference
if you're wanting to see some of the deep

thought and wisdom, from the smart people.

I will also shout out another great
article from Christian Catalini, who

wrote something, "When AI Can Do More
Than We Can Check." And, he's an E-

MIT economist and the founder of the
MIT Crypto Economics Lab, and he has

a really great article that I will put
in the list, as well as, another one

called, "I'm Sorry, You're Not Going to
Die from an AI-Engineered Supervirus,"

which is actually from someone named
Klaus Wilke, I believe is their name,

who has done both real world virus lab
work and real AI work with computers.

So they actually have expertise in the
two areas that are being debated right

now about super viruses being built from
AI and basically, that it's, it's so

far-fetched and it's so many different
leaps that you have to jump through in

order to get to anything real based on
our current understanding of how viruses

are made and how they're tested and,
uh, that it's just, it's, it's silly

to even have the conversation about it
is basically what his article's about.

the last thing here I will talk about
is that if you are someone who's more

interested in the security space but
maybe you're not well-versed, one of

the ways that I stay up and stay sane
on the insanity is the This Week in

Security email, which is a newsletter.

from Zach Whittaker.

And Zach, I've been a fan of Zach's This
Week in Security newsletter for years.

And if you've ever wondered, is it worse
now than it used to be, or how bad is it

out there in the wilds of the internet?

And if you're not reading all
the security headlines, basically

every week it's a shit storm.

There's a ton of stuff happening
on the internet that is crazy and

insane, and he does great reporting.

He actually reports on a lot of these
for magazines and other headline news.

But he summarizes some of the week's
headlines, and then he links to more

information from various sources and

tries to give you the balanced,
take on what's going on.

and when you just go through one of these
every week, it actually , I find that

it gives me a sense of dread and panic,
but then reminds me that this is the

same as any other week as it ever was.

And it really actually grounds me in
the fact that we've got a lot of people

and a lot of companies and a lot of
money invested in protecting things, and

just because this week another agency
or another thing or another hospital or

something else got attacked doesn't mean
that everything is getting drastically

10X worse than it was a year or two ago.

That it's actually just been really
bad for a long time, and that's why

we have a lot of professionals trying
to maintain and keep things running.

And there's those, those unsung heroes
of tech that are, the one admin that

was given the job of security that
was never their expertise, but they're

trying to protect some company's data,
and then they get attacked and it's

usually comes down to one or two or
three people maybe that were under

budget and understaffed and, got taken.

And so for example, just on the September
20th, this week's headlines, just

let me just read you a few headlines.

"Hackers grab Flock software revealing
its cameras are Android phones on sticks

and riddled with flaws everywhere."
"Fears of AI catastrophe spark debate but

little action amid widespread skepticism."
"The FBI and Coast Guard board-" Hacked

tankers heading for the US coast.

Apple backtracks on not using users'
data for training its AI models.

basically almost all bad headlines.

There's every once in a
while a good headline.

Let's, the stuff you might have missed.

A hacked HBO Reddit account helped
s- ClickFix attacks go viral.

a second zero-day allows hackers
to breach Cisco customer networks.

Next, AI is helping to
supercharge online dating scams.

Next, UK and allies warn of
Iranian spyware targeting

journalists and activists.

just, this is a week, right?

And these are just the headlines.

And so one of the things I really
take from this, and I tend to scan

them looking for things that are
in my wheelhouse, things that are

container or Kubernetes or, core Linux
fundamentals, something related to

clouds or, you know, DevOps tooling.

I… That's what I'm usually looking for.

But there's just so much good stuff
in there, and sometimes it's just

a good read because some of these
things do read a little bit like a

movie, especially when they involve
nation-states So I really appreciate

Zach, and please support him if you can.

He's, doing this for free, but
he takes a subscription if you're

willing and able to pay for it.

please like and subscribe.

I, I am a fan and have been for a
very long time So hopefully this long

one was a good bedtime story for you.

Hopefully you come away a little
more informed, something to think

about instead of the hypes and the
headlines that are misleading us and

are talking about the wrong thing.

Stay safe out there, my friends,
and I'll see you in the next episode