AI-Native Engineering Culture from a CTO

Michael Rollins: The big milestones
for us were we started from, you

know, everybody's using AI to code.

We moved into, " Hey, "When's the last
time any of y'all actually looked at

the code?" And all my team was like,
like, everybody didn't wanna get caught.

I was like, "Oh, 'cause I haven't
looked at it in months." Yeah,

and they were like, " Oh, God.

Okay.

Phew.

Thank you.

Thank you."

Bret Fisher: imposter syndrome that
jumps out when the junior-est member

of the team posts something in the
engineering Slack that calls out

something the most advanced engineer's
done, and you're like, " Oh, God, I'm

not even a real engineer." Like, how…

And then you're like, "Oh, wait, you
had Claude look at that, didn't you?"

And they're like, "Actually, no.

It was Codex." You know?

Bret: Welcome back to the Agentic DevOps
podcast, and I'm your host Bret Fisher.

I have my friend Mike Rollins on we went
to lunch, and the way he talked about

his AI-first engineering or agentic
engineering or agentic dev, what-

whatever you wanna call this thing,
where you're throwing tokens mostly

and not writing anything anymore, and
you're not even looking at code anymore.

And he runs a team as a CTO, a small
team in a startup, and I wanted to

talk about what that life is like.

'Cause a lot of us, you know, we-
we're in enterprise or we're, I

would call, like, legacy teams
where, we're toying around with AI.

We're certainly using it for code,
but we're not necessarily riddling

the entire pipeline of the software
life cycle with everything AI so

that we never see code again and that
everything just goes as fast as possible.

Like, most of us are somewhere in there
in the middle or near the beginning,

I think, and certainly with a lot of
the people I talk to, where they aren't

necessarily able to throw out the playbook
that we've had for the last 20 years on

how exactly a team uses Git storage and
automated deployments and test-driven

development and all the things that
maybe, were hot topics before AI, right?

So Mike and I go through a pretty
lengthy conversation, but I wanted to

get as detailed as possible to talk
about the culture, the pros and cons.

Like, what are the downsides of this?

Tell me some horror stories of how this
can go wrong in so many ways and what it's

like trying to push the envelope, being on
the bleeding edge of AI-first development.

So let's get into it

Bret Fisher: Mike, welcome to the show.

t- tell the audience, who you
are and what you do for a living.

Michael Rollins: So, uh,
my name is Mike Rollins.

the majority of my career
has been in Silicon Valley.

back in '22, Our company was getting
bought by Twitter, and then this

dude named Elon jumps in, and we were
no longer being bought by Twitter.

And as a result, I went on this multi-year
escapade of, I worked for a healthcare

startup, and then after the health- I left
the healthcare startup, or I was asked

to leave the healthcare startup, because
it turns out if you ask very difficult

questions pointedly, people don't like it.

I ended up, going o- out on my
own, and it was terrifying, scary.

I mean, from about 2023 until really
November of last year, I was a

contractor, and I did a bunch of stuff.

I was the founding engineer
for multiple projects.

I was this… I've been the CTO
now of three different companies,

though mostly fractional.

And I have at some point, the internal
clock in my head is always ticking, and

I feel this constant need to go faster.

And so as I was building things as
typically the only engineer, the clock in

my head is just faster, faster, faster.

And so when AI became, like, a thing,
meaning back in the latter half

of 2024 and then especially at the
beginning of 2025, myself and a few

other folks, a guy named Thanos, who
he and I have a podcast together that

we're kicking back off called Will
AI Snap That, we both just dove in.

And what's funny is that since
about really the entry of Claude

Code into the market, I have
been doing AI first engineering.

And what I had noticed is that by the
time we hit about May or June of last

year, I was no longer writing code.

And then right around November
of this year, I realized I was

no longer reading code And it
was this, shocking realization.

And so I basically dove headfirst into AI
j- and, you know, high risk, high reward.

The risk here is that my
family can't eat, right?

The reward is that I figure out a way
to do engineering that other people

don't know, and it's what we did.

And so nowadays I'm at this
company called, Relefy and

then also Family Digital.

I'm CTO of both.

And I have this team of four other
engineers, and really truthfully,

only one of them, well, one of them
is a designer that we've turned into

a front-end engineer, and none of
us read code, none of us write code.

we ship… We've built this agentic
system that's built from the ground

up for sharing, and it is kind of,
it's built at right angles to the way

everybody else is doing things, and we
are moving so fast, it's terrifying.

But the clock continues to tick, and I
continue to think, "How do we go faster,

faster, faster, faster?" But I, dude,
never in my life… I got to write mobile.

In 2011, I started writing code for iOS
and then Android, and mobile felt amazing.

It felt like a freaking rocket ship.

Everything changed so fast because there
was a new phone about every three months.

And now it's dude, didn't we just see a
new Frontier model a week and a half ago?

'Cause we did, and it's just unbelievable.

So anyway, that's how I ended up here.

Bret Fisher: Yeah, that's a great story.

And, I was on the hardware side
of mobile, not the software side.

So, yeah, it has become very apparent to
me in the last few years that it's like

AI, that's a good analogy of like AI feels
like it's kind of replaced mobile as the

exciting tech where that was really…

I mean, mobile kind of came around a
little bit a few years after cloud,

but cloud I feel like was pretty well
understood within four or five years and

then we had mobile and that was exciting,
and then there was like this lull.

I kind of feel like 2018 to 2022
was kind of this lull of, we

Michael Rollins: Yes, it was.

Bret Fisher: yeah, we were
all sort of exhibiting the

Michael Rollins: was
supposed to be crypto.

Bret Fisher: Oh, that's true.

Yeah, the vacuum of crypto, which,
you know, and I don't know anyone

that touches it with a 10-foot pole.

So, yeah, that was a thing, but
not necessarily in my ecosystem.

So we all were talking about
how isn't it great that Docker

and Kubernetes are boring?

Isn't it great that like microservices
and distributed services and cloud

tech are all kind of boring and, we can
just focus on the work and, enhancing

our craft and getting better at it?

And then, you know, ChatGPT shows
up and then we're all copy-pasting.

So I wanted to get into some of
the nerdy details of… it sounds

like it's a relatively small team,
so a total of five it sounds like.

There's a total of five of you.

So does that mean you have five people
making PRs into, presumably GitHub?

Michael Rollins: Mm-hmm.

Bret Fisher: And how do you go
about even starting something?

Is it like an individual
thing where you start an idea?

you mentioned this idea of concepts first
before ideas or something like that.

Bret AI July 2025: Have you been
adding text-to-speech in your apps?

One of my late-night projects
this year was creating a personal

podcast feed so I could listen to
the changelog of my favorite open

source projects while I do my chores.

I needed a text-to-speech API to do that.

I also heard from a colleague this
week who built an app for his kids to

turn their e-books into audiobooks, and
he's using the Speechify AI developer

platform to do that text-to-speech

Speechify AI is a developer platform
for real-time text-to-speech, and

they make their own voice models.

Their Simba 3.2 model ranked number one
on the Artificial Analysis Benchmark

for conversational voice quality.

They've got a free plan, streaming
and REST APIs, TypeScript and Python

SDKs, and if you wanna get fancy, you
can use their SSML metadata support

to add emotion into the speech output

Whether you're shipping a weekend project
or production voice agents at work,

you can start free at speechify.ai.

The link is in the show notes.

Thanks so much to Speechify AI
for sponsoring this episode.

Michael Rollins: well, let
me start at the end for you

Bret Fisher: Okay

Michael Rollins: because this is typically
the framing… This is the framing I've

been aiming for, and we're almost there.

What does it take for me to allow
an agent to pick up something, build

it, test it, commit it, open the PR,
make sure everything runs, listen to

all the feedback, merge it to main,
and deploy it, and a human is never

involved, and as a matter of fact, a
human may never know that it happened?

What do you have to do to make that real?

And risk is really the big asterisk there.

And I, you know, as the clock is
ticking in my head, I, I make decisions

based on a risk assessment, right?

What is the likelihood of failure
versus what is the outcome if we fail?

And what you find is that even if the
likelihood of failure is high, but the

risk associated with the actual failure
is low, you may choose to take the gamble.

Conversely, if the likelihood of
failure is very low, but the risk

is extraordinarily high, you're
pro- you may not take that leap.

It's not a gamble.

It's almost a guarantee, but you know
that if you hit the Enter on this

select statement in the production
database, and it goes sideways, like

the failure mode could be the database
disappears, and then so does the company.

And that's one of those things
where it's like I've made that

decision and had it go bad, right?

I've made… I have wiped
the company database

Bret Fisher: Oh, are you the one that's
like on the tip of the spear just so the

rest of us can learn from your failures?

Michael Rollins: when I was at Outlier,
we did machine learning, over time

series, and this was in 2018 to 2022.

That was the company that was
gonna get bought by, Twitter.

Bret Fisher: Yeah

Michael Rollins: I was on call for three
month- for three year straight, and…

I wrote direct queries in production
databases constantly, ' cause it

was the only way that we could keep
the system up and running sometimes.

And I would wake up every morning and I
would rebuild the system, and there were

a couple times that, boy, did I that up.

even in a couple, not the current company,
but the previous one, there were a

couple times I just, I wasn't thinking.

I knew I was doing something a little
bit risky, and I did it anyways, and

had to go desperately hope that we
had a recovery copy stashed away.

and we did.

And so I have legitimately,
I've blown it all up.

I'm very risk-tolerant in that aspect,
but the reason I'm risk-tolerant is

I fixed it And so when I look at…
let me give this as an example.

we had an incident occur a few
weeks ago, and all of a sudden I

realized that we could no longer
upload files to our system.

And now keep in mind,
it's an agentic system.

You create a bunch of agents.

They can all talk to each other.

It's a lot like OpenClaw, except
we were doing it first, and we

built it to be enterprise safe.

And part of that is, you have to upload
things, and they get shoved into the RAG

database, and they get shoved into the
S3 file system, and I tried to move a

bunch of things around, and I couldn't.

I was like, "What is going on?" Now,
I'm at my daughter's dance competition.

I don't even have my computer with me.

I'm the only engineer that
can fix this at this moment.

And so I'm actually on my phone,
so I swipe over to Claude Code.

I have a session running
on my machine at work.

I tap into it, and I say, "I want
you to you know, clear your context.

I want you to pull main.

I want you to get to a new working
tree, and I want you to help me diagnose

what this problem is." Within about
four minutes, I knew that the disks

that were, serving the ECS, the EC2
or ECS images were full, ' cause we

had not been pruning them after we
would deploy. And I had Claude go in,

'cause it also has my AWS credentials
as part of it, and said, " Go fix it."

It went ahead and fixed it.

I didn't even have to restart the system.

I had the whole thing back up and
running correctly in five minutes.

Bret Fisher: Yeah.

Michael Rollins: The cost

Bret Fisher: basically, or
what- whatever, whatever

Michael Rollins: yeah, whatever the
container service equivalent was.

But that was it, right?

And well, the risk was actually
pretty low, and the likelihood of

failure was almost minuscule because
I knew that I could do everything

from the command line on my machine
through the app that was on my phone.

And so the risk profiles of
everything are just different now.

so when I sit down and I think about
what does it take to allow an agent

to write code and ship it without
me ever knowing, there's an inherent

assumption built in that any failure
state is something I can recover from

in a relatively short amount of time.

And now there's an extraordinary number
of assumptions that are involved there,

and it's not like this is YOLO, right?

Our build infrastructure, our GitHub
actions, everything that happens

post-commit is methodical, it's intense.

going back to what was your
initial question, we don't

do tickets, we do features.

And oftentimes because of the
speed with which we move, we…

There's not even time to build a UI.

Our designer is often
catching up to engineering.

We have officially broken the
entire organization because

engineering moves faster than sales.

And that is a wild thing to
deal with on a business level.

like

Bret Fisher: that's when
people get fired, right?

Because there's too many engineers for the
sales team and… But that so rarely ever

happened 'cause there was always a gap.

Michael Rollins: Yeah.

I mean, the, typically the lead time
from sales to engineering is six months.

Bret Fisher: Yeah.

What do I tell the customer?

We need them,

Michael Rollins: Yeah, what
do I tell the customer?

Bret Fisher: we need the feature." Yeah.

Michael Rollins: yeah.

And I'm like, " It's already there.

Just sign the contract." they
were like, "It wasn't there two

days ago." I was like, "I know.

We shipped it."

Bret Fisher: They're so not used to that.

Yeah, especially if they're old school
and they're used to the giant lags

Michael Rollins: Yeah.

Well, and, you know, so the, one
thing to say is the mentality of

engineers has to be totally different.

My engineers have to understand
the business, and I don't let my

engineers just focus on tickets.

As a matter of fact, that's
an amazing wayl- way to fail

catastrophically at this point.

instead, I say, " Here's a thing,
and I want you to understand

it and I want you to do it."

And the engineers that are working for
me at this point, which by the way, none

of them have professional experience
before they started working with me,

these folks are making really strong
judgment calls, and that's all I want.

I want good judgment, right?

The model can, and the harness can
figure out how to write the code.

Bret Fisher: So is there a large
part, is a portion of your training

or cross-training in the team
helping everyone with judgment?

is that become an increased thing?

'cause since you're going so much faster,
you, there's less time to talk about

things, so people have to be able to
make more decisions on their own, which

means they need more agency, which, yeah.

Yeah.

Michael Rollins: There's one amazing
way to let people build better

judgment, and you let them fail,

Bret Fisher: Mm-hmm.

Michael Rollins: we shipped something
this morning, and, I had actually

started work on it and then had
to move on to something different.

And so Miko, who's a official… He would
be classified a junior engineer, right?

Miko will have been out of college for
approximately a year in the next few days.

Miko's making decisions at a senior level,
but he shipped something this morning,

and he overlooked something pretty major.

Pretty major.

Yeah.

And I halfway knew it was gonna
happen, and I let it ship anyways.

And, that is one of the things
that I allow my team to fail.

I will let the problem hit production, is
once again, it's a risk assessment, right?

Because you've done it and I've done it.

There's nothing quite exhil- as
exhilarating as realizing you

just shipped a nasty bug, man.

everything is being destroyed,
and it's your fault.

Bret Fisher: It, that is a
permanently ingrained memory that

you cannot escape and forget.

it becomes a part of who you are, of
your identity whenever you do that,

Michael Rollins: and so my
leadership style actually

relies pretty heavily on that.

I know that I'm tossing people
in over their heads, and I do.

And, but I also tell them, " Hey, listen,
you're probably gonna fuck this up.

Bret Fisher: Yeah

Michael Rollins: And when
you do, I'll be here.

You just come find me.

There's no blame.

I know you're in over your head.

You know you're in over your head, and
I happen to know that there's sharks

and they have lasers and you don't yet."

Bret Fisher: Yeah.

Michael Rollins: so,

Bret Fisher: it's challenging.

so yeah, you're giving… You're
challenging people to grow and rather

than coddling them through every
mistake that they'll never remember,

because I'm sure if you told them,
walked by the desk or, the equivalent,

the analogy of walking by the desk
and going, "Oh, what you working on?

Oh yeah, you probably wanna check
this so that doesn't fail 'cause that

would've totally broke production."
They may not even remember that in

five months or, you know, two months.

So, but basically the boss just told
them a rule, but that's, they don't

really maybe know the deep why.

I'm gonna ask real quick.

is the assumption here that you have
solid, It sounds like it got past

testing in this scenario, so that
means that the testing was lacking

for that particular assumption.

And then

Michael Rollins: it wasn't,

Bret Fisher: also assume, by the
way, that production can, fail back?

do you need… In order to give you
the confidence that this isn't going

to ruin the company, do you have
to have, some other structures to

be able to roll back automatically?

Or, is there stuff like that
in play that you're relying on?

Michael Rollins: So in this
instance, it actually wasn't a bug.

It was a missed part of the feature.

so this was a little bit unique.

But honestly, I mean, that's okay.

Here's the interesting thing.

It forced us into the decision of
whether or not we wanted to address

it, and we decided not to address it.

And yeah, you end up ha- you
find yourself in these situations

sometimes where because you can't
necessarily cover all your bases,

you find yourself making decisions
you might not have made initially.

In this instance, we were basically
changing the ownership of a certain

type of object within an agent.

So previously, it had belonged
to the user, and we realized it

needed to belong to the agent.

And so we fixed that, right?

Now all of a sudden these
things belong to the agent.

But there were a bunch of them, and they
previously belonged to the user, so we had

to make the user's core agent own them.

But if you were using a bunch of
these things in another agent, they

were no longer there And it's "Oh,
there's no elegant solution for this.

So how do we get things from the
PR to production, I think is a

great place to really hone in.

And the first thing you gotta know is
that whenever our, in our environment,

whenever you issue a PR, we stand
up a copy of prod for the whole PR.

… it is a somewhat expensive way to
run things, and it also catches

so many things immediately.

now at the same time that we're
standing up that copy of production,

and this includes the database.

This is everything, right?

It is a true copy of production.

we have just a boatload of tests
and checks and all kinds of things.

I think last time I looked,
there's 15 or 16 different

workflows that get kicked off.

we've got multiple, AI, audits.

We've got, s- I mean, look, we have tests
running, I don't even know what they are.

They just run.

Like, Claude was like, "I don't know, we
should do this." I was like, "Excellent.

Let's make that happen." now it took
me about a week and a half to literally

tear our old testing infrastructure
out and put this new one in.

but it's ratcheting.

We have rough running to make
sure that things continue to get

more and more, you know, covered.

We have unit tests, we have end-to-end
tests, we have Playwright tests.

We have, a bunch of, we have linters that
run, even though I think they're useless.

and so, we catch a lot of things
before they even hit, before the

production, before the, that we
call them PR environments, before

the PR environments are built.

We know there are problems often.

One of them is, we double-check
that we don't end up with a,

too many Alembic heads, right?

I hate Alembic.

It is literally one of the most
awful pieces of technology on the

planet if you don't use it right.

And so we've been forced to use it right.

Alembic does our database
migrations, by the way.

So that's the first
line of defense, right?

now I will take one step back and say
that every single engineer can also

run the entire stack on their machine
That's-- You have to start there.

If the engineer cannot…
Yeah, table stakes.

you know, everything in a way,
prod-production is a copy of our local

environment because even though we don't
have the data, like I'm not gonna let

you put production environment data on
your environment locally, you have, like,

all the Docker images that get built on
your local also get built with different

environment variables in production.

And so you have to be able
to run everything locally.

We also make sure that the model knows,
"Hey, you need to make sure that there's

unit test coverage for everything you
do. You need to make sure that you're

covering, you know, Playwright end-to-end
tests." I'm very, very big on… Now,

this is because we're primarily a web app,
but I've also done it with mobile now.

I want the model to run the actual
app locally and double-check that

everything it built actually works.

And so it's not just a set of tests
locally, it's also, " Hey, I want you

to fire up an, a Playwright MCP session,
and I want you to click around and

double-check that this actually does
what you say it does." hallucinations

are real, and the only way to really
catch them so that they don't become

catastrophic is to literally make the
model prove that it's not a hallucination.

so that's our testing infrastructure.

it's the best I've ever seen

Bret Fisher: what are these, what are the,
are these tests, are you using Makefiles?

are you g- you got, are you using
testing frameworks that are,

not frameworks necessarily, but
trust testing tooling that is, y-

Michael Rollins: we use Jest for
TypeScript, and then we use, I don't know,

one of the Python… Our back- our entire
system is just TypeScript and Python.

Bret Fisher: I'm a big GitHub Actions fan.

I teach GitHub Actions courses.

Michael Rollins: GitHub Actions.

Everything's based on the GitHub Actions.

Bret Fisher: and GitHub Actions
locally is often problematic

and difficult, and I generally
don't use it even though we have,

Michael Rollins: we don't,
I don't run actions locally.

Yeah, I, yeah, we don't
run actions locally.

Bret Fisher: I tend to throw stuff in
the agents file, like always run these

three linters and then always make sure
they're in the, make a make file so that

everything you do is through a make file.

Because I don't know if this is
advice or a, the way that I do it, and

I'd love for you to… your opinion
on a lot of the things I'm doing.

because often, if I go fast
enough, it's making so much stuff.

I built a Mac app over the last month, a
recording app because I'm so particular

that the thousand different options for
recording screen and video is not good

enough for me, so I had to make my own.

Michael Rollins: I agree 100%.

Bret Fisher: and as a part of it,
it started to have all sorts, like

I need to, it needs to do a dev
build, and it's got a prod build,

and it copies it into my, this place.

And so I started to say, "Hey, look,
you're just writing a lot of commands

for me that I don't even know what
you're doing. Could you make sure that

all that is always in a make file?"

And I made that a rule in my agents
file, and it started doing that.

And then I said, "Okay, always m- run
these linters when you're done at time,

at every time you edit, always run these
linters, always use this make file."

And 'cause I started to realize that I
couldn't do anything without it, right?

I couldn't sit at the command
line and know how to build my app

because I'm not a SwiftUI guy.

Michael Rollins: Mm-hmm.

Bret Fisher: and so I've started
to the point where I, those are

almost table stakes enforcements
for all of my projects, where I

want the agent locally to behave
a certain way, like running tests.

So the challenge is, because I'm using
GitHub Actions, I end up with it just

writing, whether it's a compose file
or a make file or, you know, whatever.

I'm always curious what other teams
are doing since I feel like there's

this pull to pull the testing
into wherever the agent's living.

So if you're running a harness locally,
you probably need to run the test locally

because the agent needs to run them,
unless you plan on going to get a new

coffee and you just tell the agent,
please run all these in GitHub Actions

and let me know when they're done."

And then you have to wait 20 minutes
because, or 30 minutes or longer depending

on your test, harnesses and stuff and,
So I was just curious what you do.

Michael Rollins: I actually have, it's
publicly available on my GitHub, if

you wanna go see how I start a project.

this is it.

It even has a skill that you can use

so I call this the DeltaV Launchpad.

it has a Claude skill that'll
basically set this up for you.

it has a Make file.

n- I do everything with Make.

I'm like you, I got completely just
dead tired of Okay, go do this again."

Goes-- Finally, it's just "Go use the
Make file." Make files, you know, the

Docker configs are all, compose files.

it's all pretty standard stuff.

this is all-- None of
this is rocket science.

we all know how this is built.

Bret Fisher: Yeah

Michael Rollins: And so
I just picked something

Bret Fisher: yeah, I think a lot of
the people in the my guild, the Agentic

DevOps Guild, th-there, you know, there's
a wide range of skillsets in there.

There's people that are,
like, very new to this.

They're dipping their toes in

the Copilot or Cursor, maybe Claude Code.

there's people all on the, all the
way on the other end that are, you

know, they're using s- they're using
plugin marketplaces to manage their

skill deployments to their teams and
centralizing the skill management

as, while at the same time they're l-
adding more and more agents to their

post-commit workflows to self-improve
things, on occasion, which i-is a topic

I wanna get into at some point about.

the idea that once you've sort
of matured yourself to, you know,

you've got your agents reviewing PRs.

You've got sort of like the t- the
table stakes of agents post-commit,

where you've got it, you've
got agents looking at your PRs.

You've got agents, maybe you've,
you're lucky enough and you've got

Michael Rollins: Why, but…

Bret Fisher: you troubleshoot or something

Michael Rollins: Yeah, but here's one of
the questions everybody should really ask.

Do we have a way to do this in code?

' Cause if we do, we should do it in code.

Bret Fisher: Yeah.

Michael Rollins: It's funny, I talk to
people sometimes, they're like, "I wanna

build an app." I'm like, "All right.

Tell me about it." And this one
guy told me about, he runs his pool

business on an Excel spreadsheet.

He was like, "Do you think I can
turn this into an app?" I was like,

"Absolutely." He goes, "Where should
I add AI?" And I was like, " Nowhere."

it runs on an Excel spreadsheet.

It's deterministic.

Do you have anywhere in this
process where you want a

non-deterministic choice to be made?

' Cause if the answer is
no, just run the code.

but yeah, I mean, look, I tend to
drive certain types of engineers

insane because I kind of don't care.

You know, you're like, "Well,
what are you using for your

testing?" I'm like, "I don't know.

We're using something.

I know there's tests

there." Right Who's looked at your
tests to make sure that they're

valid?" " Nobody." " How do you know
your tests are good?" "Well, I have

about 80 other things that check." Right?

And so, I don't dictate to
my team the tools, right?

I do say, "You're gonna
use Docker," right?

"You're gonna use Docker Compose.
You're gonna use this Makefile."

But that's more just
for, efficiency's sake.

you know, one of my engineers, he's
way smarter than I am, he uses Codex.

I think it's a terrible idea, but
he uses Codex, and we fight about

whether or not ChatGPT or Claude are
better on a daily basis, and I tell

him he's still wrong, but whatever.

though it is funny, we get to watch
the models leapfrog each other.

I really don't care what the tools are
as long as they get the job done, and

that means I also change the tools often.

Bret Fisher: Well, no, but I mean,
it brings up a good point that,

sometimes the reason I ask these
questions is because it, in the, in

the question of w- you know, think
of some of the things you're using in

CI and just rattle off some toolings
because we can be vague and hand-wavy

about a lot of these concepts that…

But I always like to get
into the nerd details.

But the reason I ask that is because
often the answer is, yeah, Docker Compose

files and Makefiles and like in other
words, there's nothing new in there.

There, it is exactly what it always was.

I personally, don't, I don't know
if this is the general advice you

would give, but I personally advise
a team to guide their AI to make the

choices on the toolings that they
know rather than just let it make…

'Cause in, in theory, it, if you
asked an AI before you greenfield

something what language it thinks
it would use, it might, it probably

will prefer Python or TypeScript.

It, but it might make a bold
choice, and that doesn't mean

it's right for your team.

And so people I think are sometimes,
maybe not so much anymore, but I think

they were looking at this at "Is this
an opportunity for us to reinvent

the wheel in a lot of our tooling?"

And my argument to them is n-
no, because at any point, look,

the model uptimes aren't 100%.

So if at any point you're stuck and you
have to read something, it, wouldn't

it be nice if it was in a thing you
knew in- instead of some completely

foreign… Like, I'm not gonna have
my AI building a Podman interface to

my containers when I'm the Docker guy.

So if it starts to do that, I'm
gonna guide it back to the places,

even though I may never, ever look
at that Dockerfile after the first

creation of a, the, of the PR.

and I don't know if you have standards
like that, but I've always been imagining

that teams need to actually start, even
polyglot teams, they need to kind of

settle in and say, "Look, these are
the languages we would really prefer

you stick with in AI and not just,
don't be COBOLing our production."

Michael Rollins: I've got this Instagram
channel that seems to be blowing up,

and I piss engineers off all the time
' cause I say things like, "You shouldn't

be reading code, and you shouldn't be
writing code." And s- there's quite a few

engineers that are very, very literal.

the other, another one I had recently
is like, "Nah, just make sure your tests

run. It kind of doesn't matter what they
are." And, but I had a few people get in,

like really get nasty arguing with me,

Bret Fisher: Yeah.

Messing with their identity, bro.

Michael Rollins: No, no, they were right."
They were right because my assumption

was different. So I've had a couple
people come in and say, "This is insane.

How can you do this?

It's too risky," da, da, da, da, da.

And I've had a lot of
people come in and say that.

Well, then it turns out that
they're building embedded systems.

Bret Fisher: Hmm.

Michael Rollins: They're in healthcare,
they're in banking, they're in

some kind of high security field.

I'm like, " Yeah, don't listen to me."

Bret Fisher: Right

Michael Rollins: But if you're building
web apps, if you're building mobile that

is just an extension of the web app,

Bret Fisher: Yeah.

Michael Rollins: it's already been done.

Just use the standard.

You don't have to reinvent that wheel,

Bret Fisher: you have a very valid
point about your assumptions because,

you know, you both of us op- have
been operating… I mean, I've been

in cloud native for over a decade.

you know, the h- the things that we
learned about distributed computing,

microservice architecture, you know, API
contracts, like all the sort of the things

that kind of evolved over the last 15
years, like I've been living and breathing

that at the near tip of the spear.

So when I meet a team that they have 3%
test coverage, they're not using a type

safe or memory safe language, they may
be YOLOing some old PHP stuff and they've

got a very monolithic approach, and then
they're like, " How-- We can't do this.

We're having all kinds of production
failures." and I look at them and I

would evaluate it and say, "Yeah, that's
because…" I love this phrase lately,

is that AI AI just amplifies the team
culture, and your team culture is of 2010.

You need to modernize a lot of
your culture before you go fast.

Like you can't go fast without
the safety, and the safety

was never there to begin with.

And so I do run into that
on occasion with teams.

And there's surprisingly a number
of teams, maybe not in embedded or

healthcare, but like a surprising
number of teams that have very large,

important apps that I get c- to consult
with, and I cannot believe the, I

would call it the level of immaturity
that the code base and the team has.

And 'cause we don't have accreditations
in, in, in our business, right?

We don't have plumber certifications.

So like I can't look at them and assume
that they know all these things and

that they're doing all these behaviors
that I've been doing for so long.

Michael Rollins: I have been
able to look under the hood at

multiple Silicon Valley giants.

Bret Fisher: Hmm.

Michael Rollins: It's all
shit all the way down.

Like every single engineering team
is "We have the worst stack ever.

All of our code is crap." And it's
"No, your code's fine." there's really

only one measure that matters when it
comes to talking about your product.

Does it work?

A follow-up question is,
does it work all the time?

Bret Fisher: Mm-hmm.

Michael Rollins: If the answer to those
two things is yes, it doesn't matter.

Now, there's a third implication now
recently, especially with a bunch of

the Mythos stuff, which is it secure?

And that's always been important,
and it's always been relatively easy

to exploit sloppy stacks, right?

Bret Fisher: By the way, Sloppy Stack
is the name of a future podcast.

Michael Rollins: Oh my God, yes.

That's, that's really… Yeah.

I love, I love accidentally, yeah,
yeah, break down of your sloppy stack.

Now we're gonna enter it.

It doesn't, it also sounds really gross.

whether or not your architecture sucks,
whether or not your stack sucks, whether

or not anything sucks, there's only
one measure that you have to apply

to it, which is does it work, right?

And now that d- decomposes
fast into tiny little things.

but if it works and it's built
on ancient PHP, maybe there's

not a reason to fix it, right?

Bret Fisher: I have
another, follow-on question.

So we, we're seeing a lot in the
news around, the general degradation

of code quality as people onboard
AI, and I understand how it's

happening, and I don't understand
why it's a mystery on how to fix it.

Because it feels, at least in my
experience, if I have all these review

agents and PRs and I'm using I've got
a series of skills, that are auditing,

security audits, uh concurrency audits,
performance audits, you know, these

are things that used to be very hard
for engineers to do because it required

a lot of manpower, and we would often
skip them, especially when budgets were

tight and project deadlines were close.

and continual documentation updates, I'm
just thinking, going down the list of the

things that I can now have agents burn a
bunch of tokens on, to me, it doesn't make

sense that the code quality would go down.

It feels like to me that, with enough
gates that I'm triggering AIs in, that

the code quality would, in terms of
maybe not my impression of how the

good code quality is, but at least
the way it performs and behaves in

production, feels like it should at
least stay the same, if not go up.

Is that what you're seeing?

Do you see that or do
you see the opposite?

am I wrong?

Or

Michael Rollins: I don't read the code.

Why do I care?

Bret Fisher: Yeah, well, but
you see the outcome of the code.

So obviously,

Michael Rollins: see
the outcome of the code.

Bret Fisher: So if your production,
if you were five nines before and

you're four nines now, you would notice
that, assuming you're measuring it.

But,

Michael Rollins: yeah, I mean,
listen, we're not five nines.

Like I was… We're close.

we're much closer now than
we were two months ago.

you know, it used to be I had to
take about 10 to 15 minutes of

downtime to d- per any release
because of the way our stack is

actually built, and because
of the way that we release.

But I've since fixed that.

We don't have downtime.

as a matter of fact, I mean, we probably
are close to something like five nines.

But the question is like, Well, let
me, let me turn this back onto you.

Why does code quality
matter, and what is it?

Bret Fisher: Well,

Michael Rollins: Why are you focused

Bret Fisher: that's a question.

I would say, I- what I'm specifically
talking about is failure rates

in production as a DORA metric.

that's-- and this is the problem.

This is my assumption, is when I
think of code quality, it isn't

whether you tabbed or spaced.

It isn't whether you followed the black
linter on your, you know, whatever, spec.

it is about when you got to production,
in an enterprise there will be a

security team that does a production
analysis of what's going on, and they'll

find more flags than previously, or
you'll start to have more outages

or customer complaints than before.

And so these metrics, according to the
mark- the news, the media, you know, these

things are going up for teams that are
going too fast in AI, and that's, that--

Some are deriving that as the conclusion
means humans still have to read the code.

I don't necessarily take that as
an outcome of that assumption.

I take it as, no, you need to put more,
you need to put more rigor into the AI

process, not back up to the world where
we were all reading every line of code.

So I was curious on your take.

Michael Rollins: it would be
interesting to go back to when

the Agile Manifesto was written.

Everybody had done Waterfall to that
point, and all of a sudden somebody comes

out and they say, "This is stupid. We
should, change as fast as we can." Right?

"We shouldn't plan for
a whole freaking year.

That's insanity.

What we should do instead is we should,
take stock of where we are and take

stock of where we wanna be in two weeks
and get there." I am willing to bet that

the performance of Agile at the very
beginning was much worse than Waterfall.

Nobody had done it.

Bret Fisher: Right

Michael Rollins: Everybody
was trying to figure it out.

This is worse, right?

Bret Fisher: This is,
this is that on steroids.

Michael Rollins: Yeah.

I look at the things we are doing now
and on my team that we weren't doing

two months ago, and I sweat Right?

But you know what?

My release rate, the amount of code that
has landed is hockey sticking, and I've

been forced into what I'm doing now.

I would be willing to bet
that we are shipping more bugs

than I wish we were shipping.

I don't think we're shipping the same
bug twice, at least not one that,

like… Have we had regressions?

Absolutely.

have we had regressions in this changeover
period from the old way we were doing it

until the new way we're, we've done it?

Yes, 100%.

I don't know that we've shipped
a regression since we started on

whatever this new thing is that
we're s- that we've been doing.

Now granted, we've only been
doing it for three weeks.

Bret Fisher: Yeah.

Is there like a hot tip on exactly,
do you add these to a skill?

Do you add just, do you add
new tests for every failure?

Like

Michael Rollins: we have to be
absolutely obsessive about tests.

Bret Fisher: Yeah.

Michael Rollins: Yeah … I hated tests.

I hate tests.

I hate tests as an engineer.

people that are like, "Test-driven
development," I would always be

like, "You are out of your…
That is the stupidest thing."

I could not think of a better way to grind
an engineering team to a halt, right?

we gotta ship this fast.

Bret Fisher: I'm the same way
with spec-driven development

that I am with test-driven.

I'm a shiny person and none of those
things give me the dopamine hit of shiny.

Which is why no one pays me
professionally to develop full-time.

Michael Rollins: Yeah, but you know what?

I get test-driven
development for free now.

Bret Fisher: Yeah.

Michael Rollins: I don't have to worry

Bret Fisher: the pain

Michael Rollins: Yeah.

let me throw this at you.

I was working on 15 different bugs
at the same time the other day, okay?

was doing so much that I locked
my machine up by filling up

the disk, because apparently if
you run Node, the, what is it?

The npm install and the type, you know,
the Python equivalent, that can be a

lot of space, and I only have a half a
terabyte hard drive on my work machine,

'cause I decided that's a fine thing.

Bret Fisher: that code
is small and that, you d-

Michael Rollins: And I was
building Docker images, yeah.

and, next thing I know,
my machine just stops.

not slows down, it just stops, right?

But in the process, I had
also opened up 15 PRs.

I locked up the entire, stack.

I accidentally flooded the GitHub
Actions, overloaded the Lambda that

runs our, database cloning, where
we clone production into the PR

environment, and found a bug in the
process, which was very nice to find.

but it was not the first time that
it happened, where I had basically

launched this clone storm that locked
our entire PR infrastructure up.

Everything was still running,
but you couldn't do a new thing.

I've never tried to land 15
PRs at the same time before.

Have you ever done that?

Bret Fisher: Not even

Michael Rollins: Has that… Have
you ever heard of anybody doing that?

Bret Fisher: Yeah, I mean, there's
the whole new, what's the PR gating

thing where you combine them?

Michael Rollins: as, PO it's like
merge, merge streams or something.

Bret Fisher: yeah, merge stacks or

Michael Rollins: yeah, it's basically
what GitHub just totally blew up.

Oh man, GitHub sloppy stacked this hard.

so, but listen, yes, of course it's, it
may, yeah, maybe it is degrading, right?

but nobody's ever done this before, man.

Look, we were doing it before
Anthropic and before OpenAI.

We were doing this style of
engineering back in July of last year.

Yeah.

Anthropic and OpenAI did not adopt it.

nobody's done this, man.

And so, yeah, are we finding
things are degrading a little bit?

Sure.

you know how long it takes
me at this point to put a fix

into production if I notice it?

20 minutes, 30 minutes, assuming
I have to build the images.

Bret Fisher: I recorded a very similar
episode to this topic yesterday with,

someone from GitHub Next and then
someone from Microsoft Research, and

they're both working on the agentic
workflows feature set for GitHub,

which is currently, I believe, in
beta, and I'm very excited about it.

I thought it was very weird at first,
but after spending an hour and a half

with them, I'm all on board and I'm very
excited about spending more time with it.

He talked about that he's also working
across 22 projects with multiple PRs

in each project, and that he does
everything that-- because they're

GitHub, they do everything through
Copilot, everything in GitHub.

And so he spends most of his day
as a researcher that's, or as a

developer researcher, I should say,
prompting through github.com/, what-

whatever the little agentic, page
is where you can prompt like you're

in Co- Code GUI or Claude Code GUI.

and he was saying, you know, he does it,
He spends a lot of time on his mobile,

doing it, and, sort of talked me through
his day to day and I was like, "Man, I

feel so antiquated," 'cause I'm still got
this habit of if I think of something, I

go and write it down on my phone, but I
think I'll do that when I get back to my

computer because I'm just so not driven
to use Dispatch or I'm just not there yet.

Yeah.

I'm-- I know I have it and I just don't
default to it yet as, "Oh, I can kick, I

can kick this off right now. I don't need
to wait till I'm back at my computer."

Michael Rollins: So this
is, this is what I do.

I have two machines that I write code on.

one is this laptop here, and
one is my machine back at work.

I typically use my machine
back at work for this.

I have remote sessions or remote
environments or something, turn on

automatically in Claude Code, and then I
also have Auto Mode turned on by default.

Bret Fisher: Right.

Safety off.

Michael Rollins: no.

I leave safety on.

Auto mode is not YOLO

Bret Fisher: oh, right.

Sorry.

Auto safety mode.

Right.

I'm sorry.

I'm th- I was thinking of

Michael Rollins: it's legitimately
called, there's planning, except that

it's whatever the default, normal default
state is, and now they added Auto Mode.

Auto mode actually has a safety
classifier in it, and so there's

effectively another agent that's over here
running that says, "Is this dangerous?

Is this dangerous?

Is this dan-?" No longer
can you do that, right?

Bret Fisher: Right.

Michael Rollins: it'll just smack it down.

Bret Fisher: Sorry, you don't,
real quick, you don't enable,

sandboxing in the Claude Code?

Michael Rollins: actually Auto mode may be
some kind of sandboxing in the background.

Bret Fisher: You could call it that,
but yeah, 'cause it's gating the tool.

it's gating tool access basically, right?

Like it's a programmatic deterministic
gate, at least as maybe not auto,

maybe part of it's a little bit of
an, a mini LLM or something in there.

Michael Rollins: I'm fairly…

Bret Fisher: deterministic way to do it.

Michael Rollins: Yeah, but then there's
also, then there's your unsafe mode.

Bret Fisher: un- hands-on.

Michael Rollins: Okay.

So the danger with unsafe mode is
actually, auto mode and unsafe mode

are very similar, but there are
certain things that auto mode will

not allow the agent to do, and one of
them is just go find a module to use.

and that is super-duper
important right now.

the number of tool chain attacks that
are currently in flight are insane.

And the security landscape is, it's
actually one of the things that,

keeps me up thinking about it.

Did we ship something that has a
security vulnerability that was

intentionally put there so that our
stack would be exploitable later?

Bret Fisher: it sleeping on my machine?

Michael Rollins: And is
it sleeping on my machine?

Bret Fisher: waiting to take my

Michael Rollins: Yeah.

but I have these remote sessions running,
and I, I'll be sitting… I used to not

sit on the couch and watch TV with my
wife because I felt like I needed to go be

doing something, and I'd have these ideas.

I don't love watching TV, but I like
being with my wife, and it was like,

okay, well let me go sit with her.

But then I'd have these ideas, and
I'd be like, "I need to go do this."

Well, now I just pull my phone
out and I tap a few things on into

Claude, and I just put my phone away.

And it goes and it tests everything
and does the idea, and then it

comes back and it's "Okay, yeah,
it's ready to ship. Are you ready?"

I mean, it's a wild world.

Bret Fisher: you can't start a new
project from the phone, can you?

On Claude Code specifically?

Michael Rollins: Yes, you can.

Bret Fisher: Okay.

Michael Rollins: when you say a
new project, what do you mean?

Bret Fisher: for example, if I don't
have remote enabled on a, on an

existing repo that I've cloned, or I
wanna start a new code project and I

don't have a repo yet or a folder, I
can't get it to initiate a local new

session on my Mac remotely, right?

I kinda, like you said, you had
remote on always, or you have it, I

didn't even know you could do this.

It's actually a pro tip is, just turn
that thing on for everyone, all of

them, 'cause I always forget to, to
slash enable remote in my sessions.

Michael Rollins: All right, so first
off, you have to give Claude Code

the access to your GitHub repo.

If it doesn't have a GitHub
repo, you can't use it.

Now, if you wanna start a new project,
you go create the new repo in the GitHub

app, and then you come back to Claude
Code, and you give it permission on your

phone to just use that, and then it can.

And you can start the, you can start
it in the, there's like a cloud

version that'll run and make all the

Bret Fisher: And you're using, the
connections feature that actually

Michael Rollins: Yeah, I don't
even remember what it's called.

Bret Fisher: Yeah, it's an
automatic connection to GitHub

from the Claude Code web UI.

Michael Rollins: Yes, that's correct.

And now because of the way our stack
works, because I can issue a PR and stand

up the copy of production, I actually
don't have to have something running

on my local, but I can continue it.

And so if there's something going on
and I'm like, "Huh, I would-- I need

to go walk away for two hours, but I
wanna make sure this thing progresses,"

I'll just get a notification.

I can look at it when I'm able and
say, "Okay, go do this, that, and the

other." So that is, that is a pro tip.

Yeah.

you can, if you want, be running through
First Landing Park, with 25 pounds of

computer equipment on you, and pull your
phone out and advance your project again.

You know?

Bret Fisher: I don't see a lot of,
those weight v- exercise weight vests

with laptop sleeves in them, but, you
know, maybe that's the nerd equivalent.

So if there was… we've talked a lot
of high-level stuff, and I really,

I think this is gonna be really good
for people to s- sort of hear a team

that's… I w- I wanna make a point here
that I feel like one of your secrets is

that you're in tune with the business,
which we, you kind of started with that.

Your team is aware of the business
goals, sales objectives, like in, in

terms of what the product needs to be,
where we're headed, why does this thing

exist, what do our customers look like?

Because that allows them more agency and
allows them to… Also, there's a metric

of judging risk there, because like you
said, if you were a healthcare, if you

were a fin- fin healthcare company that
was, that, you may not have the same

posture because you're appropriately
dialing up the risk tolerance to the level

of the organization that you're in, and
that is different for every organization.

So I j- I feel it's like a, feels
like it, this bears worth repeating

that if you're not… Do not try
to shove this round peg in a square

hole of, you're in a high safety,
you have, you're on a DARPA contract.

you probably, yeah, you probably
should be doing old school co-pilot

code completion with the AI at best,
where every line is sweated over

and you're just saving keystrokes.

Michael Rollins: I've always been
the engineer that was connected

to the entire organization.

It's just my nature.

I like talking to people, and I'm
curious about how the business works.

However, that is a skill that
engineers are gonna have to have in

the future, at least to some degree.

When we started this, the very
first thing we realized was that our

infrastructure was the first gate.

So we went ahead and we fixed our
infrastructure, and we thought,

"Great, now we can move forward."
But then we broke product.

Product could not keep up with us.

There were only three
engineers at this point.

Product could not keep up.

Then we broke design.

Design is consistently behind in our
system, and it's not design's fault.

It's that there's now four other
engineers plus our designer who

actually commits code, right?

Because that's the only way for
him to keep s- keep up with us.

Then we broke sales, right?

I would have to get on sales calls and
consistently during the sales call would

have one of our sales guys be like,
"Yeah, we're gonna have that in, you

know, a couple months." And I'll say,
"No, it's in the product right now."

Right?

I have closed deals because it
was already in the product and

I was the only one that knew,

Bret Fisher: Yeah.

Michael Rollins: right?

Um, we

Bret Fisher: fresh

Michael Rollins: Yeah.

We, we don't have much marketing, so I
don't know if we've broken marketing.

I suspect we have.

But when you're in a position where
the engineering organization is now the

fastest organization in the company,
the only way for you to move safely

as a business is for your engineers
to know how the business works, right?

So before we ship things, the engineers
that work with me know, " Hey,

do we have any demos coming up?"

' Cause that's a real question.

We're small.

We have to ask that question, right?

Are there any important
things happening today?

Because that's something
else we need to know.

And so these are the questions
that engineers typically don't

ask, and they're gonna have to.

Engineering is different now because
the speed at which engineering can

progress is effectively putting
pressure and force on the rest of the

business and breaking it in the process.

And I've never seen anything like
this, and that's how I know it works.

Bret Fisher: Yeah.

Y- you're describing this as what I
would maybe label as a pro-engineering

culture, where a lot of the messaging
we see in the media is around cost

optimization of reducing engineering.

And what I'm hearing, the the language
out of you, which is clearly part of

the business culture of your company,
isn't looking to save the money by…

Because you're so efficient,
let's reduce you down to two

engineers instead of five, right?

Let's get rid of the designer and
burden someone else with that workflow.

I don't hear that out of you,
and that sound- I'm assuming

that's pretty deliberate.

It's a deliberate strategy to not
starve the engineering organization.

Michael Rollins: Well, I mean, to
be clear, The idea that what we've

built is what we've built with just
four or five engineers is absurd.

There's no way I could've presided
over this thing being built with

the number of engineers that we
have with the, without these tools.

Is AI a cost savings?

Only insofar as it is an
accelerator to your individual team.

AI is not going to be cheap.

Boy, have we discovered that recently.

Yeah.

I'm like, look, man, the minute Claude
Max and the $200 equivalent for OpenAI

goes away, small companies like mine
are gonna be in a world of hurt.

It's too expensive.

So, but the option is, but we can't
afford to do it any differently either, so

Bret Fisher: Yeah.

That leads to a question around cost.

Do you, um… I have this diagram
which I wanna show you real quick.

I'm trying to explain in my courses
to people that are, you know, not

near as, as advanced as you all
are yet and, or as experienced.

And I'm taking the project management
triangle, which was traditionally

it was the rule was there's
three thing- levers you can pull.

You get to control two, you
meaning management, and you don't

get to control all three, and
that is quality, cost, and time.

And so the whole adage was you might
be able to, you know, you might be

able to reduce the cost and, and
resources, but that's gonna cause

the time- timeline to stretch.

So you move two parts of the triangle,
the third one's gonna be affected.

And I'm starting to correlate this
to what I'm maybe calling like the AI

engineer project management triangle,
which is a sort of an internal metric

that we… Assuming there's not unlimited
budget, which some teams are very lucky

and they can just token max to the,
all day long, that we really kinda get

three new choices we didn't have before.

That is model choice, harness
choice, and where or when to use AI.

Because obviously you could have AI
robot, you could put an agent into

production and have it Ralph looping
every second of the day, constantly

looking at stats in production to
look for any potential failures, and

it will burn thousands of dollars of
tokens a day if you let it, right?

So not everyone gets to choose max AI in
every part of their software life cycle.

So if we get to choose model, which
effectively not only is a time thing

because Haiku is way faster than Opus,
but it's also very much a cost thing.

And then which harness choice, which
is maybe more around functionality

and, ease of use because Claude, I
feel like, is so, so far ahead of

everyone else that it leads to a lot
of new functionality that we don't

yet have on some of the other ones.

And then when and where to use AI,
which some organizations maybe don't

even let you put it into production
yet for helping with troubleshooting.

And then there's these esoteric
things of the environmental impact,

the privacy impact, and then the
political impact of certain things.

I have members of my community that are
in Turkey, for example, where they're,

relegated to not being able to access
all the models, or they have privacy

rules where they're not l- allowed
to move this data outside their data

center, so they have to then be able
to somehow house local models that they

have to run open weighted or whatever.

I was curious if there's anything in
here of value that you would think

that, my idea has merit in the fact
that this is a new paradigm that we've

never in the past, I don't recall the
last time I had to worry about cost

in my daily activities that might blow
my budget as an engineering manager.

This is a new thing.

Like maybe cloud, but that
wasn't happening in a day, right?

Michael Rollins: So first off, the most
important part of this is the harness.

There was a Stanford paper, and
the Stanford paper came out and

basically said, " You can improve the
performance of the model by 6X if the

harness is correct." And so harness
is really what you need to focus on.

And while Claude Code is one of the
best harnesses in the world, if not the

best, there's still an extraordinary
amount of room for optimization.

So our models, our harness, so we
actually, yes, I code in Claude

Code, but in our production
environment, we deploy agents.

This is what we sell, is
this agent infrastructure.

And our agents write code.

Now, our agents do not get tool calls.

I don't let the LLM touch
many tool calls at all.

What I do instead is I package, we package
the tool calls into an agent SDK, and

the only way for the model to touch the
tool is through writing Python code.

Now, what does that end up doing for us?

Well, number one, most models are really,
really good at writing Python code.

You can actually deploy small
models to write Python code, right?

But we realized we have this im- immense
cost saving right now that we have

not leveraged, and we're going to.

Because if our agents are writing
code to express themselves constantly,

well, I know how to save code.

As long as I can find that code
again, I can actually go reuse it.

I don't have to spend the
tokens recreating it, right?

And all of this, I've built this Instagram
agent, that I've been using for m- my

Instagram account, and I actually built
my own Instagram integration in Claude

Code, and I moved it into my agent,
and I said, "Whenever you do anything

with Instagram, go use this code."

And so what I found is that it has
been using it and improving it daily.

Bret Fisher: Mm-hmm.

Michael Rollins: And I realized, oh,
I think this agent is now using less

tokens, ' cause it just goes and
runs the code it's already built.

And so that's one of those things
where it's oh, well, if the harness

can be used to make the model better,
the harness can also be used to

reduce the number of tokens that are
necessary to run on a daily basis.

And so putting the harness at the
center of that thing is actually

the way to look at it, right?

The central connector between the
cost of the LLM, the time that the

LLM takes to run, and whether or
not it's safe to run is the harness.

And in point of fact, you can even
structure your harness such that

you can observe the inputs and
outputs to the harness without the…

ever leaking secrets to the LLM.

Bret Fisher: Mm-hmm.

Michael Rollins: Because one of
the other things we do is we proxy

everything in and out of the harness
that's running on our remote system.

And since we're proxying it, I
can actually inspect the traffic,

and I can do that with code.

I don't have to do that with an LLM.

And- Yeah … it's fast.

I know how to do it.

I know the things I'm looking for.

And so now if I start seeing the
harness producing stuff that it should

not produce, then I can smack it down.

And conversely, I don't have to
give the harness all the secrets.

I can inspect the traffic on the way
out and insert the secrets in flight

without ever exposing them to an LLM.

So the real the center of
that triangle is the harness.

It is the most important
thing in what we're doing.

Now, are the models, amazing?

Yes.

But now the models are good enough.

Go focus on the harness.

Bret Fisher: Yeah.

Is that, are you using
LiteLLM as your proxy?

Michael Rollins: No.

Bret Fisher: Do you know?

Michael Rollins: No, we rolled our own.

Bret Fisher: Oh, okay.

Michael Rollins: So in terms of bringing
in the LLM, we just bring the LLM in

through the, um I don't like LiteLLM.

There were at least some significant
issues with it previously.

We use the LangSmith LangGraph library,
and we just make sure that our LLMs

conform to that particular structure.

Bret Fisher: We've got teams that I'm
working with that are trying to figure

out when it comes to local human agent
harnesses, that, you know, they're

trying to understand usage per person
and trying to figure out, give model

access as well as maybe inject things
at some of the sandboxing tools are

also learning how to inject things,
in the, on the fly in the HTTPS.

so the last question I was gonna see
if I could wrap this up with the last

question, and I'm gonna show you a
graph again 'cause I have another one I

spend, I try to spend some of my time…
I'm trying to, and I'm really bad at

it, so I'm forcing myself to do it.

so what I have on screen is something
that I'm working on and calling the AI

Engineer Maturity Model, but it comes
from a different aspect than every single

other one that I see on the internet,
and I spend a lot of time looking at what

everyone else is talking about, of like,
how do I onboard my team to the skill,

the human skills they need to really
complete the life, the software life

cycle of where all the AIs belong, right?

It's not just about
Claude Code on my desktop.

There's lots of other opportunities, and
of course, the industry is now starting

to figure out that if you speed up the
number of PRs you're shipping, but you're

not improving the things after that,
then you're just gonna have a giant

bottleneck in your PR stack, and your
production won't actually get faster.

But this screen that I'm looking at is
a fictional screen that on it is the

idea of over time, as you implement
different human workflows or different

human skill sets that the team ramps
up on, we assume that your overall

team productivity improves, and there
will be times, like you said, when you

first are doing certain activities that
team productivity actually declines.

Like the day that you implement Claude
Code for the whole team and they're all

using it, that's a rough week, right?

That's a really unique thing you need to
give people time to onboard, so there's

probably gonna be a lower productivity.

And then there's other points at where
I see teams that maybe they've, they

fully adopt agent harnesses locally, but
they really haven't done much in CI yet.

They're barely even
doing PR review agents.

They maybe not even using that yet.

And they realize now that they have
this CI backlog, so I would argue

that becomes a team productivity
dip because now they're spending all

their time staring at code that AI
built, and they haven't really kind

of leapfrogged to the next level.

So I was looking at this like imaginary
list of things you can do in a team

to improve the overall productivity
of your software deployments.

Do you see like activities or
certain things that sort of rise up

as the obvious things besides just
using Claude Code on your desktop?

Is there activities that you think are a
big tell on whether the team is mature or

not in their AI, s- software life cycle?

Michael Rollins: Um, I
hadn't thought much about it.

Uh, bec- partially because I
haven't encountered teams that are

doing it the way we're doing it.

Um, I b- I promise you, I've
had a lot of people look at me

directly and tell me I'm insane.

They're like, "This is insane.

This isn't real software engineering."
And then they come back, they're like,

"Oh." Here's what I can tell you.

The end of that graph looked like
this gentle line that just went up.

Yeah.

And there's a point in there where
that gentle line just becomes vertical

And that's the point where you do have
agents fixing things without human input.

The big milestones for us were
we started from, you know,

everybody's using AI to code.

We moved into, " Hey, "When's
the last time any of y'all

actually looked at the code?"

And all my team was like, like,
everybody didn't wanna get caught.

I was like, "Oh, 'cause I haven't
looked at it in months." Yeah,

and they were like, " Oh, God.

Okay.

Phew.

Thank you.

Thank you."

Bret Fisher: They, They didn't know
where you were going with it, right?

Like, you were, you

yeah.

yeah.

Oh, by the way, imposter syndrome
that jumps out when the junior-est

member of the team posts something in
the engineering Slack that calls out

something the most advanced engineer's
done, and you're like, " Oh, God, I'm

not even a real engineer." Like, how…

And then you're like, "Oh, wait, you
had Claude look at that, didn't you?"

And they're like, "Actually, no.

It was Codex." You know?

But it is everybody using AI full time?

The next step is when was the
last time anybody read code?

Not just write it, read it.

Do you have full CI/CD yet?

And then beyond that, it's as simple
as, you know, I'm sure that you

could throw some milestones in there,
you know, what's your… How fast

is it to sh- to fix a bug, right?

Because in lieu of a perfect
release, how fast can you fix

the release if it goes sideways?

By the way, you did ask one question:
Do you have a single button rollback?

We do, but it's only good for 10 minutes.

Okay?

So we gotta be real careful.

but then finally is can the
agents just do your job for

you, and under what constraint?

And that is when I think you're at
full AI maturity, that when it is safe

enough to let agents move in your system
without getting human interaction, you

are now scaling yourself infinitely,
because every step change that you

add to those autonomous agents is an
infinite scaler for the rest of the team.

One thing I will point out, though,
and we can close on this, is when

you make mistakes in this style
of engineering, they're big.

I'm not talking about shipping bugs, okay?

I'm talking about a misstep
in the architecture.

I'm talking about a misstep in, holy
shit, how did we manage to ship that

migration order on the database?

When you make a mistake like that in
this style of engineering, it's not

a self-wound, it is a decapitation.

And so every engineering team
will hit the point where they're

like, " Oh my God, what have we
done?" And we've done it twice.

And both times we ended up having to
regress the system by two weeks, which

doesn't sound like a lot until you think
about the fact that we have commits

that have been 60 commits behind.

We've had PRs 60 commits behind
because they were five days old.

Michael Rollins: So the cost
of the catastrophic failures is

higher, and that's the one thing
you do have to think through, so

Bret Fisher: Do you think about that
in terms of are there different tiers

of risk in the way you approach AI in
different repos or different parts of a

repo where you're like, "Well, this one's
dealing with the database migration, so

we have extra safety involved with that.

Michael Rollins: Yeah, so we
do have-- We use an owner file.

we need to be using an owner file.

I just keep forgetting to put it in.

But there are very specific
files that if we- Say again?

Bret Fisher: Get confession hour.

This is confession

Michael Rollins: Yeah.

Oh, believe me, we're not perfect.

Everything I've told you has been
relatively true, and maybe I've

waved my hands once or twice, right?

but the-- Yeah, we need
to put the owner file in.

If our system prompt changes,
I need to know about it.

If our list of tools changes,
I need to know about it.

If our, look, we ship multiple database
migrations in almost every single PR.

I mean, I cannot explain how many
times we have metamorphized the

database, in just a few days.

but there are files that if they
change, I want to know about it.

And by the way, one of those is going
to be, like, what are your imports?

What node modules are you bringing in?

What Python libraries are you bringing in?

'Cause those are important as well.

Bret Fisher: Yeah.

I can see how that would trigger
different, additional safety checks

in GitHub Actions where… I mean,
I do at times make directory-scoped

actions that are… It's also
really good just for any action.

This is a pro tip with GitHub Actions is
a lot of times your actions that run all

the time don't need to run on s- they only
need to run on certain types of changes.

And GitHub Actions, turns out they c-
actually have a lot of filters involved

so that you can say, "Hey, don't run the
action linter unless the action file's

changed," and that's only in this one
directory, so there's no even reason

to even fire it off unless it hits

Michael Rollins: I probably could,
if I probably paid you for an

afternoon, we I could get so many more
efficiencies out of our system, dude.

It's insane.

Look, I'm not DevOps,
I'm not infrastructure,

Bret Fisher: Yeah, at this
point I'm just a skill.

I'm just a skill.

I just have a GitHub
Actions workflow skill.

My new rule is every new conversation
I start with an AI needs a skill.

If it's a one-off question, fine.

But if I'm planning on this gonna take
an hour or two, we're gonna have a

conversation about it, I'm looking for
a skill to give that agent to enhance

its ability, because I personally have
noticed on my projects in the last two

months that the skill in every case has
always improved the outcomes, or at least

got me faster to the outcome, whether
it's making sure the skill is teaching

it the latest version of Go, making
sure that it's adhering to the latest Go

patterns, or whether it's the front-end
design skill that everybody's using, or

whether it's a GitHub Action skill that
enforces certain linters and certain

security patterns that I behave with.

I just feel like there's always a
good, good reason to have at least

one skill in your, conversations.

Pro tip.

Michael Rollins: Yeah.

Bret Fisher: Hey, man.

So what's the one thing you're
gonna excited to do next?

What have you not done that's, on your
to-do list of, I wanna try this out"?

Michael Rollins: Turn agents loose.

Get them to fix things end to
end without me ever knowing.

Bret Fisher: Continual self-improvement?

Michael Rollins: To be able to have
an agent pick up a report from Slack,

figure out if it's real, create the
issue, fix the issue, commit the code,

build the PR, test everything, merge it
to main, and ship it, and I never know.

When that starts happening, our
engineering team of five will look

like an engineering team of 80.

Bret Fisher: Check out agentic workflows
in GitHub if you haven't seen that yet.

it's a-- That's the actual term,
agentic workflows, because it writes…

it's like a full safety version of
putting an AI prompt in a workflow

without all the risky behaviors
that everybody's using right now.

There's like sandboxing, there's
all these different extractions.

And one of the things that it
can do is you can put it on

like a, kinda like a Ralph loop.

According to what I heard yesterday, was
they designed their system so that it's

not all Monday at 9:00 AM, and like they,
they, they distribute these things out.

It has inherent, rules inside it, so
if you ask it to be an issue reviewer,

it won't do all the issues right now.

Like it's not gonna l- overload you with
all of this work, which is what a lot

of people I think do, and initially the

Michael Rollins: I will

Bret Fisher: them a bunch of work.

Michael Rollins: I will say this, if
Copilot is my only choice, I won't use it.

Bret Fisher: Yeah.

Well, none of this has to do with Copilot.

You

Michael Rollins: Okay.

good.

'Cause this,

Bret Fisher: All of this with Claude Code.

Yeah, you pick your own agent.

Michael Rollins: Okay.

Bret Fisher: framework is agent agnostic,

Michael Rollins: Excellent.

Then yes, I may try it.

So

Bret Fisher: yeah, I think the,
the push button features in

GitHub are all Copilot, right?

You turn on the PR
review in your settings.

that's, th- these are just,
features that happen to use Copilot.

This is more of a hobby kit approach,
where you're at the command line

with this stuff and using YAML files.

I'll show it to you someday if
you don't ever get a chance.

Mike, thanks so much for being here.

Can you give us your full
name, and title, and company?

Michael Rollins: Mike Rollins,
CTO of Relify and Family Digital.

You can also find me at rollins.io online.

Instagram is rollins.io, and uh, I
think that's the major points there

Bret Fisher: You talk about AI on
Instagram, so if people wanna hear

more thoughts from you, I would
recommend Instagram, Awesome, man.

Well, thank you so much for being here,
and we'll, I'm gonna definitely have you

back on 'cause we're gonna talk about
you pushing the envelope even further, so

Michael Rollins: Let's do it.

Bret AI July 2025: Thanks for joining
us, and I'll see you in the next episode.