AI Hype vs. Reality. Real Stats from Laura Tacho, CTO of DX

Laura Tacho: Why is it that
a developer can only spend

one hour a day writing code?

It's that they are in meetings all
the time, having interruptions.

they are waiting on CI, they're trying
to navigate through code bases We see

this reflected in the data and when
we look at AI time savings right now,

which is median, like three hours,
45 minutes per developer per week,

and then we look at how much time
they could be saving by reducing dev

environment toil, by having better
documentation, by reducing meetings.

AI doesn't even cover a 10th of what they
could be saving if the other parts of the

developer experience were also increased.

So, speaking to your kind of.

Deep seated fear, maybe about
like the over promise of AI.

I think one of the ways that AI has been
overpromised is that it is a silver bullet

that is gonna solve so many problems
when in fact developer experience is

still a way bigger lever for almost every
company than AI assisted engineering

at least at this point in time.

Bret: This is one of my most important
and favorite episodes this year.

I'm so excited to let you listen to this.

If you are currently using AI
or even thinking of using AI in

software engineering, you'll wanna
hear from our guest on this show.

Laura Taco, who's a friend of
the show, been on many times.

Laura is the CTO of DX, that's the
DevEx and dev productivity company

where they help organizations
measure development team work.

And in this episode, we're focusing on
her team's work on the gains and losses

of using AI in your software lifecycle.

My co-host, Nirmal is back adding his
experience from what he's seeing on

the topic at AWS and their customers.

And if you haven't heard Laura before,
she's been on this show many times

talking about containers and DevOps,
but years back she started focusing on

engineering management and improving
productivity and measuring outcomes,

which happens to be very important when
you start to introducing a new workflow

to everyone's job like AI is doing.

Her pivotal work at DX has her focusing
exclusively on how professional software

teams are actually adopting ai, where
the snags and rough spots are, and how

your team can properly adopt AI today
for maximum productivity benefits.

Also, a quick reminder to smash
that like button and review

the podcast in your player.

If that's your chosen way to listen or
watch, because that's the way to help this

episode cut through the hype and noise out
there on the internet around AI adoption

so that other engineers might find Laura's
breath of fresh air on the subject.

It turns out it's all not doom
and gloom and we all are probably

still gonna have jobs and AI is
not as good as they claim it is.

That's a spoiler spoiler on there,
TLDR, but thanks so much for

watching and let's get to the show.

Hey, welcome to the
Agentic DevOps podcast.

I am Bret, I am here with my cohost
today, Nirmal Mehta, welcome, Nirmal.

Nirmal Mehta: Thanks for
having me back, Bret.

I'm Nirmal Mehta.

I'm a principal specialist, solution
architect and containers tech lead at

AWS and These are my opinions and not
of my employer, Amazon Web Services.

Bret: I'm glad to have you back.

And we're, we're
completing the trio today.

Laura Tacho is back with us.

She's been on so many podcast
episodes I can't keep track anymore.

welcome Laura.

I'm glad to have you here.

Laura Tacho: Hi, Bret.

Hi, Nirmal.

It's so nice to have
the trio back reunited

let's hope the audience can brace
themselves for what's about to come.

Nirmal Mehta: You're so kind.

Can't wait to get into it.

We have such interesting
topics to dig into.

Bret: Laura.

you work at DX.

Tell us about DX and what you do there.

Laura Tacho: So I am CTO at DX DXs,
a developer intelligence platform.

That means we are getting data insights
qualitative self-reported from developers

about developer experience about velocity.

Productivity metrics and helping
organizations use that data in

order to improve, in order to make
and validate their AI strategy, in

order to make developer experience
better and help engineers get more

work done with less friction and
less drag across their systems.

a very exciting role.

I get to look into so many
different organizations.

Netflix, ADP, Vanguard,
Pfizer, you name it.

I've had a chance to work with some
really exceptional leaders who believe

that developer experience is an
incredibly important driver, and that

is such a nice environment to work in.

Bret: So Nirmal and I have got to watch
your rise of fame recently around AI

because you have been speaking heavily
for months now, if not a year, around,

productivity specifically related to AI.

Can you tell me, you recently
launched something significant.

What was that?

Laura Tacho: The AI measurement framework.

Bret: AI measurement framework.

What does that mean,

Laura Tacho: What does that mean, Bret?

Bret: Does it mean the AI is good or bad?

Laura Tacho: yeah.

Well, we help you find out.

one of the things that I enjoy about my
job at DX is that we are not affiliated

with any AI vendor or any other kind
of vendor for a tool in the SDLC.

What we try to do is get you the
data so that you can interpret

it and make decisions on what you
should do for your developers.

because of that, we're in a
really unique position to.

guide the industry on measuring the impact
of AI because I have a front row seat to

400 plus companies and how they're using
AI, what's worked, what hasn't worked.

I have a bunch of longitudinal data.

I've been working on developer
productivity metrics, as my sole

focus for four or five years now.

And then before that, I've been in
developer tooling for like, 15 years.

before Docker?

Before, yeah, before we knew each other.

back in January of last year, we wanted
to answer this question of like, how

should I measure developer productivity?

And that's where the DX
Core 4 Framework came out.

So I worked on that with Abi
Noda, who's the co-founder of, DX.

We worked with Nicole Forsgren,
other researchers and put

together that guidance.

Well, it didn't take
long before AI really.

kind of blew everything up
late, you know, late 2024.

A lot of companies were adopting it
and they were looking and thinking,

do we need totally new metrics?

Like how do we measure the impact of AI?

How do I know if I'm having
a return on investment?

How does this all relate to things
like DORA Metrics, the DX Core 4?

we came up with the AI measurement
framework, which is really

targeted specifically to measure
how and where AI is being used.

when you use that together with the DX
Core 4, you get a comprehensive picture

into, speed, quality, maintainability,
developer experience, business

impact, all of those factors that AI
is supposed to be helping us with.

maybe you can find out if it is or if
it isn't, and decide what to do next.

Bret: Nirmal and I started this
podcast at London, which we were

all at London for KubeCon earlier
this year, I think in April.

And that's when Nirmal and I recorded
the first episode of this new podcast.

at the time, we were still trying to
figure out, I feel like so much has

happened since in the last five months.

At the time we were still trying
to read the tea leaves around how

AI is useful or not in the software
lifecycle, and where is it most useful?

Where should we putting
our energy and attention?

And that was really one of the major
premises for even starting this

thing was everyone's talking about,
frontier models and dev tooling that

adopts AI and everything's shiny.

Everything's flashy, and we
were really interested in, where

is it actually useful today?

Because you can easily just sit down
and be overwhelmed with everything.

But since then you've come out with
this framework, you've been talking

over the place, multiple conferences
around the truth, and we've started

to get some sensational headlines that
you talked recently on the Pragmatic

Engineer Podcast around everything
from 30% of Microsoft Code is AI to.

now we see things of despair, of
there's actually negative productivity

in certain teams, and it just
feels like it's all over the board.

can you distill all that down to what
you're seeing and what's fact from fiction

Laura Tacho: let's rewind.

There's a couple things that you
said that maybe we can kind of paint

the picture of what's reality here.

you mentioned one thing I
wanna dig into a little bit.

You're talking about frontier models and
doing everything on the bleeding edge?

all of us and our collective gray hair,
we were around for containerization.

before the show started, we were talking
about how some companies were okay

with pressing snooze on Kubernetes.

They were okay with pressing
snooze on containerization.

Maybe they're even okay with
pressing snooze on the cloud.

And that just doesn't seem
to be happening with AI.

there's really like a lot of
hunger to like adopt it, adopt

it now, get the maximum gain.

and you know, thinking about that
and what you said about the frontier

models, it very much felt like what
we were doing How long ago was that?

12, 13 years ago.

With Docker, that very much
was on, on the frontier.

And what we found was actually
the hardest problems are kind

of the most boring problems.

It was a lot of lift and shift.

It was a lot of replatforming.

It was a lot of just
like enterprise legacy.

that is the majority of what
everyday developers are doing.

They're not doing frontier work with
models they are trying to navigate

and Cut through the spaghetti code
in their 10, 20 plus year code base.

just showing up to work every day,
doing the best they can and enjoying

the craft and, trying to help their
company hit their business objectives.

when we talk about a lot of this, like
what's in the news, there is a lot about

the frontier and the bleeding edge.

What's really tricky though is
there's a big gap between the frontier

and the bleeding edge and like the
experience of an quote unquote everyday

developer and what they do every
day in the same way that we saw that

in Kubernetes Docker time, there's
quite a gap between those things.

So that's one thing I didn't
totally answer your question.

Nirmal Mehta: so,

Laura Tacho: a good place to start though.

Bret: it.

Nirmal Mehta: so there's a
difference that you're seeing, right?

we have that traditional gardener
hype cycle, Like early adopters.

and then the late
majority and the laggards.

I think you were mentioning that
you've not seen some, situation where.

The laggards and the late majority
are actually pursuing this technology

aggressively as if they're early adopters
without understanding deeply the value

proposition why is that happening?

Laura Tacho: I think it usually
is, it comes down to money.

there is a lot of sensationalization
sensational headlines about.

What AI can do and its
impact on the marketplace.

and its impact on sort of like bottom
line P&L on the, the economic situation.

And I think that's been a huge
driving force for a lot of these

companies that previously for any
other technology would've been very

fine to be in the late adopter, or
late majority laggard kind of place.

But the promise, the financial
gain for AI is just so promising.

because it not only helps them operate
in a leaner way, but when their

competitors also have access to that
technology, they will outpace them.

I think that's the real threat
when I talk to CTOs and senior.

exec teams from companies that previously,
would've been in that late majority,

is that it's, it's not really about,
it's not about cutting headcount.

It's not about necessarily cutting costs.

It's that their competitors have access
to this technology and they're so worried

about being outpaced they want to catch
up as fast as they can to not lose

footing in the marketplace and their
competitiveness, because that's the risk.

and that is a huge disadvantage
if that would come true.

Nirmal Mehta: So that implies
that the CTOs and the leaders that

you're talking to in companies that
do face, real competition, right?

if they're representing industries
where there's not necessarily that

much competitive, pressure, then that,

Laura Tacho: Mm-hmm.

Nirmal Mehta: that might not be
necessarily true and they might be

able to hold out a little bit longer
and truly see if there's an advantage.

Laura Tacho: Yeah, I think I see
competitive pressure on two ways.

One is like in the marketplace, right?

Like capitalism, but the other
places in the marketplace for talent.

And that is also a very big concern
for these companies that you know, they

feel like they're behind on AI, because
they're worried that because they're

not adopting these tools, they're no
longer an interesting place or rewarding

place for their developers to work.

And that they're gonna have a brain drain
people leaving their company and going

to work for places that are adopting
these tools, embracing them more.

I think the market as it is right
now isn't very conducive to that

happening, but it's definitely something
running, in the background of a lot

of executives minds about, making it.

attractive for talent to stay
reducing attrition risk, that kind of

competition in the talent marketplace
along with, competition, in the

marketplace for their software.

Whatever businesses, they're selling.

Bret: one of the thoughts I've had
early that we've discussed earlier

was if there's any marketing win for.

Big tech, specifically the AI
companies, which is all of big tech now.

it's, they convinced us all from
average Dev to CIO or CEO, that AI

is going to speed up everything and
that they need to adopt it everywhere.

we talked earlier where's the precedent?

where's the previous precedent of
what we're dealing with right now?

I struggled to come up with one because
when the cloud started, it was one company

then Docker came out, it was one company
and there were other ones trying to,

play around with containers and stuff.

But there was this singular
focus on a single company.

And now we have an entire industry
whether it's CIO Magazine or the

Pragmatic Engineer, you know,
newsletter, like they're all telling

us that these tools are being
adopted by everyone else, not us.

So we have the feeling of being left
out, the feeling of being left behind,

and the main reason that everyone
believes this is better is because

it improves the speed of business.

we've all been hypnotized by it and
obviously we have people that are and,

waiting and all that stuff, but it feels
like they are so, a minority as opposed

to other trends that it does feel like.

It's this just tsunami of tooling,
of solutions, of promises that I just

don't think we're really prepared to an
industry to like separate the truth from

the, fiction and the hype from reality.

And we're just not really
well equipped right now.

I feel for that

Laura Tacho: Mm-hmm.

I think you're very
levelheaded in your approach.

you ask a lot of good questions
about wait, but what about X, Y, Z?

where do you put yourself on that?

skeptic versus advocate?

I mean, you can be both, right?

This isn't mutually exclusive.

Like I'm skeptical about a
lot of things 'cause I see.

The real data, but also like you
two truths, like we can be hopeful

about something that will be there
in the future, but also skeptical

about the current capabilities.

how do you reconcile that?

Especially 'cause you're steeped
in this every single day.

Nirmal Mehta: great question and
thanks for throwing it back at us.

I think, one thing that we've learned,
all three of us, in our journey adopting

like bleeding edge stuff is, new
technology paradigms is to be, to have

a healthy dose of skepticism, especially
around value proposition around the

capabilities of technology, right?

Like early containerization full of sharp
edges that have, have now been dulled,

Laura Tacho: Mm-hmm.

Nirmal Mehta: over the last 15 years.

But, spinning up a container was
full of, issues back a long time

Laura Tacho: Mm-hmm.

Nirmal Mehta: and, we all had a healthy
dose of skepticism, but we also saw a path

to value for folks that were adopting that
technology and immediate use cases, right?

Like, going from spinning up a VM
and waiting an hour, as your chef

scripts go through and install apache
server, to in two seconds having 20

containers with the same configuration.

It's a very direct line from seeing the
technology and the value proposition in

terms of cost savings or whatever, right?

If you wanna follow the money.

My healthy dose of skepticism with AI
right now is that it seems like we're

in a perfect storm of other factors that
are going on, and is a lot of pressure

there's so much.

Money involved, and so much pressure
from all sides a tremendous amount

of fog over this technology.

on one side, having adopted these
tools, I'm privileged enough

to have almost unlimited access
to a lot of these technologies.

And enablement from some
of the world's best folks.

my colleagues

Laura Tacho: Mm-hmm.

Nirmal Mehta: create this stuff every
day, that use this stuff every day.

it's hard to push back
against that feeling of magic.

When I first started doing some of
the code generation with, ChatGPT

last year, and then moving over to,
Cline earlier this year and doing some

agentic, work with Cline and setting
up multi-agent to do operational tasks,

to do code generation and prototyping.

it does provide value, Like even,
helping, come up with, questions

to ask guests on a podcast.

there's value there.

At the same time, we have the
ending of zero interest, in

the United States economy.

We have global economic turmoil.

The promise of these tools to,
improve productivity in not only

developer, code generation, but
all kinds of different avenues.

and a kind of, under, under the
breath, like, oh, that also means

we don't need as many people.

And it's, I'm, I am healthily skeptical
of what if the technology will actually

deliver on like these hyped up, premises,
like 60% productivity gains, At the

same time, I don't know if that matters,
I don't know if the leaders at these

companies will actually care about
whether there's value there versus using

it as an excuse to execute against other
defensive maneuvers to navigate all these

other things happening in the world.

I end up very confused and, I really
appreciate that in your latest

talk at lead dev, you have this big
slide that said, data beats hype.

I think it's not only that data beats
hype, which I really appreciate and

I would love for you to get into, but
We have to present that data as fast

as possible because the hype seems to
be accelerating decisions are gonna

be made before the data is even there.

And I'm really worried that damage is
gonna be done maybe a year from now

we're like this fever dream of adopting
these AI tools didn't come to fruition.

But you know, it doesn't really matter
because we're all not working anymore.

Right?

Laura Tacho: Mm-hmm.

Nirmal Mehta: Like, does that even

Laura Tacho: Yeah,

Nirmal Mehta: great, we have data
that shows that this is not like, this

isn't me, all the things that, all
the hype, but it's kind of too late.

I feel like that's where I am.

And if I feel that way, I feel like
our audience is probably feeling

that same way or will be soon.

Laura Tacho: I think there's
already decisions being made

based on hype for example, knowing
how much code that's running in

production was authored with AI.

Is a very important thing to understand
when it comes to your security posture.

Understanding like changes to SRE,
understanding how you might need to

change or harden your build and test.

it's an important thing for businesses
to know, like how much of their

code is actually AI generated code
is actually making to production.

Because there's different decisions
you'll make if it's 0% or if it's 90% Up

until three weeks ago, there was no way
to know, there was no technical solution.

you know, broadly speaking, there were
some pockets of technical solution

of being able to track acceptance.

So a suggestion, like a code suggestion
in an IDE through an accepted suggestion.

So that's in your IDE through human
authored edits, committed open.

PR merged PR in production.

There's just no way to do that.

Sort of like provenance for code.

But still, we see headlines six
months ago saying, oh, 30% of our

code is being written by software.

from my position on the inside,
I sit in AI data all day long.

This is literally, it's my, it's all I do.

It's all I do.

It's all I think about.

There is no way, there is no
way to know that reliably.

You can, of course you can do
really structured sampling.

You can do self-reported data
and self-reported data does

get us most of the way there.

but that's not what this was.

This is coming from, this is coming
from like, you know, snippets or an

executive estimating in some cases.

How much of the code from
their teams is being written?

Just very sort of unreliable.

very lossy.

Very lossy.

Yeah.

Vibe, vibe, metrics.

And so, I do really have a lot
of empathy for A CEO who does not

have a background as a developer
and cannot distinguish claims.

due to lack of subject matter expertise
about what is actually authentic

and what, what is technology really
capable of versus what it's not.

of course the CEO should be seeking
out subject matter expertise, and

it is the role of the CTO or VPs,
of engineering data, to bring that

knowledge into the organization.

But I a hundred percent believe there
are decisions being made about AI

strategy and investment on, like,
we can call them vapor metrics.

They're just sort of
like, or vibe metrics.

I like that.

Nirmal, it's like vibe metrics.

it's about gut feel and not about.

measurement, not out of Ill
will or malice, but just out of

lack of telemetry because this
has been accelerating so fast.

can you imagine, having to decide on your
budget for Docker spend or cloud spend

for the next fiscal year when Docker
couldn't even tell you like docker ps,

um, how many, how many processes were
running or what the size of an image

was and you were trying to forecast
your, image registry costs that's

kind of what this would feel like.

which doesn't make sense, right?

It's very dangerous.

but because of the hype,
it's where we find ourself.

right now, at least
in, in a lot of pocket.

It is getting better.

I have to say that in the last couple
months has gotten a lot of better, but.

Nirmal Mehta: Yeah, just
one last comment on that.

At the end of the day,
data will win, right?

data that backs up the
value proposition will win.

there might be some pain as folks
adopt these technologies preemptively

to address competitive pressure.

your approach at DX and doing the hard
work of collecting data, you know,

even if it's anecdotal and not the
best, quality data at, at this time,

it's still the only way that, both
leadership and folks that are being

asked to adopt these technologies and,
and find those productivity gains or

else we'll be able to really navigate,
getting something out of these tools.

'cause at the end of the day, I do.

unlike some of these other technologies
that have come out in the last 10 years.

I do see, you know, in, in my own
adoption, I do see a path to value.

I have received gains
from using these tools.

So there is something there
with respect to the percentage.

I think your approach with the AI
measurement framework and collecting

that data is gonna be key to figuring
out where the actual value is and also

changing how, you know, I think there's
another, uh, another aspect to this,

from a potential developer or platform
engineer perspective, I'm seeing a trend

of, their leadership telling them, they

Laura Tacho: Mm-hmm.

Nirmal Mehta: use these tools.

And a lot of folks who have tried these
tools like maybe once or twice at some

point in the past year, didn't work for
them, maybe don't really know how to use

it properly or at that time was not the
most effective way or effective model.

And because things have changed so much.

we've gotten so many more best practices.

They're, skeptical.

Like, why do I have to try using this?

Like, I'm being forced to use a tool?

and that's never a great position, but
data is the way for those folks to push

back against leadership and potentially
adopt those tools in a way that will

actually be productive for them.

Laura Tacho: absolutely right.

I think, I've seen lots of companies use
the measurements that come out of the AI

measurement framework, not just to show
the ROI to their leadership, but also to

bring that back to the developers to show,
oh, you know, you might have been like

thinking, sensing, feeling, perceiving
this about your own productivity,

but here's actually what's happened.

There has been a 30% lift in
pull requests or merge requests

for those using AI daily.

you're also reporting that code
maintainability is not slipping and that

you're still having the same amount of
effort to review in AI assisted diff

versus one that wasn't AI assisted.

And that can actually be really powerful
for companies to have a validation loop

and get people interested in maybe giving
it another try because it's not hype.

It's data.

It's not someone saying, oh, this is
gonna give you a 10x productivity boost.

It's like, well you probably are
gonna save a couple hours and

it's gonna make it a lot more fun
and reduce your cognitive load.

So why don't you give it a try?

Or we have this new use case
that helps you, modernize

this old bit to, the new bit.

with 70% of the way there, you can
just do it with AI and then you only

have to take care of that last 30%.

doesn't that sound much better
than doing it all by hand?

Yeah, maybe I'm gonna give it a try.

data is important both for
leadership, but also it's very

important for developers themselves.

Bret: in order to get the advantages
you actually need to do more

than just add AI to your IDE.

it's almost like a spray and pray has
been the attitude of, the AI hype cycle

as long as you've got AI code generation
happening, as long as you just throw AI

into your CI, like just, it's somehow
going to magically make everything faster.

we're now seeing evidence
that, it's more than that.

which is true of anything in tech.

I don't know why we stopped
believing that for a moment

'cause it's always been that way.

what is some of the stuff
that's happened lately?

Laura Tacho: Yeah, let me show
you actually, I can just talk

you through some of the metrics.

We can look at some of the new
stories and then I'll show you

what's really going on out there.

this is a good.

Juxtaposition of what we're seeing a
lot of open up LinkedIn, it'll take you

30 seconds to find something like this.

So we've got sensational headline.

I mean, this one's not
even that sensational.

It's like pretty grounded.

It's only saying 10% more
productive at Google.

and then talking through their
measurement methodology TLDR on that one.

They're measuring time
saved per developer.

10% more productive is on the low end.

We're seeing other reports saying, it's
writing 30% of our code, or it's saving

our developers 20% of time per week.

There's so much variability here because
there's so much variability in what

actually is being counted when we say
AI and developers in the same sentence.

For some, we're looking at only code gen.

For some we're looking at AI assisted
engineering, which can be anything from

refactoring, debugging, production,
debugging, lots of other things.

Atlassian, their state of, DevEx report,
which I'm actually gonna talk about.

I think this episode will come out after.

but I'm doing a live event to talk
through their state of DevEx report.

Atlassian has put AI in Jira,
Confluence, all of their productivity

suite of, project management
tools, task management tools.

they see a lot of developers
saving a lot of time when you

add AI to Jira, for example.

But that's really different than
adding AI to an IDE, you know?

And so we have to ask the question,
and those of you listening, when

you look at a number, ask the
question, what's the boundary?

What's the start and stop
condition for this metric?

what are they actually measuring?

who's being measured?

Is it, a random sample?

Is it customers of a particular company?

ask yourself, does that
company sell tools with AI?

Are they trying to.

show me favorable results so that I
might be interested in buying their tool.

Like there's, there's a lot of questions
to ask, like data literacy wise.

So over here we've got these
headlines about, 10% more productive.

this graph here that we see is a study
for MITRE, and what they did was take

16 developers and they kind of like
followed them around for a period of time.

They are working on open source
projects and they have a pretty,

already pretty good, like moderate
to heavy familiarity with AI tools.

They're all professional
programmers and they tagged some.

Tasks as AI allowed and AI not allowed.

And so the ones that were AI not
allowed, they definitely couldn't use

AI on the ones that were AI allowed.

They could use it however they wanted to.

And they had them kind of do some
pre-estimation of how much time

do you think this is gonna take
you to do this task if you use

AI versus if you don't use AI?

And they had them do some kind of
predictive, estimations then they

observed them actually, screen recordings.

They also did some exit
interview style kind of things.

What they found was that the
tasks the developers used AI

were actually 16% slower than
the ones they didn't use AI for.

So even though their perception
of the change in time, if

AI allowed was positive, the
actual result was 16% slower.

And this is very valid data
gathered in a rigorous way.

It's hard to find 16 developers and
follow them around for 240, tasks.

this was a very big footprint.

Bret: Yeah.

Laura Tacho: The the thing though, this
is about open source developers, um, and

specifically they're talking about open
source developers who are working on tasks

that are a little bit more like on the
bleeding edge on the frontier, where code

generation was a huge part of the workload
that they were trying to use AI for.

That is not very similar to, again,
that every day engineer or working

in the enterprise who is not doing
bleeding edge frontier kind of work.

and so while the MITRE study is
very interesting and, and definitely

points out, we need more research.

When we try to generalize this and say,
well, using AI for AI assisted engineering

is gonna give your developers a 16%
slowdown across the board, irrespective

of what type of task, irrespective
of type of developer, tenure company.

That also oversimplifies to the
point of being incorrect as well.

and so we're definitely seeing
this kind of from both ends.

the skeptics using the MITRE study to say,
AI is a bunch of BS and you shouldn't use

it to the people on the other side saying,
but you know, Google's saving 10% of time.

It's just really hard to know and
there's not a lot of literature

on it because it's so new and we
don't have longitudinal data because

it's not been around for very long.

We've only been able to study
this for like two years.

It's not like we can look back to the
1970s and look at patterns and see

how things have changed over decades.

Nirmal Mehta: So you just brought up
a really interesting thing, which is

the digging into that MITRE study.

It was around a specific
use case, code-gen.

what in your, framework and when
you were talking to your customers

or to your survey respondents,

Laura Tacho: What

Nirmal Mehta: the

Laura Tacho: yeah.

Nirmal Mehta: outside of code-gen?

Laura Tacho: Yeah.

Nirmal Mehta: is maybe not the majority
of what people are doing every day, right?

Laura Tacho: Yeah.

And I think that's a
really important thing.

So I'll also be transparent
when we do research.

A lot of it, like if you see
me post something on LinkedIn.

That's most likely aggregated
customer data that I have access to.

When we publish reports and guides like
the one I'm about to show you, we do seek

out external, participants as well to not
only include people that are customers of

DX because that's, very self-selecting, in
a certain profile of company that really

cares about DX developer experience.

Bret: they're already tip of the spear.

I've never been lucky enough
to work at a company that had

anyone focus exclusively on DX.

Laura Tacho: Yeah.

Bret: I've always been fascinated in
your work because it makes every job that

I've had and every contracting client
I've had, feel like they're amateurs.

when I read the reports and the
information that you all put out in

Dora and other, sort of, tip of the
spear, what are the elites doing?

it always gives me like,
what are we even doing?

The rest of us are just winging it.

we don't have these methodologies.

We don't have these pipelines of
data that we're pulling out of

all of our utilities in order to
gather stats to change behavior,

which I feel like is a lot of this.

It's like, if it's not changing
behavior, then is it worth anything?

so it's exciting to see some
real numbers come out of this

Laura Tacho: Yeah.

Bret: showing us that, yeah,
this is like any new technology.

We have to figure out where
it works and where it doesn't.

It's not universal, it's not a panacea.

so yeah, let's jump in.

Laura Tacho: Yeah.

you say that so eloquently,
Bret, like if it's not changing

behavior, is it even working?

what's the point?

Absolutely, and that's why I get up
every day is to like the companies that

you work with that don't have this,
you know, trying to get them, giving

them tools and resources, enablement
so that they can have this because

you don't need crazy ETL pipelines.

to have information like this, you can
get so far with self-reported data from a

survey that you run in Google forms, it's
remarkable how well those methods work,

even at scale, the overhead does get,
cumbersome at some point, but you don't

need to wait for perfect ETL pipelines.

Anyway, we looked at 180 different
companies and talked to developers

who were saving the most time with AI.

I just asked them like,
what, what are you doing?

stack trace analysis and refactoring
existing code were the top two

use cases that we're saving the
most time mid loop generation.

So like, code gen didn't
come in until number three.

And then we have test case
generation that's also up there.

Like those top four are kind
of the biggest time savers.

Actually, I think this is from AWS Nirmal.

We talked about this at
KubeCon when we were in London.

The average developer at AWS only spends
how much time of their week coding?

it's double digits but it's
in the low double digits.

that's very consistent with what we
see across the industry that developers

are spending about 25, 20 ish to
30% of their time actually coding.

a lot of it is planning, doing
maintenance work, doing other stuff.

So, applying AI to other tasks, yeah.

Nirmal Mehta: yeah, I did find it.

AWS published in December 3rd, 2024, a
post that said that Amazon developers,

spend most of their time on tedious
undifferentiated tasks, such as learning

code bases, writing and reviewing
documentation, testing, managing

deployments, troubleshooting issues,
or finding and fixing vulnerabilities.

the stat from the article is no
longer in that post, but it was,

developers reporting, spending an
average of just one hour per day.

So this was at reinvent 2024 in the

Bret: Yeah.

Nirmal Mehta: by Matt Garmin, where he
said that they were spending on average

one hour a day on actual code, development

Laura Tacho: absolutely.

Yeah.

Bret: lines.

Nirmal Mehta: Yeah,

Laura Tacho: that was talking
about what are some of the use

cases that can bring the most gain?

I wanna show you this other thing,
which is really, this is like

aggregated, kind of dummy data.

This isn't data from any customer,
so I would never do that.

but the spirit of this is accurate.

and I've actually seen way more
pronounced versions of this,

but this is looking in DX.

So, we do quite a lot of,
visualizations to help you

analyze the data in easier ways.

But when you look at.

Why is it that a developer can only
spend one hour a day writing code?

It's that they are in meetings all
the time, having interruptions.

they are waiting on CI, they're trying to
navigate through code bases Nirmal, like

literally exactly what you just said.

We see this reflected in
the data and way down here.

it's not always this low for every
company, but when we look at AI time

savings right now, which is median, like
three hours, 45 minutes per developer

per week, and then we look at how much
time they could be saving by reducing

dev environment toil, by having better
documentation, by reducing meetings.

AI doesn't even cover a 10th of what they
could be saving if the other parts of the

developer experience were also increased.

I think speaking to your kind of.

Deep seated fear, maybe about
like the over promise of AI.

I think one of the ways that AI has
been overpromised is that it is a silver

bullet that is gonna solve many problems
when in fact developer experience is

still a way bigger lever for almost every
company than AI assisted engineering

at least at this point in time.

Nirmal Mehta: That's super interesting.

So another way to interpret that
data, from what I'm looking at is

that, the traditional DevOps, like the
DevOps maturity of the organization,

CI/CD automation becomes even more.

It was already critical

Laura Tacho: Mm-hmm.

Nirmal Mehta: competitive

Bret: Mm-hmm.

Nirmal Mehta: and value this AI world
get the value out of these AI adoption.

It becomes even more critical.

you have to have that automation in place.

and, that's what I'm seeing here, right?

CI, wait time, deployment, lead time,
those are, that's just pure CI/CD, right?

Laura Tacho: Yeah.

Nirmal Mehta: automation.

so that, that tooling
needs to be in place.

So you think that, folks that are
working on are, are, are pushing for AI,

tooling adoption with their developers?

net benefit is that, they'll, they'll
drag themselves into modern DevOps

because they need to do that first, to
create the foundation for getting the

value out of adopting these AI tools.

Is that a fair

Laura Tacho: Yeah.

Nirmal Mehta: See that

Laura Tacho: that is a
really fair way to see that.

I posted about this before.

Here's, a post from Abi.

I think what you're describing
Nirmal is that like DevEx is AgentEx.

I think there's, you know,
there's some subtleties.

I think what you're saying is, maybe
by using AI these companies can

drag themselves into the future,
which is great because the reality

is that a lot of what will benefit.

Agentic models, agentic workflows,
good documentation, organized

code, fast feedback loops,
clear project requirements.

Those are things that also benefit people.

And it's too bad that a lot of
companies are only willing to invest

in those things for a robot and not
for the people that were there before.

But

our opportunity now is that wallets are
open and the end result is the same.

And so if the robot and pleasing the
robot is what's gonna get companies to

spend money on these things that are
existential and critical and important,

then you know what, I'm fine with that.

Let's keep the wallets open and
let's keep moving into the future.

Bret: The reason

is less important than the outcome.

Laura Tacho: Yeah.

Bret: uh, mean, to be fair, we weren't
all pitching better documentation

as a business productivity boost.

that's never how I positioned it or
any other developer I worked with.

Laura Tacho: Yeah.

Bret: again, like we're going back to
this, that the hype cycle is promising

AI will make business faster, it makes
us write code faster, supposedly.

Therefore, business goals
will be met faster.

And I mean, this podcast, we, we
actually like to think of this

podcast as focusing on AI after the
commit, where so many other podcasts

are focused on before the commit.

That's why we don't talk
about which model we're using.

And if we do

Laura Tacho: Yeah.

Bret: dev tooling, we're
usually just talking about

it because we're all into it.

It's fun, not because it's the focus
of an episode, we've had, other guests

on this show and our, and Nirmal and
i's other show, DevOps and Docker talk.

We've had multiple shows now over the
last four, eight months talking about

the idea that AI is going to improve.

the standardization and expectations
of teams when it comes to the rigor

of everything from documentation
to automation, to standardizing

on infrastructure as code, like
all the things that we've all

been focusing on for over a decade

Laura Tacho: Mm-hmm.

Bret: to others and share our opinions.

But we didn't necessarily go at it with
everyone knowing that, this is gonna save

us money, this is gonna save us time.

And I feel like, you're right, that
AI, I guess the negative is it's

hyped, but the positive is we might be
able to use that hype to actually get

business to move on some of these things
that we've been trying for so long.

Because if I go to the team and
say, Laura's new project shows, oh,

look at this new talk for Laura.

It shows the stats that say, if we have
better confluence, if we have better in

repo documentation, if we have better
project planning that's linked into

other things, our AIs will actually
perform better for us and save us money.

Maybe that's what will
actually move the target.

So you heard it

Laura Tacho: Yeah.

Bret: just go spin that
to your leadership.

maybe you can get more, AI tooling
and more of the, stuff that

you actually wanted to work on.

we need more documentation,
but I don't know anybody

that's clamoring to make more.

So I'm excited that AI might
help me write it better and keep

Laura Tacho: Yeah, yeah,

Bret: find the outdated stuff better.

there's many opportunities
for improving it.

Laura Tacho: yeah.

I think documentation is a really good
use case for AI in a way that saves.

Time as a second order consequence,
like a second order indirect effect.

It's harder to estimate because,
the reason that documentation

has never been, it's always
been like a nice to have, right?

It's been a vitamin.

not a painkiller.

And I think that's been the experience
in general of developer experience

It's something you do to take care
of yourself because you're good.

It's not like my arm is cut
off and need to bandage it.

in reality that's what
developer experience is.

It's that your arm is cut off because
on average, developers waste 20% of

their time each week because they
can't find the right documentation.

They're waiting for builds to finish,
they're waiting for all this stuff.

20% of time, if you can imagine a roofer
putting shingles on your house, throwing

one out of every five shingles down.

to the ground or one out
of five nails did not work.

That company would go out of business
and no business leader would tolerate it.

But because it's hidden in knowledge work
and behind developers and we can kind

of brute force our way through it, we
don't know the boundaries of the problem.

And that makes it squishy.

It makes it seem like a vitamin of oh,
give the developers nicer documentation.

They'll like that when actually,
if you were to measure and, and use

some, you know, measurements like we,
we do in DX and try to identify how

many times per week does a developer
lose 30 minutes or more because

they can't find information due to
outdated or non-existent documentation.

look at that number now.

That number is your arm cut off?

Right?

That's not a vitamin anymore.

there is a bit of marketing around
developer experience and a lot

of talking about what it is.

'cause it's not ping pong and beer.

This is like existential problem.

And I know that we all know this and
that's why we have been working on

DevOps and developer tooling because
we know the real business impact of

it, but it has been historically really
hard to tie that to Dollars and cents,

which is what the business needs in
order to keep their wallets open.

I think AI, if anything, as you said,
it's overhyped or, hyped the right amount.

I don't know how much hype is the right
amount of hype, but, people sure have

their wallets open and maybe now's
the time to, take advantage of that.

Bret: I was just gonna reiterate and
repeat a little bit back to Laura,

that, in my entire career, the three
things on every project that get cut

first, and I always like to say this in
project meetings when we're all about

to cut something, it's documentation
monitoring and disaster recovery testing.

Like those are the, or just testing.

Those are the three things I see
on every project that left, on the

cutting room floor when the budgets are
tight or the timeline didn't get met.

if AI could improve,
just those three things,

Documentation, monitoring and testing.

If we could just improve those three
things, I feel like a bigger portion of

that graph that you were showing, and
for those that weren't seeing it, it

was a line graph of all the different
places that devs spend their time.

And at the very bottom

code generation.

Like the very bottom of it was ba pretty

Laura Tacho: Yeah.

Nirmal Mehta: so interesting.

It's

Laura Tacho: I.

Nirmal Mehta: like a, an outcome of this
data is that, if you're out there, if

you're listening to the show and you're
a dev or a platform engineer, engineering

team, and you're in your sprint, retro,
and you're looking at your backlog, if you

scroll all the way down, there's probably
a line that says, improve documentation

that has never made its way up.

Or, you know, refactor this
critical little piece of code

Bret: Yep.

Nirmal Mehta: improves the speed of CI/CD

Bret: Increased test coverage.

Nirmal Mehta: into, or,
one of the monitoring ones.

There's probably these, These
user stories that have been on

the backlog at the bottom You just
gotta scroll all the way down.

It seems like you, and, and if your
organization, if you're sitting there

and you're sprint retro and you're
figuring out, you're going through

your backlog and planning for your
next sprint, and your organization is

asking you to adopt AI tools, to, show
an outcome of 30% plus productivity

gains with using these tools, right?

Regardless of how you're gonna use the
tool, you might take this opportunity

to take a look at those items on
your backlog and push 'em up into the

new sprint because you can now say
that those are tied with this data.

You can show that those, backlog
items, those forever backlog items

are valuable in actually seeing those
productivity gains from these AI tools.

Is that the right interpretation?

is that what you're saying, Laura?

Laura Tacho: I think that is,
a really good interpretation.

I think where things get tricky is
building the business case around them,

Nirmal Mehta: Okay.

Laura Tacho: it's, really been
difficult for developers to

say, sorry, we're not gonna do.

X so that we can work on our documentation
increase test coverage build out different

test cases, work on our CI to reduce
flaky tests or make it run 15% faster.

It's been really difficult 'cause
this shouldn't be extra work, right?

we don't wanna overload
devs with extra work.

We need to take something away.

That's where things get really tricky
because when we have a feature that

we know is gonna bring in, you know,
we have a, it's $500,000, let's just

say that I can't hold up documentation
improvements or, or CI/CD improvements

next to $500,000 and make it as close
to an apple's to apple's comparison.

It's so difficult to do that
without data because we can't.

conceptualize how much money
that is costing our company.

And so often what we end up is like,
sure, that feature might bring $500,000

of new revenue in the door, but this is
gonna allow you to accelerate delivery

for everything else in the future by 20%.

And how much is that worth in terms of
salary, cost avoidance, and time to market

realization for your future investments.

we can do the math and figure that out.

but that's where platform engineering
teams, where software development

teams, like app development teams
are having the hardest time right

now is making those arguments.

And that's where DX, I mean that's our
really, our backbone is like drawing

the boundaries around how much it's
costing you to have these problems.

Helping you figure out how to
prioritize them so that you can

reduce the friction and stop
wasting 20% of your time every week.

'cause that adds up really
fast and it's just not fun.

So, why don't we, why don't
we talk more about AI stuff?

Nirmal Mehta: Yes, assistive
versus agentic was another

thing, but Bret, go ahead.

Bret: Yep, So

Laura Tacho: Um,

Bret: agentic.

What does that even mean?

Laura Tacho: what does that even mean?

So I think that.

When we talk about AI or like when AI is
talked about in general, it's kind of like

this big pool AI is such a broad term.

even thinking about those use cases that
I talked about AI for code gen versus

AI for refactoring code versus AI for
running after the commit pipelines,

that's already a huge variation.

And then when we talk about the
interaction modalities of AI, we've

got, am I talking to ChatGPT in a
separate window asking it like, oh,

how would you plan, this feature?

Like what are the questions I'm missing
If I'm gonna do a pen test on this?

Like what are some edge cases that
I might have missed versus code

completion in the IDE versus agentic
workflows that need human approval

versus ones that are running in swarm
mode with many agents doing work.

AI is such a huge term.

And so when we talk about The shift in
the spectrum from assisted to agentic.

We're talking about human in the loop
ness, maybe I would call it like that.

with assisted engineering,
we've got chat modality, we've

got auto complete in the IDE.

as we get closer to agentic, maybe
we're relying more on workflows that

are defined, and running, you have an
agent doing each of the workflow for

you, but you're still saying, yes,
do this and verifying and then saying

proceed and verifying and proceeding
very much human in the loop versus

like a totally agentic workflow, which
is like, I need an app that is going

to analyze my Pokemon card collection
tell me which one has the highest market

value and send me a push notification
every morning with the highest value

card, build it and deploy it to AWS.

And you don't look, you know,
you're not in the loop at all.

that's a very big spectrum versus
like pair programming with ChatGPT to

giving something production access.

So as an industry, if I were to imagine
all the companies I'm working with and

see what they're doing and put them
on a scatterplot, we're definitely

more toward the assisted with a
little bit of dabbling in the agentic.

But I would say more like workflow
is starting to be that early adopter,

early majority kind of bleeding edge.

I've seen some interesting examples
of agentic, but nowhere near industry

wide adoption of ag agentic models,
or patterns of working right now

Bret: That

Laura Tacho: September 1st, 2025.

Bret: yeah,

Nirmal Mehta: because Bret constantly
is asking me, but Nirmal, our, our

companies and customers actually
using these agents to do like SRE

tasks or operations or DevOps.

Like are they, where's
the actual use cases?

And I,

Bret: me the money is, yeah.

Show me

Laura Tacho: Show me the money.

Nirmal Mehta: I'm gonna deflect Bret's
question that he's constantly asking

me to you, Laura, I know you just
mentioned that, on that scatter plot.

it's definitely not the majority, but are
you seeing some early signs of, agentic

workflows, not only for business use
cases, but also for DevOps, tasks like

operations, SRE, CI/CD, things like that

Laura Tacho: yes.

I would say as a short answer, yes.

I think the complexity is the
variable influencing, how and to

what extent that's being used.

looking at SWE-bench, this hasn't changed
since I, looked at it recently, but we've

got 67% of tasks resolved and a lot of
these models are hovering in like the 55%.

So like the technology
is also just like not.

There.

So this is, for those of you that don't
know about this leaderboard, this is

like testing the model's capability
to complete a task end to end and

have it be correct a coding task.

Bret: Yeah.

Nirmal Mehta: one

Laura Tacho: um,

Nirmal Mehta: or is it through a workflow

Bret: It

Laura Tacho: um, Ooh.

Bret: tool.

if you look at the list,
each one might be like.

Just this model or this model
plus some additional tooling

that turns it, like warp

Laura Tacho: Yeah.

Bret: You know that their agent model
that's using a particular LLM like

sonnet four gets this percentage.

So just to clarify, when Laura says 70%,
which she's saying is that that particular

setup, like that LLM plus that tool

Laura Tacho: Mm-hmm.

Bret: 70% success rate on hundreds,
potentially of GitHub issues

that it's trying to solve for
a particular software project.

And that's what we've talked about
before we've actually mentioned this

is I'm so glad you brought it up.

Because

Laura Tacho: Mm-hmm.

Bret: best models are only getting three
out of four or two out of three PRs

Laura Tacho: yeah.

Bret: Correct.

Laura Tacho: Yeah.

Bret: it's, it's not fixing.

And so, and that, and a lot of us.

we are all very lucky.

We have access to the best
models, to the best tooling.

we live in a western country where
we can afford all these things.

I do hear on occasion feedback that
a lot of the world isn't here yet.

A lot of the world has to use either
free models or local models, or

free stuff like the free ChatGPT
equivalent or whatever tokens they

get for free with GitHub Copilot.

And they don't necessarily have the
best tools unless their company buys

them because they're just so expensive.

Unless you live in a, in a country that
is indexed to the US dollar, essentially

Laura Tacho: So I think the
technical capability isn't really

there for organizations to rely
on agentic workflows for things

that are gonna touch production.

Have I seen lots of experimentation
with agentic workflows for stuff used

internally and doesn't necessarily
hit end users Absolutely, definitely.

there's a lot of experimentation,
but when we talk about it widespread,

especially in the enterprise, we're
still away ways away, from that being

reality for, for a lot of folks, I
would say like one step back toward

assist, you know, one step away from
Ag agentic, which is like workflows.

I have seen many kind of enterprise,
fast scaling, high growth organizations

have their platform team responsible
for creating workflows that, for

example, do legacy modernization or
dependency updates and make those

available to other development teams,
their customer developers internally

as part of their platform offering.

So this is not quite agentic.

There's still more human in the loop,
there's more human verification.

but companies that are really
investing in AI have done a lot

of training and enablement to.

Get all of their developers with not
just an understanding of like how

the, the tools are used, but like
organizational understanding of how

to apply it to their business context.

they are experimenting with, you
know, agentic adjacent, we'll call

it that, agentic adjacent, workflows.

that's exciting and promising.

I think, by the end of the year,
maybe in Q1 of next year, my answer

will likely be very different.

Nirmal Mehta: Well,

Bret: Yeah.

Nirmal Mehta: have to bring you back to
give us an update on what you've seen.

so

Bret: a moving target

Laura Tacho: I wanna share just one,
you know, kind of thinking about how AI

can't make up for all of the devex gains.

I think agentic has a lot of promises,
but there's still so much work to

do when it comes to identifying
organizational use cases for AI

to help deliver software faster.

This is, um, I'm gonna publish
this I think this week actually.

Um, this is looking at data from over
a hundred companies ish that have

developer populations between 105 hundred.

And I'm looking at their
annualized cost savings.

these companies have said, this is
what a full-time employee costs.

This is how much our developers are
using and saving from AI each week.

And so this is the annualized cost
savings in US dollar from using AI.

obviously there's gonna be a trend line.

The more developers you have, the
more you can save because math.

but even if you look
at some of these like.

200 developer, 300 developer mark.

We've got companies saving over 6
million, in recovered salary costs.

And like, you know, this company
not that much of a different size.

It's like not even breaking a million.

It's not that these companies
up here are using Ag agentic

workflows to do a bunch of stuff.

It's that they have done really
good training and enablement.

They've found organizational use cases.

they've equipped their organizations
with AI instead of spraying and

praying to their individuals and
hoping that curiosity and grit

will take it the rest of the way.

there's so much room for
optimization all around.

Ag agentic workloads are incredibly
interesting and I'm very optimistic.

the reality is that I just don't see
them being widely adopted, especially

in customer facing environments
and in the enterprise just yet.

But again, my answer will probably change.

Nirmal Mehta: So this is interesting.

Is

Bret: Yeah.

Nirmal Mehta: normalized based
on, the developer's salaries?

Laura Tacho: the companies that
I've included in here have given us

information about how much a full-time
developer at their median salary point.

this is adjusted.

I have another version of this,
which I'll publish along with this,

where I pivot basically on region.

and you can see like the, the lower
bands there's a lot of Europe down here.

'cause the salaries in Europe
are just generally not as high.

these are US companies and then
like Asia companies are kind of

spattered, across, but that seems to
be like a pretty in aggregate, like a

representative trend, for what we're
seeing, also extending to bigger companies

Nirmal Mehta: I would
love to see that pivot.

on the left hand side,
the annual cost savings.

per, normalized, developer salary
or something like that, like a

little bit normal normalized, just
so that it's easier to compare.

this is fascinating.

Laura Tacho: yeah.

Nirmal Mehta: information.

so what are some of the, you
kind of mentioned, I mean these

Bret: It

Nirmal Mehta: are actual, this is like
real money that we're seeing, right?

Bret: Yeah.

And if I had to describe this to the audio
listeners, looking at a graph that is.

all over the map.

there is nothing here that is certain,
there is no clear location where like,

this is where everybody, like the majority
of people are seeing this level of

Laura Tacho: Yeah.

Bret: It's basically every company
of like this size you said, right?

100 to 500 engineers, I think is what you

Laura Tacho: Mm-hmm.

Bret: and they're all over the map.

I mean, as you can might have
guess the ones that are saving

the most money are rare.

there definitely seems to be more
concentration on the low end of benefit,

but there is no clear cluster here yet.

this doesn't look like a typical sort
of Gartner, approach where everyone's

huddled in one place, I think this is
just indicative of any new technology

that we don't really know how to use yet.

We don't really know the patterns of

Laura Tacho: Mm-hmm.

Bret: know a lot of people are trying,
and you might be wasting your time.

You might not.

Laura Tacho: Absolutely.

that's exactly how I interpret this.

And I use this to set up, like it's,
just look at the, for the audio

listeners, we've got on the X axis
engineering organization size and on

the Y axis annualized cost savings
in USD, but only for companies

from a hundred to 500 engineers.

So if we look at companies that have.

200 to 250 engineers, We've got
all the way from, five and a half

million down to 800,000 for the
same exact size of engineers.

And so what that tells us is I mean,
maturity, we can look at how long

has AI been introduced, but what it
comes down to in my experience, kind

of like anecdotally and I'll put
together the dashboards to prove this.

'cause I do have data to prove it as
well, but it is developer experience and

it is training and enablement technique.

if a company makes licenses
available to engineers and says,

okay, go for, it encourages people.

Even make a Slack channel, share your
good practices that will not bring as

much, effectiveness as actually Nirmal
borrowing, like your experience-based

accelerator, like Amazon's kind of
method of bring a real business problem

and then bring the technology together
and like, let's figure out how to solve

the real business problem as a team.

And then create a blueprint
for that and spread that out.

That has been incredibly useful.

booking.com, for example,
they have 3,500 engineers.

they have a 30% increase in merger
requests, throughput, but they were

able with that technique of the
experience-based accelerator doing

modernization projects and getting
70 to 80% of the way there with AI

using that technique from Amazon.

the median adoption for developers in
an organization is a little over 50%

of developers using AI on a daily or
weekly basis for AI assisted engineering.

booking has 60 plus percent using
it on daily and weekly, and of those

70% are using it on a daily basis.

So they've been extremely successful.

They have a huge ROI, it comes
from the training and enable.

and treating it as an organizational
tool and focusing on the workflows

that AI helps to enable and not just
an individual productivity tool where

Nirmal can type twice as fast Now.

Nirmal Mehta: Interesting.

So this mirrors the same kind of
conversations I was having with

organizations two decades ago, or
a little under two decades ago,

Bret: Hmm.

Nirmal Mehta: With the adoption
of DevOps practices, organizations

back then, I think Target

famously had like a dojo, a
devops dojo where they would bring

teams in with these um devops.

Subject matter experts take a workload,
and re-architect it and train the

trainer and get the developers and
the platform engineers up to speed

around that specific workload.

migrate it, modernize it, and then.

Put it through production
and then take the next team.

And so what you're saying is
this is no different, right?

AI adoption, developer experience with,
AI tooling, organization, if they want to

be successful with these tools and really
see some gains, they need to take the same

approach, of standing up some kind of,
outcome oriented, what do you call it?

Like a, uh, dojo if you want.

We could probably call it dojo,
like an AI dojo, for adopting these

tools to be successful and not just
say, oh, here is an X, Y, Z license.

report back to me in a quarter
where, you know, how you've been

30% more productive regardless of

Laura Tacho: Here's

Nirmal Mehta: doing.

Laura Tacho: the bottom line and
maybe a good, um, bottom line to,

wrap our conversation up, which is if
organizations wanna have organization-wide

impact increased productivity across
the board, they need to treat AI as

an organizational tool, not as an
individual productivity speed up.

And the organizations that do that,
that treat it, like you said, like have

an AI dojo or treat it with, you know,
enablement in the same way you would with,

with any other kind of organization-wide
tool, they're seeing big impact.

And the ones that are just spraying and
praying licenses aren't gonna see that,

that impact because it's, there's nothing
like, there's nothing new under the sun.

Right.

this is a tool.

As any other developer tool,
no one would've bought.

Jenkins licenses are like Docker
licenses and just expect like, oh, we're

gonna modernize our whole application.

It's gonna run in Docker now.

here's a license.

Go figure it out That didn't work then.

It doesn't work with AI now.

even though it's sparkly and
shiny and there's a lot of

hype, like the fundamentals in
the physics haven't changed.

and I think there's a lot of
level-headed leaders, like both of

you, Bret and Nirmal, treating this
with the right amount of scrutiny

and skepticism and optimism.

leaders who can do that are gonna
be the ones that come out ahead

than the ones who are, you know,
spraying and praying and following

victim to the hype, unfortunately.

Bret: So you said something about two
or three weeks ago, what happened?

Two or three weeks ago.

Laura Tacho: this has been a core
concern for so many of our customers

and so many like CTOs and VPs.

'cause again, what decisions would
you make if you knew that 60% of your

developers were using AI for, for AI
assisted coding, but 0% of that code

made it to production versus if you
had 10% of your developers using.

AI tools daily, weekly, and
50% of the code going to

production was written by AI.

those are two wildly different
scenarios, and it's important

to figure out where you are.

we at DX have a tool that runs on the file
system layer so it can track in any IDE

in the terminal, whatever, code completion
from AI tools as well as copy paste and

then track the human editing and then do
that on a per line basis, at the commit

level so that we can look at the ratio.

Of AI code to human authored code in
a given commit, and then we can track

that commit all the way to production.

So there's like pockets of tools that
could do this before, like Windsurf had

some pretty good telemetry into what
was going on, but before we just sort

of had acceptance rate, which is, you
know, accepted suggestions and really

no insight into what happens even like
before the commit, after the acceptance,

and then definitely after the commit
and into customer facing environment.

So.

excited about this.

It's technically very cool.

Bret: Is this.

Nirmal Mehta: scary to me.

Laura Tacho: industry

Nirmal Mehta: That,

Bret: Is this a part of DX?

is this a part of the DX platform,
or is this an open source project?

what is this specifically?

Laura Tacho: it's proprietary,

Bret: Okay.

Laura Tacho: the DX

Nirmal Mehta: sorry, that shot should
not have been my first reaction to that,

it sounds really cool, but it does.

So the scary part to me is it sounds very,

Laura Tacho: Brothery.

Nirmal Mehta: yeah, like survey and,
you know, we could have another whole

entire episode on this, but this
kind of comes back to, incentives

by leadership to use this hype for
ulterior motives, let's just say that.

Laura Tacho: Yeah.

Nirmal Mehta: I think it is important
for us to know how much of that AI

code is out there in production.

I think that's a very legitimate concern
on all aspects, especially from a platform

Bret: yeah,

Nirmal Mehta: and a
DevOps team perspective.

I think very crucial
information, but at the

Bret: to be fair,

Nirmal Mehta: for monitoring
copy and paste from a developer

like, oh boy, you know.

Laura Tacho: Yeah.

Bret: Yeah.

I mean, it's not actually
uploading the copy, right?

we're not talking security concerns.

We're just talking.

Laura Tacho: no,

Bret: Because this is, this sounds
way better to me than a decade ago

when I was freelancing through certain
websites on the internet to random

companies when I was first starting
my freelancing business 15 years ago.

back then you were billing per hour
and the tool you were using for

consulting would screenshot your
computer every five to seven minutes

randomly and send it to the client to
prove you were working on their tool

because you were billing per hour.

That's way more invasive than this.

I had to put up with that 15 years ago.

But, anyway, this sounds a lot better.

Laura Tacho: Well, again, like the
double in the details, so change

failure rate is one of the four
key DORA metrics that organizations

have been tracking for a decade.

in order to get that number.

We ask a question to individual
developers, this is how it's

done for the Dora report as well.

And then accelerate, you know, in
the last month or whatever timeframe,

how many of the changes you pushed to
production resulted in degraded service.

so we can have a data point for the
individual and how many of their changes

resulted in degraded service, but we
don't look at them on an individual basis.

We look at them in aggregate.

the point is about measuring the system,
not about measuring the individual.

Nirmal Mehta: Mm-hmm.

Laura Tacho: And the same
is true for these AI tools.

Like we're not.

Looking at, Bret had 70% AI authored
code and 30% where Nirmal had 10%

AI authored and 90% human authored.

We're looking at it in aggregate to
figure out what's the shape of the

code that's running in production?

What's the security risk?

How is AI being adopted?

Is it being adopted by, you know,
like bearded men are using AI a

lot more, you know, like, different
attributes and demographics.

Unfortunately, I have to tell the
audience out there, DX does not

have the attribute of bearded men.

Bret: it's not a checkbox.

Laura Tacho: actually that's not true.

We can do any custom attribute.

So, Bret AI

Bret: Okay,

Laura Tacho: if we were on our
own team, I would definitely

have that as an attribute for

Bret: great.

Gray beards.

I want the gray beard checkbox.

Yeah.

Laura Tacho: totally.

but it's all, the same threats
and the same kind of concern

about Big Brother surveillance.

measuring the system versus
measuring the individual.

They all still exist.

And even more so with AI because
the, the threat of like, my job

is gonna be at risk because this
technology is gonna replace me.

I feel like we're in a pressure cooker
and there's pressure coming from all

Nirmal Mehta: Yeah, that's

Bret: Yeah.

Nirmal Mehta: I feel like we're
in a pressure cooker right now.

Laura Tacho: Yeah.

Bret: Yeah.

And so if the audience can feel anything,
we're with you, we feel the pressure,

you're probably listening to this podcast
'cause you probably have a little bit of

that pressure that you're feeling to stay
up to date and be on the leading edge.

part of it's curiosity and
passion around that new tech.

Part of it's the obligation
that we feel like our jobs are

somehow maybe at risk someday.

I mean, there's so much confusion
and news about everything.

I go up and down in emotions
every day around AI.

it's probably generated more
frustration and emotion in me than

anything in recent memory that
I can think of in terms of tech.

Laura Tacho: Mm-hmm.

Bret: So, given all that we've talked
about what people can do from here,

obviously it's a complicated answer.

I feel like what we're saying is, really
anything in your software life cycle

potentially could be beneficial with AI.

But you need to make it a project and you
need to take it seriously and you need

to train people and do all that stuff.

I feel like that's the big take.

But I wanted to switch a little
bit to a slightly adjacent topic

that we haven't discussed yet.

Context switching.

one of the potential hopes I have
and I think others have for AI is

that it may actually help developers
with all this context switching.

And you showed a graphic earlier
of like the top 15 activities a

developer does on a given week.

Everything from meetings to staring
at Jira to improving documentation to,

oh yeah, I also get the code right?

Laura Tacho: Mm-hmm.

Bret: all of those context switches.

Nevermind the idea of switching
between one code base and another.

Um, do you have anything on con?

Is there, are there, are there
lessons learned from this?

Is there tools we can look at?

Because it's one of my top
goals to improve my ability

to context switch with AI.

I don't have any good tools.

don't have any good methods
or any good information on it.

So I was hoping we could talk about that.

Laura Tacho: I wanna, just take this
time to highlight that focus time as

a driver of developer experience is
capturing exactly that, like cognitive

load context switching kind of tax
that developers pay from like bopping

around from different tasks, but then
also having their flow interrupted,

like, you know, I'm coding and then.

I have to go to a meeting.

We don't get enough focus time to
really get into the flow state.

And so that is one of the kind of
hypotheses is that if we use AI and we can

automate the toil that's interruptive, do
developers actually get more focused time?

And I wish I had like a more conclusive
answer because I did look at this

very recently and like the answers are
a bit all over the place right now.

Even controlling for.

usage and different things.

we track it as, a first level, primary
metric on this AI, impact board

where we can do group comparison.

So if we wanna compare like all AI users
to non-AI users, we can look at how

that impacts their focus time, because
that's really important in this case.

the group that's using AI actually has a
lower amount of, Focus time, which isn't

Bret: Rent.

Laura Tacho: isn't great, but we can see
how that trend evolves, across like the

different usage of like infrequent monthly
to weekly to daily and how the focus

time is impacted because that is, again,
like the mechanics and the physics of.

Good devex and like good team
habits, like focus time is still

important even when we have AI.

And a lot of companies that I work
with do have the hypothesis that AI

can automate, toil give more time for
focus and increase the innovation ratio.

So this percentage of, time spent
on feature development, we see this,

kind of sample group has 4.8% higher
allocation toward feature development.

So that's, that's great.

Their innovation ratio, we're looking
at things like quality, speed, velocity,

maintainability, and kind of making sure
all those things stay moving together so

that we don't over index on one metric.

use AI to preserve context, I leave
little notes for myself in chat.

GPT.

And within my five hour Claude code,
session, I find it so helpful to

not feel the pressure to have to
write a bunch of post-its to myself.

Like I can just

Mm.

from my task and come back to it.

And I've like really accelerated
the time it takes to get back into

things just because I have this like
programmer who never gets tired and

is always there for me, except after
five hours when he magically goes away.

Nirmal Mehta: Okay.

as we're wrapping up, for our
listeners, we have individuals trying

to navigate these rapid changes,
that pressure cooker kind of feeling.

what's some advice you would
give our audience, that they

could use today to navigate that?

Laura Tacho: I think first
thing is sort of a mindset.

Switch, like average median time savings
per engineer per week from using AI

tools for AI assisted engineering
is a little under four hours.

that data comes from more
than 38,000 developers.

When you see a headline, know that
that's a headline for a reason

and like get curious about what
might be going on behind it.

So think that's step one is just to
like identify what hype have a better.

idea of identifying what type.

I think the other thing is, it's.

Really difficult to make an impact
with AI if you can't measure it.

you know, if you don't know
what's actually happening.

And so getting instrumentation in place
for measuring things like maintainability,

DevEx, speed, throughput, quality, these
can be done lightweight with a survey.

it doesn't need to be this huge robust
ETL process and data lakes everywhere.

You can do so much for getting
visibility into how your organization is

operating now and how AI is changing it.

Do that, easily for very low
cost other than the time it takes

to do it for you as a person.

that data is gonna be instrumental
in helping you make arguments for or

against certain strategic decisions.

on a leadership level, I
think that's another thing.

and I think the maybe more tactical
way is if you are feeling the pressure

for AI more in your organization.

Look at how AI is being presented.

Is it an individual productivity
tool where people get, a

license and then they're sharing
interesting things that they do?

Or are you doing this AI dojo kind
of scenario where you're looking

at problems that teams have and
problems that the organization

has and finding ways that AI can.

Solve those problems organization
wide, that's where you're

gonna get the most leverage.

focus on those things.

if you do those three together, it's
gonna take some time, but this sort of

gets you out of the fog a little bit.

truth be told it doesn't actually matter
if Microsoft is writing 30% of their code.

Because you're not Microsoft.

it's really difficult to not get a bit of
FOMO or not feel the pressure, especially

if you're getting pressure from your
exec team about, well, I saw this in

the headlines, why aren't we doing that?

But coming back to them with data
about what our business needs and what

we're doing, and how I know that we
need to do this and not this next.

That is what is gonna get
you ahead and stay ahead.

instead of just chasing metrics, vanity
metrics that you see in a. LinkedIn

post on a random Tuesday afternoon

Bret: Unless it's yours,

Laura Tacho: unless it's mine.

Bret: unless it's your post.

Laura Tacho: none of my
metrics are vanity metrics.

they all have adequately sufficient n
sizes, or I call it when they don't,

when a median might be unstable.

But yeah, it is really hard to
differentiate that because not

everyone, you know, LinkedIn is for
half of a brain and half of a thumb.

and a lot of, the shiny graphs
get the, most shareable.

Bret: speaking of that, so, uh,
thanks so much for being here.

Where can people find out
more about what you're doing?

where are you chatting?

On the internet.

Laura Tacho: I am only
on LinkedIn these days

Bret: Okay.

It's an easy place to be.

Laura Tacho: I tried to do Blue Sky.

I really did for you,
Bret, and for you Nirmal.

And I just really couldn't do it.

Bret: I don't get likes
there, so I'm the same way.

no one sees and pays attention to my stuff
other than Nirmal and a few other people.

Laura Tacho: other than.

That's very cute.

Um, find me on LinkedIn I'm
the only Laura Tacho out there.

and you can read about more of like
my research and guides on getdx.com

if you go to the research tab.

I'm doing a lot of research and white
papers, and I heard from Nirmal, if you

Google me, you'll come up with a lot
of interesting, talks that I've done

Nirmal Mehta: Yes.

Laura Tacho: I've been, at the
Enterprise Tech Leadership Summit

in Vegas, talking about booking.com
and how they scaled AI adoption.

I'm gonna be doing some, other keynotes.

I'll be at Cloud native Austria.

Actually, it's not, it's
a CNCF, like adjacent, not

Bret: Hmm.

Laura Tacho: but I'm
very excited for that.

I'll be keynoting there
as well in October.

Bret: Ooh.

Laura Tacho: I'll also be at KubeCon.

I'm gonna be at.

Platform Engineering Day, talking
about the real adoption rates of

AI tools across the industry to
help dispel the hype a little bit.

we'll be at Backstage Con,
and I'll see both of you there

Nirmal Mehta: Yep.

Laura Tacho: exciting.

Bret: that's gonna be an exciting talk.

we talked the last, KubeCon we were hoping
for more talks, this turnaround on the

actual using of AI for DevOps and platform
engineering rather than the running of

AI and GPUs which is what everything's
been focused on for three years.

it'll be fun to see that kind of stuff.

Nirmal Mehta: Yep.

Laura Tacho: totally.

Bret: where can people find you?

Nirmal Mehta: with you, Bret.

so Agentic DevOps fm, our podcast
that you're listening to right now.

on top of that, I'm on LinkedIn.

Metaverse hq, on Instagram
and Blue Sky, and then, mostly

on LinkedIn, Nirmal K Mehta.

you'll find me.

Bret: Yeah, we're all on LinkedIn.

Laura Tacho: kind.

Nirmal Mehta: Yes.

Laura Tacho: Yeah.

Bret: LinkedIn as much as anywhere
else now, so yeah, it is what it is.

well, Laura, thank you
so much for being here.

We're gonna have you back, actually,
this is a spoiler that we already are

planning about maybe another episode,
but we won't talk about that yet.

We'll talk about when it actually happens.

I'm excited to have you on this show.

We're so glad to have you back and
I love this new focus of yours.

Nirmal Mehta: Yes.

Bret: Yeah, we were on the container
bandwagon for a while, but I think

this might just be your moment
and, Yeah, we're here for it.

Nirmal Mehta: we're here for it.

Thanks, Laura.

Bret: All right.

Laura Tacho: Thanks, Nirmal.

Thanks, Bret.