Why AI pilots fail
Sep 2026
Dave Tobias and Jake BurnsWhy do promising AI pilots fail in production? Dave Tobias and AI strategist Jake Burns unpack the data, cost, expertise and employee buy-in needed to turn experiments into impact.
Sep 2026
Dave Tobias and Jake Burns)
){kind=logo}
){kind=logo}
){kind=logo}
“Too often, leaders implement AI without communicating a compelling reason why, or they rely on vague goals like ‘becoming more agentic.’ The ‘why’ needs to be tangible and connected to the company’s mission. But before leaders can communicate that connection, they must understand it themselves: Why are we doing this, and how does it advance what our organization exists to achieve?”
Jake BurnsAI Strategist and TechnologistEpisode 1 transcript
Intro - 00:01
Hello, and welcome to the Reality Layer podcast. I’m Dave Tobias, chief product officer at Nearmap. Every good decision runs on a layer most people never see. The data, the truth on the ground. This season, that means insurance, underwriting claims, catastrophe response, and policyholder experience. Each episode, we sit down with people closest to it, the ones making these calls, and living with what that data actually shows. Let’s get into today’s episode. Today, I’m joined by Jake Burns, an AI strategist, technologist, and former AWS executive in residence. Four decades working in technology, he’s advised more than a thousand executive teams around the world on AI, cloud, and large scale transformations.
Dave - 00:43
In this episode, we’re gonna explore why so many AI initiatives stall, what it really takes to move from prototype to production, and how leaders can make smarter decisions about the people, the technology, and the strategy behind AI. Welcome to the podcast, Jake. But as you’ve been in some of these rooms, Jake, with these executives in some of the biggest companies in the world, they’re trying to figure out AI. What are you hearing from these leaders right now? What are they excited about? What are they confused about? What are they scared about?
Jake - 01:15
It’s kind of a mixture of all of those things. And there’s also different cohorts of people and leaders within the world right now. There’s those who are just really excited about this and just want to kind of go all in with it and there’s those who are skeptical. There’s very few that are in the enterprise and in large organizations that are really understanding how to do this properly and to be frank that are being really successful with it. So we’re still in kind of this mixture of excitement, but a little bit of disillusionment as projects are stalling and progress isn’t going as fast as possible in these large organizations.
Dave - 01:53
So you yourself, you’re someone who’s building with these tools. What are you seeing personally right now when you’re using this stuff that you’re like, jeez, this is moving faster than I got? Or the flip side of that, what is not moving quite as fast as maybe we all think it is?
Jake - 02:09
It’s funny you mentioned that because this has been a really incredible week with at least four major model releases, and last week was similar. So it seems to not be slowing down at all. So it’s a full time job just to keep up with the progress of kind of the models that are coming out, the new technology that’s being developed. But some of the areas that I’ve been working on from a technical perspective is really giving agents memory, which is, like, interestingly, still not a solved problem. I mean, there’s partial solutions, but it’s one of the things that’s gonna be really necessary in order to get the most out of this technology. The way I look at it is the technology that we’re going to use for the next, let’s say, decade, I estimate we’ve embedded about 10% of that so far. In terms of like solving business problems, really what I’ve been focusing on the most and what I’ve seen kind of the biggest need for is just getting organizations to be able to take advantage of the large quantity of data that they have, just kind of trapped in silos in the organizations. This is, like, a problem that every large organization seems to have, and it’s a problem that is solvable with this technology.
Dave - 03:08
So I’m glad you brought that up. We’re talking here today about insurance carriers. I’d never walked into an insurance carrier that didn’t have vast amounts of data. Insurance, by definition, is a super data rich environment. We have application data coming in. We have property intelligence. We have inspection reports, claims data. We could argue some is accessible and in the right format, clean. Some is not. What do you tell them? Where do you tell them to start so that they can start executing on some of these tools and ultimately get to faster, better, more complete decisions?
Jake - 03:43
So what I used to tell them is that you have a prerequisite and that is to kind of fix your data problem. It’s like data strategy. It’s kinda like information security. It’s like last priority all until it becomes first priority. And now it’s first priority for everyone and there’s all this tech debt these organizations need to address. Now the problem is for a large organization addressing that kind of tech debt like the data hygiene specifically, which is just can be sprawled across an enterprise, collected through acquisitions and mergers, just tied up in silos, just even knowing what you’re dealing with, that could be a multi year process for these organizations. So kind of the conclusion I came to was it’s a little bit unrealistic to be prescribing that because you can’t wait on this technology. If you wait that long, then you’re going to fall behind. And so I’ve kind of changed my approach to let’s work with kind of what you have today. Let’s make the technology better so that it can work around kind of limitations within your organization while you simultaneously kind of address the tech debt because it’s still a good thing to do.
Dave - 04:44
Well, that’s interesting. You’re saying there’s no time like the present. This train is moving, and if you wait, you’re gonna miss this train and the next train and the next train and the next train because new models are coming every day. And so that’s what I’m hearing you saying is don’t get too caught up on the minutiae, albeit the data is important and the cleanliness is important, but you gotta start.
Jake - 05:05
You can use this technology to fix the data problem. That’s the interesting thing as well. So the technology is a moving target. It’s not just the models. It’s everything surrounding the models. It’s the harnesses. It’s the applications. It’s the tools. If you just kinda get started, even though there might be a gap between what the technology can do today and what you need it to do, just get started today. Like, sometime in the first half of the project, the technology is gonna be where you need it to be to accomplish that goal. So I think people tend to aim too low into what they’re trying to accomplish. I guess one piece of advice is just aim higher. If it doesn’t feel unrealistic, then you’re probably not aiming correctly as to, like, how you should approach solving these kinds of problems.
Dave - 05:43
So you’ve described AI to me sometimes as being deceptive in certain ways because it’s really easy to create a proof of concept. I could Vibe code you DocuSign. I could make you a working version of DocuSign, give me a few hours. But do you really wanna sign your home loan on Dave’s DocuSign or your next $10,000,000 deal on Dave’s DocuSign where you can’t really prove the provenance, there’s not that trust and certification, that gap between POC, VibeCode, let me show you something working, but a POC to actually productionalization into the enterprise, it’s a big gap. Is it bigger than leaders expect going into this?
Jake - 06:25
Yeah. It’s huge. That’s one of the top three, maybe top two problems organizations have today is they can all create the POCs. They could even have their engineers create pilots that work, and then they fall apart when they try to scale them up. The past, like, how do I create a pilot stage, they’re stuck in the how do I get out of pilot to production stage. And that’s where they need the most help.
Dave - 06:46
So in that gap, that messy middle in between POC to pilot to production, where do you see this falling apart?
Jake - 06:53
The problem is that it falls apart in multiple places, even just cost, for example. The range of what a solution can cost is almost infinite and I think this is just true with technology in general, like the more powerful the technology, the wider that range become. With AI, it’s wider still, so there’s this perception that AI is very expensive and the reason is because there’s very few people who implement it properly and when you implement it improperly, it is very expensive. It can be prohibitively expensive, but the interesting thing is when you could solve the same problem and create the same system and do it slightly differently and it could cost nearly nothing.
An example is a company I’m working with today. They had a false start, which most do, and they spent multiple millions of dollars on system that never got out of pilot stage. Been working with them for a couple months and we have a early production system that costs a very, very small fraction of that. So just a different approach to solve this exact same problem.
So my point is like, it doesn’t have to be expensive and if it is prohibitively expensive, I think it’s a good idea to kind of slow down and take a look at it and kind of evaluate, are we doing this the right way? Don’t just keep going.
Cost is just one of many areas where this falls apart. The people and culture is another big one, but that’s probably in the top three as well. So creating a prototype or a POC doesn’t require buy in from the wider organization, but once you start scaling things up, it does. And that’s when you start discovering there are people within your organization. This is true for every organization, that they don’t have a positive attitude towards AI. They don’t want it to succeed. Either they’re skeptical about it working or they fear it because they see it as a threat to their job or a threat to their ability to do their job because they don’t understand the technology.
People would be probably the second one. There’s governance and there’s observability and then there’s security and compliance and all these other things that you have to worry about.
Dave - 08:45
So you said something interesting to me before, which was employees are resisting AI more now than they were in the early days. So if that’s true and people are actually starting to resist more, if I’m a leader in a big enterprise, how do I introduce this to my people in a way that maybe gets them to think differently about how these tools will help them and help the organization?
Jake - 09:10
I think that’s the most important question you could possibly ask because if you don’t get your people on board, then it’s pretty much impossible to be successful with anything serious. So there’s a couple things that need to be addressed, put in two categories.
There’s one, like, the confidence in the technology and so one of the reasons why the attitudes are getting worse is because we have just more failed projects that we did, like, a year ago. Again, it’s an uncomfortable truth, but most of these projects are not going well. They should be going well. The technology works. It’s just there’s not enough people with high expertise.
The other reason is the fear reason and it really comes down to, like, did you get buy in from an organization? Any kind of transformational change, and this is kind of like the ultimate transformational change, is only possible if the people within the organization want to transform along with the organization. In fact, they are the ones that transform. The organization is just made of people. So if they’re not ready to do that, if they’re not wanting to do that, then that step one has not been done and you can’t go on to step two.
So getting buy in from an organization, you could think about it as like just kind of three things that are necessary.
There’s kinda like the why. So, like, a lot of leaders, they’ll be implementing AI systems, and they never really communicate why we’re doing this or they communicate a kind of very uncompelling why. Like, we’re doing it to be agentic or we’re doing it for some reason that it’s not tangible. So tying it to kind of your company’s mission is very important.
Second is how and you could frame this as kind of like having confidence that it’ll succeed. The truth is that nobody wants to be part of a failed initiative. Everyone wants to be on the winning team. If they don’t see a reason how it can succeed, then they’re very unlikely to go along with it. So while they may trust you as a leader that you have their back, they may not trust you as a leader that you can pull this off and if you don’t have that second kind of trust, it’s very hard to get buy in from an organization.
Making the people within your organization co owners and co architects of the system, giving them a sense of ownership can kind of solve that how and give them confidence.
And the last thing is what’s in it for them? So if you leave it to their imagination, they’re most likely gonna think I’m engineering myself out of a job or this is gonna be replacing my job, but the truth is if you look at what’s going on in these organizations that successfully deploy these systems, people’s jobs are evolving and the demand for workers is actually going up in these organizations that are successful with this. Now they’re different kinds of jobs, but there’s a transition path into those kinds of jobs. People wanna know what this is gonna be good for me rather than bad for me.
Dave - 11:34
All problems are people problems in an organization, in my opinion. AI can intensify that, of course, but the technology can always be fixed, changed. You can get the bugs out, etcetera, etcetera. The people problems are the toughest ones. Just to give the listener some hope here, Jake, have you seen organizations do this well?
Jake - 11:55
There are enterprises that have pulled it off, so it’s certainly doable. It’s not a large percentage. It’s not the majority of them. They’re the ones that happen to have strong leaders that understand these things that we’re talking about, about buy in. They have the trust of their organization and they have a vision like they’re doing this for a reason, they know why they’re doing it and they can communicate that.
All problems are people problems and all success are people success, right? The technology doesn’t do anything on its own. It’s like AI is a force multiplier, so it takes what you have, whether it be your people, your data, all of these things in your organization, and it multiplies it and it’s neither good nor bad. So it can multiply your mess, but if you give it people who have a positive attitude and have skills, they could be tremendously powerful and force multiplying that.
So people are at the core of all of this and if you solve your people problems, if you get your people motivated and skilled, it’s been my experience that they will solve all the technology problems for you and you don’t have to worry about those.
Dave - 12:55
Often what I hear in the insurance world is we’re a regulated industry. We have insurance rates and regulations we need to deal with. Have you seen companies in regulated industries be able to work through these issues? Because to me, I don’t want myself or our industry to use that as an excuse.
Jake - 13:12
It’s not an excuse. This is completely doable in regulated industries. Now I think the problem is that most people, when they think about AI, they just think about chat GPT or Claude or any of these consumer grade applications.
In fact, this is one of the reasons why you need to adopt AI in your organization is because if you don’t, then the safety valve is that your people go out and put it into a public chatgpt.com and then your data is out there.
So people are gonna use this technology and the key is for any enterprise, you wanna protect your data and you wanna do things in a compliant way. In highly regulated industries, the stakes are just higher but, you know, the approach is pretty much the same. It’s just build an enterprise grade system with this technology and I have yet to see an industry that’s regulated so much that you can’t use this technology.
You just got to make sure that the data is either, you know, depending on your requirements, it could go to an enterprise API or with the system that you own, so data stays in your network and it only goes out through providers that provide zero data retention and maybe have the compliance requirements that you need.
But there’s even more strict requirements where you can’t even have enterprise APIs. You can’t have the data leave your four walls at all. And that’s where I think kind of open weight models and locally hosted models become useful or working with a company that has those capabilities in house that could provide you with the compliance as to what is that you need.
Dave - 14:34
Well, we see in the insurance world, and then there’s some public
Jake - 14:37
releases about some of the insurers
Dave - 14:39
building their own LLMs and keeping them within their walls. I think for the reasons you mentioned, twofold, one, for security and compliance, but also because their belief is that they have some special data that can give them an competitive advantage fed into that tooling and that LLM and that harness that they don’t want others to have access to. Cost comes into play there too, I would imagine.
Jake - 15:02
For sure. And I think training your own model isn’t even really necessary. You could kind of host your own model without training it, but it’s very important to realize that these consumer grade systems, they’re highly subsidized. So they make the cost very attractive because they’re getting value out of your data.
You’re always paying something, you’re either paying money or you’re paying with your data, but if you have proprietary data and data is your differentiator within your organization, then it’s a very bad deal and you should probably just pay the money and it doesn’t have to cost a lot of money to use these public models through enterprise APIs.
It just takes kind of understanding what you’re being charged for and how to optimize for that. So just without getting too technical, it’s a metered service, so it’s just like using electricity.
So you can think about there’s very wasteful ways to use electricity within a business and there’s very frugal ways to use electricity within a business. Now, if you’re paying per token metered, how can we design these systems in such a way that we are using fewer tokens?
I’m confident I go just about any enterprise and take 90% off of their token cost within a few weeks. I mean, that’s how much waste most organizations are, how wasteful they are with their tokens just because they don’t really understand how the technology works and how to optimize it, but it is optimizable, very highly optimized.
Dave - 16:22
It reminds me in a lot
Jake - 16:23
of ways of early cloud, and I know you are a big part
Dave - 16:26
of that and transforming organizations over to cloud infrastructure where it seems so cheap and you’re like, I’m gonna put all my storage here, and I don’t care. And it’s gonna be I’m just gonna put it all there. But over time, the subsidies ran out, and it became very costly. As organizations, as enterprises, we need to think about it as a core infrastructure, like, we think about our phone systems, our cloud compute, and everything else. Right? It becomes core infrastructure over time, doesn’t it?
Jake - 16:53
It does. And I would even generalize that more. It’s a skill. Like, using it as a skill. And I think this is kind of the biggest failure mode is just not really appreciating that.
And it kinda goes back to what you’re saying before about it’s so easy to create a POC, gives us this false sense of like, oh, this is easy. Anyone could do it. Very much the Dunning Kruger effect is like, we’ve never had better examples of it than with AI.
You learn a little bit, you say, oh, this is easy, and then all of a sudden, you scale up a system where you’re sending all of your data in every request and, of course, it’s gonna cost a ton of money.
I see it at organizations where they’re just sending everything to, like, the most expensive model and maybe you need the most expensive model for certain things, but then vast majority of the tokens can be generated by a less expensive model.
So many levers that you have, but the problem is that not only do the people who are implementing this generally not know what those levers are, they don’t even know that those levers exist. So there’s a lot of education that we still need to do within this industry kind of to scale success.
Dave - 17:48
But what I’m hearing from you is don’t let cost be the thing that holds you back from getting going. We see a lot of these POCs fail. Why do you think if you were gonna pick one or two spots, is it bad execution? Is it that they picked the wrong thing to try to solve? Is it that they didn’t go big enough like you said earlier?
Jake - 18:06
Yeah. Let me pick one. It’s for execution. The root causes are just not having expertise and not appreciating the need for expertise with this.
It’s so funny to see these job postings and, like, you know, five plus years requirement of agentic engineering and stuff like that. The best people in the world have only been doing this for a few years, right, and the best people in the world are not available, generally speaking, so you’re probably gonna have somebody who’s been doing this for a few weeks or a few months and how much expertise can you really have in something in that period of time no matter how smart you are.
Organization could fail three, four, five times in a row very spectacularly and very expensively and the board will just tell the CEO, do it again, try again, keep trying. And that’s the right answer, honestly. Like, you can’t stop trying with something like this because we had such a huge disadvantage not using this technology.
But it’s a real conundrum because if you’re continuing to fail and it’s very expensive and you can’t stop trying, what do you do?
So it’s really about it’s execution that comes from a lack of expertise and so what leaders need to do is like this is what I see they accept this kind of excuses.
We’re three months in, we have seen known results, but my engineers and my people are telling me, just give me another month, we’ll get it.
I think you have to be kind of like ruthless with your expectations. You need to expect if you’re not seeing results within weeks, then you probably don’t have people who are able to do. More weeks and more months are not going to help.
If you’re not seeing results within a fraction of the budget that you set aside, then more budget is not going to help.
So really kind of course correct as early as possible, stop and kind of evaluate what’s going wrong, get different opinions from people and I know it’s not easy, but try as hard as possible to get at least one person who has a track record and a high degree of expertise in doing what you’re trying to do, even if it’s in another industry because people get so kind of obsessed with like insurance, for example, like people in insurance want like have you done this in insurance before?
I think it’s kind of the wrong answer, the wrong question. The question should be have you done this before for any industry? Because the lessons translate.
Now, of course, there’s insurance specific things that need to be considered but chances are your organization has insurance specific expertise. What you’re lacking is an agentic expertise And so you just need to get that expertise and then expect results quickly and inexpensively.
And if you don’t get that, realize that you don’t have that expertise that you think you have, and don’t keep going until you get that.
Outro - 20:33
That was the reality layer. We hope this conversation gave you a cleaner read on the reality on the ground. If it was useful, subscribe whenever you listen, and share it with your colleague shaping the future of PNC.
)
Meet the Host
Dave Tobias
Dave is Chief Product Officer at Nearmap, where he leads product strategy across the property intelligence portfolio. He co-founded Betterview, the application now used by carriers across the US for underwriting and risk selection, and joined Nearmap through its acquisition in December 2023. He has spent more than a decade building the tools insurers use to understand property risk.
)
Meet the Guest
Jake Burns
Throughout his career, Jake has led large-scale IT transformations, which today are the basis for his firm belief that any enterprise can successfully adopt the cloud and save money at the same time. He served as VP of IT Operations at Live Nation and now as an AWS Executive in Residence, Jake works with enterprise technology executives to share experiences and strategies for how the cloud can help them increase speed and agility while devoting more of their resources to customers.
See the Reality Layer
Every episode covers claims, catastrophe response, and the AI reshaping insurance technology.
View more episodesExplore the latest
)
Nearmap Labs
Explore the research and early-stage technology behind what's next in property intelligence. Get hands-on with new tools before general release, and meet the people building them.
Explore Nearmap Labs)
Customer Stories
See how carriers put property intelligence to work across underwriting, renewals, claims, and catastrophe response, and what changed once they did.
Read Customer Stories)
Webinars
Go deeper on the topics the show covers. Sessions with the practitioners and product teams closest to the work, available on demand.
View Webinars