Guest: Jon Reed, Co-Founder & Analyst at diginomica
Host: Andreas Welsch
Episode: What’s the BUZZ?, Season 4, Episode 23
Published: September 23, 2025
Episode Summary
Jon Reed, Co-Founder and Analyst at diginomica, joins Andreas Welsch to answer what comes next for AI agents in the enterprise as the fall event season begins.
They start from the underwhelming GPT-5 release and an MIT study in which, Jon says, 95% of the generative AI pilots examined did not get past the pilot phase, and they ask what the 5% that succeeded did differently. The conversation then covers AI readiness, including culture, data and the middle managers who must enable their teams, and the security, trust and authentication questions that arise when agents act for a company. Jon closes with a fall watch list that covers customer success stories, MCP and tool calling, and caution about agent handoffs across vendors.
Key Takeaways
- Jon reads the MIT study as less damning than its headline suggests. About 5% of the pilots it examined got past the pilot phase and achieved big success, and he cites 30% success for projects run alone against closer to 65% with an external vendor.
- Jon calls domain context the last mile problem. Out-of-the-box language models do not understand specialized industry settings or customer data, so in the view Jon relays, the systems built to incorporate domain insights matter more than model sophistication.
- Jon says AI remains an accelerant. In his words, “when you apply AI, you enhance what you’re good at. But AI can’t make you good at things you’re not good at already.”
- Andreas recommends weaving data work into an AI project as a phase zero when no dedicated data budget exists, and Jon recommends rolling out use cases while they clean the data.
- Andreas asks whether a company’s own agents are authenticated and whether another company’s agent is trusted, and Jon advises starting with MCP and tool calling while staying cautious about agent handoffs across vendors.
Timestamps
- 02:29 Why the GPT-5 release signals limits of scale
- 03:40 The MIT study on generative AI pilots
- 09:42 The last mile problem for domain context
- 17:24 Agentic AI for long-tail procurement spend
- 23:42 Shared learning communities and middle managers
- 27:16 Security, identity and authentication for agents
- 33:22 Data platforms and zero-copy scenarios
- 35:50 Data work as a phase zero of an AI project
- 43:30 MCP, tool calling and agent-to-agent protocols
About Jon Reed
Jon Reed is Co-Founder and Analyst at diginomica. He joins What’s the BUZZ? for a third conversation with Andreas Welsch this summer.
In this episode, Jon describes the AI readiness research he has done since the previous episode and his plan to document more customer success stories at the fall vendor shows.
Full Transcript
Read the full transcript
Andreas Welsch: Today we’ll talk about what’s next with AI agents, and I’m so excited to have Jon Reed, esteemed analyst and co-founder of diginomica, with me again. Hey, Jon. Thank you so much for joining.
Jon Reed: Yeah, so glad to be back. It’s really been great to have this dialogue going this summer, and it’s been great to get the feedback on it as well. And I’d like to think we’re getting somewhere, but things are also moving really fast, so we’ll see how we do.
Andreas Welsch: Yeah, exactly. Look, the first two episodes that we’ve run were a huge success. We had lots of great feedback, lots of interest, and people engaging in a good dialogue. So if you have any questions in the audience, please feel free to put them in the chat. We’re super excited to hear from you and we’ll answer them as we go along here. But Jon, without further ado, obviously it’s been an amazing summer. We’ve come from the spring event series. We are now seeing agents mature a little more. We’re talking more about security, trust and authentication, and these kinds of things, but to me those are only some of the topics of what’s next. What are you seeing? What’s next for you?
Jon Reed: Every day is a different day, and it’s actually been, I think, a pretty significant week. It’s going to be interesting heading into the fall event season. A lot of big shows are on deck. I know you’re going to a number of them also, and we’re really going to get a gut check pretty soon on customer adoption and where everything stands. But one thing I thought was really fascinating, just in the last week, you had the underwhelming release of GPT-5, with even a lot of AI fanboys disappointed in the release. And I think the big takeaway, more than anything, is that traditional approaches to just throwing data scale at AI problems aren’t working very well. We’re seeing a real divide now between the big AI companies and this pursuit of artificial general intelligence, which we could debate, versus a really different approach to enterprise use cases that I think is far more promising, that recognizes more technical limitations and constraints, but then applies them very cleverly to enterprise and industry scenarios. We could talk a little more about that and why I think that’s a really important distinction, because you can run the risk of thinking that if these models are underwhelming, then they’re not going to work for my project. Along those lines, this week was another instructive one, because we saw a ton of headlines for this MIT study that 95% of generative AI projects fail. What’s interesting about that is that as soon as it came out, a bunch of people published little articles on it, none of which I think were particularly insightful. And then it turns out to be a group inside of MIT that is working, I think, on some specific stuff around agentic AI. They immediately pulled the study offline. I found it, but now we’re all talking about a study that no one can actually really find. I think that’s a really good summary of the potential for confusion. And when you dig into the study, and we can maybe talk a little bit about it today, it’s not like this damning “we can’t do anything with AI” thing. In fact, what they found, alongside the idea that 95% of these pilots they looked at in their study, which is a pretty small sample size, by the way, a few hundred companies, didn’t get past the pilot phase, is that the 5% that did had really big success. That’s the tantalizing part. Not just small success, but big success. And the other interesting thing is that when you dug further into the study, and I don’t know totally how to reconcile this yet, there were different numbers that indicated that 30% of projects were successful on their own, but closer to 65% were successful when working with an external vendor. This is something we talked about in past episodes, where I think a lot of customers right now are going to avoid trying to recreate their own cutting-edge AI architectures, and turn instead to trusted vendors to help manage the complexities of these environments. So I just think it’s really interesting, because in just a week’s time we had so many interesting juxtapositions between different ways of thinking about the exact same thing.
Andreas Welsch: Yeah. It’s hard to comment without coming across as being a little cocky. But I think if you’ve been in the trenches, if you’ve seen this play out over the last few hype cycles, machine learning being one of them, you know that it’s not that easy. It’s not that simple. It’s not just about the low-hanging fruit and “let’s boil the ocean and see what we can do with this magical technology.” It’s about where we have a problem, whether we can measure the impact, and whether we can solve it with the data we have and the technology that’s available. In very few cases is technology the limiting factor, right? Either we haven’t understood or articulated the problem well enough, or we’re not bringing our people along. Yes, there’s technology involved as well, but as we’ve also seen in previous releases, whether it’s GPT-5 or 4 or 3, some of them were more groundbreaking than others, but technology itself isn’t necessarily the limiting factor. I like that you mentioned the MIT study. I was going to bring that up, too. I need to update my slides. For years, I’ve been saying it’s 85%. Even here on the back cover of the AI Leadership Handbook, it says 85% of AI projects fail. That goes back to 2018, when Gartner said they don’t deliver value, not just fail. Now we’re at 95. Okay, fine. But I think it goes back again to this fundamental part: have we even understood what the problem is that we’re trying to solve? The second part for me, which I’ve also been sharing for years, is don’t build everything yourself just because you can. There are vendors, there are companies out there that do that for a living, in the areas where you use software anyway. So look at where the opportunities are. Is there AI in these products we already use? How do we get it? Is it an additional subscription? Is it a higher tier? Does it even add value? And start there, instead of first building your AI platform and everything else around it for this one little use case that might or might not play out the way you hope.
Jon Reed: Absolutely. And I want to talk a little more about this AI readiness concept that we touched on in past episodes, because we did get some feedback on that. But one thing I wanted to share is a really interesting juxtaposition with this broader failure-rates topic and the underwhelming GPT-5 release, and these other vendors, whether it’s Grok or Anthropic, also releasing, I would say, incremental improvements in a lot of ways rather than drastic jumps like we had between GPT-2, 3 and 4, for example. As a contrast to that, I was watching a video on YouTube. I’m not going to call out the video, because I’m somewhat critical of part of it and I don’t want to get into a back and forth with this person, but it had to do with turning LLMs into domain experts. And I think this is the heart of the enterprise play. What this individual said is that when it comes to vertical AI applications, the systems that you build for incorporating your domain insights are far more important than the sophistication of your models and your pipelines. So they were saying the limitation these days is not how powerful your model is, but whether it understands the context in your industry for your customer, and whether it performs the way you need it to for that industry. I think that’s really interesting, because how do you apply these large language models to specialized industries? There’s a last mile problem, and the last mile problem has to do with giving that model the specific information, context and understanding for that particular industry setting. When you take that on, there’s really promising ground to be made. The key thing to understand is that LLMs out of the box aren’t going to understand a lot of these specialized contexts, because even though they’ve been trained on the vast internet, they haven’t necessarily been trained on these very specific domains and this customer-specific data, for a variety of reasons, including privacy. That’s the really interesting contrast, I think. You have someone like this talking about achieving upwards of 90, 95, 98% accuracy rates in those domains, focused on specific constrained problems, and that’s really exciting. Now, one of the really interesting questions is whether that enterprise promise that people are starting to see is enough to maintain the whole AI economy. The answer is uncertain, because the big AI companies are counting on valuations that I think extend pretty far beyond that. This doesn’t necessarily save the whole AI economy, but what it does is create opportunities for people like you and me and opportunities for vendors that focus on the enterprise to bear down on these industry problems and provide LLMs with the context they might not have.
Andreas Welsch: I think so too. To me there’s so much potential that is still untapped. And I see that largely, like we’ve talked about in our previous episodes, in the small and medium-sized business sector: people and organizations who have a need to drive more automation to increase the speed at which their business is able to function, but who might not necessarily have the deep pockets or the resources that larger or global organizations do to get consultants to fix this or build an army of engineers to do that for them. I think there is a huge opportunity in at least two areas. One, how do we use the tools that are available to us? Could be things like Copilot or ChatGPT or some other tools. Some basic AI literacy: how do I use these tools safely and responsibly? How do I make sure that whatever I pass on to my colleague or to my customer has been checked by myself or by somebody on the team, so I don’t just generate AI slop and send it on? On the other hand, how can we, in very concrete ways, optimize and automate some of our standard processes? Maybe that could be an RFP process, where we send an RFP out to a number of suppliers, then we get responses, and now we need to figure out what they responded to and which parts they can actually complete. All of these things where you still have a lot of people doing this work today, as one example, right? Or content creation, newsletters, simple things where organizations have people doing these tasks for a good amount of their time a week or a month. We’re not tapping into those yet. So to me, that’s a big area. I also want to say I was just in Europe for a couple of weeks and the conversation there was really interesting. When this topic of data and agents came up, yes, models are great, they’re trained on vast bodies of information and language, but they don’t know your specific business data and processes and product formulas and what have you. The next topic that came up was maybe it’s small language models, either specialized or trained on industry knowledge, or that you fine-tune on your business data. That could be one approach. The next topic that came up was whether we are seeing a pendulum that keeps swinging from on-premise to cloud, and now, with costs and privacy concerns, are we seeing the pendulum swing back, with larger organizations putting Nvidia hardware back in their data center to run these models on premise so they have better control? Those are two of the big questions that came up, and to me they’re super fascinating. I’m not seeing that here in the US yet. I don’t know if you are. I think here it’s still API based: we call a model from somewhere. Certainly some organizations and industries are more concerned about what’s happening with their data, what vendors potentially do with the data and with the prompts they have. But it was more noticeable in the conversations I had in Europe.
Jon Reed: Yeah, and this is part of the experimentation, in a good way, that people are going to go through to find the right size models for their use cases. I’ve talked with vendors who start with larger models and then are able to scale that down, which is a little bit like what DeepSeek did to some extent. Basically you can diffuse or boil models down to smaller models once you have established a use case. But there are characteristics of different size models that have to be analyzed too. For example, larger models are often better at the language understanding part. If you have customer-facing use cases, you might want the larger model, which does a better job of understanding what the customer is saying, because they might use different words that mean the same thing. Whereas if you have, for example, a team of super users using an internal model, they can be taught the right ways to interact with that model. They might do just fine on a smaller model where they realize, okay, I need to prompt it with these kinds of words, but it won’t understand these other words. It’s just really interesting. I did a podcast this week with Aaron Harris, CTO of Sage, and they essentially found they had to train an LLM, because out-of-the-box LLMs just didn’t understand finance terms well enough to actually use them. We got into that in the podcast. Sometimes you’ll need to do that. Other times you won’t, and other vendors are doing just fine using RAG and tool-call-type context and a more out-of-the-box model. So again, it comes down to use case. In the case of the finance one, for example, in the podcast we talked about how if you ask a question like “what stock item will I need to reorder soon?”, ChatGPT wasn’t recognizing that as an accounting question. It didn’t recognize the accounting terms. The Sage model that they built understands that stock refers to inventory, and that’s the world the model is operating in. It just depends on your use case. That’s why you have to dive into this with expert partners and expert advisors to help you head down the right path. If you do, I think you start to build on these use cases and notch some real wins here, even though in the background you have this study that says 90% of projects fail. But the kind of projects that are failing, in my opinion, are not the kind that you and I are talking about right now.
Andreas Welsch: That brings up a good point. I did a webinar a couple of weeks ago with cists who are in the procurement space, and we had a panel of experts and also got lots of questions from the audience. Some of the audience questions were like, “With AI, and especially agentic AI, what can I do in procurement?” And there were some really great examples where they said, for example, something like the long tail: you have individual vendors or users in your business who buy from individual vendors, but it’s not at a volume where you can negotiate large discounts or larger contracts, and it’s hard to manage and hard to analyze and categorize the spend and consolidate it, so people are not doing it. But now with agentic AI, they said, we can actually do that. So we are not replacing people doing this job, because nobody’s doing it anyway, and we’re giving businesses better insights so they can consolidate spend and optimize there: category management and real value in business results from that. To me that was a great example. Also, what I heard over the summer was the whole debate about whether entry-level jobs are going away. Everybody panicked. We looked at labor statistics, and STEM graduates seem to be in a higher unemployment category this year than they have been previously, things like that. Then luckily the debate switched to: if we don’t have entry-level roles, how do people become senior? Who are the next senior and expert-level roles? So please don’t just eliminate the need for early-in-career roles. You need to figure that out too. Anyway, two stories: one is where we deploy agentic AI and how it can add value, things like long-tail spend in procurement where we’re not looking at this anyway, and on the other hand we’re not taking jobs away. This mindset is really important, because I also see too many leaders think about where they can cut cost, where they can cut spending, where they can reduce headcount. To me, the bigger question is what I can enable if I have this booster, not just a productivity booster but different insights, new insights, scaling our knowledge, scaling our information acquisition, retrieval and reasoning. I’m not seeing a lot of leaders think about that yet, but I would absolutely encourage them to approach it with that mindset rather than “where can I save?”
Jon Reed: Yeah, and this is something you and I hit on in our last discussion around use cases: the importance of companies taking a step back and deciding what their innovation is heading towards in their industry. What do they want to be known for? I’m hoping that a lot of companies want to be known for the way in which they excel in how they serve their customers and the opportunities they provide for employees. That doesn’t mean they’re not going to want to be operationally efficient. Of course they do. But it’s so important to have those goals firmly in mind, because as we discussed in our service scenario last time, it affects your decisions on whether you’re going to deploy AI to serve customers better, for example round the clock after your service employees go home. Are you going to use that to reduce your service team, or, ideally, based on the use case I wrote about, enhance the service you can provide to VIP customers by taking admin off the plate? So again, it’s the same kind of thing. And the same is true from the talent side, right? Do you want to use AI to cultivate talent, as you were talking about, or are you going to use it to just get rid of your junior staff, if you can get away with that? I would argue that a lot of junior staff are more talented than bots still. But let’s just say that you’re going to try that. That becomes a very self-defeating model going forward. You need to ultimately make technology, and I don’t care if it’s blockchain or AI or whatever you think, subservient to your bigger goal, and not allow the technology to take over the goal and say, “I can’t help it, AI claimed those jobs.” No, they claimed those jobs because you allowed that to happen. You can redeploy those people. You can treat them differently. And when we talk about AI readiness, which I think will be the next topic in our discussion here, a big part of that is the cultural part: how do you cultivate a culture of employees that is excited about using these tools, and not having them imposed upon them or feel like “I’ve got to work harder because I’m going to lose my job,” but more like “I’m being provided with secure ways of experimenting with these tools so that I can ideate use cases for my company to excel.” That’s the kind of environment you want to create.
Andreas Welsch: So that brings me back to the MIT study that we talked about at the beginning, and one of the nuggets in there was that we need to empower middle managers to empower their teams and their team members. It’s not enough to say “We’re AI first, you need to use AI,” because your employees are struggling. They don’t know what this means. Where do I start? How do I even use it? How do I know that something I create is good? Maybe they don’t even know that you can create something that is not good or factually inaccurate. I heard two great examples last week in a session that I did. One of the participants said, “Yes, our leadership team is AI first, you need to use AI in our company,” and people look around and say, what does it mean? So I helped them figure out how they can use, for example, ChatGPT or Copilot or AI in other parts. Another participant said, “We’re going about this a different way. Our leadership also encourages the use of AI. We have smaller groups or communities of multipliers or champions, if you will, and we come together and say, here’s how I use it in my function. Here’s how I use it in marketing. Here’s how I use it in sales. This is what’s worked really well. Here’s how I got to that point. Here’s what I had to change. Here’s where it fails.” So we create this culture of shared learning and community-based learning. We accept AI as a new technology, as a new way of working, still with a mandate that we want to use more AI to be more efficient and effective, but we also give people the opportunity to learn together and share visibility and praise of how people are using it in our organization. To me, those were two excellent examples: the same goal, different execution, and very likely different results as well.
Jon Reed: Yeah, and if you can get your hands on the report, and hopefully it will be issued at some point, they get into this 5% and what made those 5% successful. It’s quite interesting to dig into that a little bit. The more successful areas are no surprise: high-value use cases that are integrated deeply into workflows and scale through continuous learning rather than broad feature sets. Less generic productivity things, more very specific functions that can be automated. They talk a lot about back office admin being successful, as opposed to throwing all of the money into various marketing and sales agendas, though they did note that lead gen can be a successful area, as well as service. What I think this report didn’t get into is that a lot of industrial use cases are gaining momentum too. I talked with a vendor just last week about some of their success starting to embed shop floor functionality. You can start with things like quality assurance and equipment monitoring. You’re not giving agents control over your shop floor yet, necessarily, but there’s a lot of really interesting stuff. And again, it comes down to customers that are ready. The reason the AI readiness topic is powerful is that AI remains an accelerant, to me. If you have really good processes and good data platforms in place, AI is just going to help you outperform that much better. But if you are struggling in those areas, I don’t believe that today’s AI can save you from yourself by applying it onto problematic siloed data, problematic process flows, lack of leadership buy-in, stifling workplace cultures, people clinging to their jobs like life rafts. These things are not conducive to AI deployments. That’s why the AI readiness conversation is so important: when you apply AI, you enhance what you’re good at. But AI can’t make you good at things you’re not good at already.
Andreas Welsch: Beautifully said. I don’t even know what I should add to that, other than a confirming nod in your direction. There’s an additional component that I’ve been thinking about a lot more lately. I hosted Steve Wilson, one of the co-leads of the OWASP Top 10 for LLMs report, several times on the show, a couple of weeks ago as well, and we got into this topic of how you secure your agentic AI. I saw some interesting examples of vendors thinking about security and identity and authentication, and it got me thinking, right? Today we have largely humans sitting in front of a screen, clicking on the screen manually. Maybe you have some service accounts or some API keys if something is happening in the background. But now, already with agents, we have a non-human entity that acts on your behalf or acts on your company’s behalf. A lot of times when we talk about AI and agents, we talk about my company’s view: how can I use this, how can I optimize my process, how can I improve my operations? But there are two, maybe three things that I think we’re not talking about enough yet. One is how you build and scale and configure your operations for agents on the other side that will send you requests all of a sudden, more and more, much more quickly, at a higher pace, a higher velocity, higher volume. Are your systems designed for that, or how do they need to change? How do we need to think about agents at some point doing business with other agents? Is that other agent even trusted and trustworthy? Are they really representing who they say they are? Is it really my business partner that’s behind this and their agent interacting with my agent? So trust, security and authentication, even in this scenario between companies, become much more important. And what we’re just scratching the surface with is whether the agent in my business is even authenticated and trusted.
Jon Reed: Exactly. And the thing that I have seen again and again this summer is that we cannot underestimate the security issues involved here. There are new threat vectors, and the risk management around this is really important. One way to think about this is a difference between a human scenario, where you welcome someone into your house because you trust them, and an agent would do the same thing. But if you say you have free run of the house, do whatever you want, an agent might do that too and just let you do whatever you want. Once it has authorized you, it’s not going to ask questions, whereas with a human, if you started rifling through my laptops, even though I gave you free run of the house, I might say, “I don’t need you running through my laptops right now. What do you need right now?” Agents aren’t really well equipped to ask those kinds of questions. Once you’ve been granted access, the access you’re giving these systems is quite powerful, so it’s not a deal breaker, but you have to have this at the forefront of your mind at all times. And it’s important.
Andreas Welsch: Yeah. Even on a smaller scale, say you’re a sole proprietor or small business owner, yes, you have access to tools like Manus or OpenAI’s Operator, or now the automation workflows, basically. But what happens when you give it access and it sends confidential information to another customer, to another prospect, that wasn’t intended for them? They shouldn’t see what you quoted your actual intended recipient, but now they do. Big miss. So to what extent do you give these agents, these technologies, access to your system, to your data, and how do you mitigate those risks? I think that’s a real question we need to ask, and that vendors also need to help figure out if they want to see more adoption. It’s fine that it can click on a screen and does things independently, even if it takes a little longer than it would take us as humans. But this part of risk management, risk mitigation, and just understanding what could go wrong and what that could mean, is so important in this discussion, especially as we’re talking so much about hype and all the future and everything it’s going to enable. It works well until it doesn’t. And then the question is, who’s at fault? What’s the damage? So it’s better to think about that upfront, as always.
Jon Reed: Indeed. Since we last talked on the AI readiness topic, I’ve spent a lot of time looking at this, and I think we’ve covered a lot of different components of it: things like leadership and culture and process discipline and data. The interesting thing is that the data part is still something a lot of vendors are arguing over. It does seem very clear that AI services thrive on different kinds of data presentations than we typically have had in the past. What’s the most optimal way to do it? Some are saying you need a real-time live data stream, because AI thrives on the latest live data. That’s true to a point. But there are times when real time can be prohibitively expensive in certain domains. I talked with some vendors about this who said yes, that’s true, but some systems that I might need for AI aren’t being updated in real time, and that’s okay. So you’re back to the right-time thing. And some people talk more about cloud-based access and how important that is. Other people insist that it doesn’t need to be in the cloud. There’s a lot of discussion of edge-based devices, but I think the edge stuff is a little tricky, because the compute on edge devices is often not enough for what you need, and then you have various cloud processing steps from edge devices. So no one has figured out all of the answers to this question. I think one prevailing trend is not to move data around as much, but to think more in terms of zero-copy scenarios, where data can reside where it resides and you have some kind of intermediary system that can help you with this. It’s a new way of thinking about middleware, but in an AI context, I guess you could say. No one has figured all of this out yet, and there’s no way to avoid some of the pain around it. The way I look at it, a core component of AI readiness is going to be your data platform vision for your company. That’s probably going to be a multi-year scenario in terms of how you make your data most accessible to these systems. But the good news is that you’re going to find places where you can start, where you do have a high caliber of data. And people say these systems can handle unstructured data. That’s true to a point. But even with unstructured data, these systems thrive better when it has certain kinds of recognizable labels and formats that they can make sense of. So you’re going to learn as you go what the best data is for those systems, and that’s why trusted advisors are so important. I just wanted to say there’s no consensus on all of this yet, but the one good thing we know is that starting with areas where you have quality data is a really good idea, because you need to be doing actual live work with quality data to see what the results are and iterating on that, rather than testing these systems with data that is not your own, because it will not give you the input you need on whether to go forward or not.
Andreas Welsch: I spoke in Munich at an event a couple of weeks ago, and afterwards some attendees came up to me and said, “You talk about AI, you talk about the need for good data, clean data, fresh data, accurate data, all of that. We know that our business, our data is nothing like that. And we’re not the only ones, right? So how can we do this? We’ve been trying for years to get our data in order, to get our leaders to buy in, to get money, to get funds and do things. We don’t really have time to do a multi-year roadmap. What can we do?”
Jon Reed: Right.
Andreas Welsch: And so one of my recommendations was, think about how you can do some parts of it as a phase zero of your AI project. If your leadership or your board gives you money to invest in AI, yes, you need to get your data house, lakehouse, what have you, in order to do AI. So think about where you can weave this into existing projects or new projects that we’re starting, with a manageable effort to make progress. Similar to what you said, data is not a new topic. It’s been there for years, for decades, and it hasn’t been solved yet, because usually it’s not the sexy, shiny thing that people want to invest in or get visibility or credibility for. But we need to do this work now more than ever, if we expect to have AI, if we expect to have agentic AI on top of that, using the data to reason, to make decisions, to generate proposals, to research information, all of that. So the time is now. If we don’t have a dedicated data budget or a data project, see if you can weave that in as a phase zero to an AI project before you get started with it.
Jon Reed: And a lot of the excitement that I feel about enterprise AI, I don’t know about you, is the combination of structured and unstructured data, because behind transactions in systems are back-channel conversations about particular customers and relevant information about the history of interaction with that customer. The idea of combining all of that into one way of engaging with the system to find out “tell me more about this customer,” not just their sales history but the feedback that we’ve gotten from them and all of that, and having all of that as part of one workflow, is so appealing. The good news is that a lot of companies have that somewhere. I don’t think it necessarily has to be cloud, but the reason cloud products have been appealing is that when you move to a SaaS product, a different level of data discipline is often imposed upon you, because you need to become a little more vanilla, if you will, to fit into a classic SaaS environment. So what you’re looking for from a structured data perspective in terms of AI readiness is systems that are not incredibly heavily customized internally, but are a little more out of the box, where you have had to impose some discipline around structure and work through some meta issues around customer names or regions or whatever it is you’re looking for. And like you said, everyone has been burned by these initiatives in the past. The good news is that with iterative projects, you’ll have the ability to come back to your team and your leadership team and say, “Okay, we started by working with this. The data was already pretty clean here. We got a win. We were able to increase customer satisfaction. We were able to reduce turnover. We were able to land new clients or optimize lead gen, or whatever it is.” That gets leaders excited, and then you can say, “And now we need to go a little further here.” That’s the hope. In the past, the problem was that we were doing these massive cleansing efforts, but there was never really any external result. It was “when we finally get it done, now we can look for the use cases.” The idea here is that you roll out the use cases as you’re cleaning the data. That’s the distinction. If you can do that, I think you’ll be able to develop momentum to move into the thorny areas, and it will happen, because what you’re going to run into is that your users or your customers will ask questions of your AI system that it can’t answer because it doesn’t have access to certain data. And that’s going to be your next project: how do we add that data into this mix?
Andreas Welsch: And it sounds like that’s a much better problem to have than the agent answering questions where it does not have the data and making things up, right?
Jon Reed: Oh, much better. Yeah.
Andreas Welsch: So, definitely lots to do and lots to work on, for sure. We said that today we’ll talk about what’s next. We covered security, trust and authentication. We talked about AI readiness: on the one hand, getting your people and your leaders ready, and on the other hand, data being a big topic where we need to increase readiness to do AI on top. I’m just super excited to see how things are moving in the industry and how quickly. Two years ago, a year and a half ago, roughly, the first agent frameworks came out, then vendors latched onto them. They said, okay, here’s a real opportunity, and maybe to some also a threat, we need to do something about this. Now they’ve shifted. Now there’s the adoption, starting slowly but surely. I’m excited to see more examples. By the way, I also feel not enough companies are talking about their agentic deployments yet. Sometimes there are little bits and nuggets that you pick up in tech media and coverage, but I have seen very few companies say, “Here’s how we’ve deployed it. Here’s how we looked at tasks and how we broke this down into what the agent does and what the subject matter expert does.” Very few, maybe for good reasons. Maybe it’s the competitive advantage for those that are already ahead. But I’m sure there will be more that we will see, and to me that also unlocks the next wave of what is possible, what we should be thinking about. We talked a couple of weeks ago about Moderna combining IT and HR to look at roles. Whether or not that’s the right approach is, I think, to be seen. We’ll see that probably in the next three to six months or a little later. But the first companies taking this initiative and this approach, looking at how we can divide work, where we need people because of their unique capabilities and knowledge and where we need agents, and how that looks: it’s moving super fast and it’s exciting to see how that progresses.
Jon Reed: Right. And as we move to the end of this third discussion on what’s next, when I go to the fall shows, one of the big things I’m going to look for is what you just described: I’m going to try to document more customer success stories of what’s been achieved so far and what the hard-won lessons are. I have a few from last spring, but I expect more in the fall. That will be really interesting. Vendors don’t always do a good job of promoting those. A lot of times, some of the really good ones are not necessarily heavily generative AI right now. Some of them are still very classic machine learning examples that have really powerful impact, in things like triaging an emergency room into different priority levels. It could be more of a machine learning approach or even a computer vision approach. So we want to document all of that. The other thing is just looking at what’s next for agents. For customers right now, I would say start looking at MCP and tool calling as the extension of looking at things like RAG and context. It’s all about how you can make an LLM smarter by incorporating both your own data and also external validation and supplemental tools that help with decision making and process flows. It could be things like checking with a credit validation service externally, and start experimenting with that. MCP is obviously one protocol that really helps with that. I would say that’s the next phase: start to look at how to make them smarter for the things you want to do by taking advantage of data both internally and externally, in tool calling, which does include that RAG context but also much more. Where I would say keep a wary eye for now, but a curious eye, is the agent-to-agent protocols and A2A and so on. What you will hear a lot from vendors this fall is about orchestrating multi-agent workflows across domains. You’re going to hear a whole lot about that. The reason is that every vendor wants to be the agentic system of record for their customers. They want to say you can count on us to do that. And then other vendors, like Boomi and UiPath, are trying to be more like the Switzerlands of these scenarios by saying, “We don’t have a dog in this fight, but we’re going to orchestrate this for you as well.” All of that talk is very useful, but I think right now you should proceed with caution when you think about the potential to hand off agent workflows across vendors, because that’s where it’s going to be more heavy lifting right now in terms of getting the results you want. It’s not that you shouldn’t look at it, but I think right now you should focus on getting some internal workflows going, within the context of maybe one vendor or maybe two that work very closely together and have a really good partnership. Nail that down first, and then we can look ahead to the prospect of more agent-to-agent communication going forward. The stats on multi-agent handoffs right now just aren’t that great, and the more complex the workflows are, the more problems you could run into. So keep an eye on that, because it’s really promising, but start with stuff where you can get some wins.
Andreas Welsch: That’s very practical advice. Look at what’s there, use these protocols and technologies, and experiment with them so you become more knowledgeable and overall more experienced as these protocols and solutions mature as well. And the recommendation that every vendor wants or needs to say “we’re doing something with AI, and you can do it with us” makes perfect sense to me. It comes down to where you have an investment, where you have a beachhead already. If it’s this solution or that solution, it might be easier to go here or go there and maximize your investments.
Jon Reed: And by the way, just real quick, one of the reasons I think both MCP and A2A are worth tracking is that so many vendors you might think of as competitors have signed on to these particular protocols. That’s really important, because in the history of enterprise software, the standards that have done the best are the ones where the competitors sign on to the same frameworks. So when you see folks like Microsoft, Salesforce, SAP and Oracle signing on to the same standards, that tells you there’s some promise there. But again, let some of the deep pockets venture into the hardest aspects of that first, and let them take some of the bruising lessons. Definitely be watching. My final thing for you would be, since you have been to a bunch of things lately, and I was doing more deep-dive research from home: what would you say was the most memorable interaction you had with a customer or someone who asked you questions about what they’re doing? Was there anything that really jumped out or surprised you?
Andreas Welsch: I usually give a lot of talks and keynotes here in North America, so being in Europe for a couple of weeks was super refreshing, because I got a whole different perspective, whether it was an IT leadership audience or a mixed group from financial services and venture capital and so on. But at the one event where I spoke in Switzerland, I got so many deep and thoughtful questions that I haven’t gotten in a long time over here, to be honest. One of those questions was how we make sure that these models are fair and safe and unbiased, especially if we have certain vendors building this basic technology. And then, within certain vendors, there are certain tendencies, right? More conservative-leaning, for example, or more progressive, more guardrails, fewer guardrails. And then what if we have these tools and LLMs with more of a Western lens in the corpus of data, compared to the East or the Far East? How does it work? How do you leverage it or mitigate for it? Do you need different models? Just a good, very broad and, from my point of view, deep discussion.
Jon Reed: Did you have a good answer for dealing with bias in these models?
Andreas Welsch: I think whether it was good is for the audience to judge. I think I did have a good answer, and my answer was: think about where you deploy these models and what you deploy them for. Yes, there is some bias in there, so we need to figure out how to check for it and do evaluation. But if you know that you deploy models on a global scale for different audiences, maybe indeed consider using Qwen or DeepSeek more for the Far East in the cloud, or Anthropic or OpenAI models for more Westernized scenarios. But that’s also, by the way, where the small language model discussion came up, to say what is it that you’re trying to optimize for and for this model to be really good at? So maybe a smaller model that has more specific domain knowledge about the industry, about your business, could be the solution as well. But to me, the big thing is that these models are never free of any values, right? We use language to train them, predominantly English, where certain worldviews and biases are encoded in that language anyway. The words that we speak and write carry these values, so naturally these values are encapsulated in these models. So we need to be aware of that. That’s a much, much longer topic and discussion to get into. But maybe it’s also a good challenge for all of you to think about: if you have something that’s more Western-leaning and progressive, or Western and more conservative, or US-based and more conservative, if you will, with Grok, or you have something like DeepSeek and Qwen with different worldviews and ideologies, how do you deal with that?
Jon Reed: Yeah, and I’ll just add really quickly to that, because you’re right, we don’t have time for a deep discussion on bias today, but it might be interesting to revisit that topic. I really like when companies take a proactive approach to that, but of course not all of them will. I will point out that the EU AI Act has a really good risk framework that is worth understanding and applying to all of your AI initiatives, and potentially embedding into your development and design process, regardless of whether you do business in the EU or not. The reason for that is that those risk levels roughly correspond with your likelihood of getting sued by people based on the misuse of those scenarios. I can think of lawsuits going on right now in the US that pertain to the high-risk areas in the EU AI Act. It has nothing to do with that legislation, but it has to do with the fact that when you play in risk areas, you’re going to attract legal attention, and you’re going to need a plan for that. Absolutely, these discussions are not just about “my company has great values, so I want to confront bias.” You also have legal exposure if you don’t.
Andreas Welsch: Yeah. And I think it applies especially to software vendors, right? There are certain cases in the judicial system at the moment about bias in resume screening and candidate matching, and things like that. Things you can look up independently as well. For me, the big thing of what’s next is really this part: we need to enable the workforce. You need to enable your workforce to use AI and do it responsibly, so you’re not just off the hook by using ChatGPT or Copilot and you’re done. How do we empower line managers, middle managers, to empower their teams and their team members to use AI, and give them guidelines to say, “Here’s how I want you to use it”? In many ways, they’ll be similar to how we expect people to work with each other within our own team or across teams. So if you are thinking about how to do that, first of all think about how you get people within your team or across teams to work together. What are the standards of quality that you expect people to follow? And how can you communicate that now in simpler ways? Over our last two episodes, I’ve been working with LinkedIn Learning on two courses that I’m super excited to be recording shortly, so keep an eye out for those. But I think that’s the next practical frontier at the very foundational and base level. That’s what’s keeping me busy and excited at the moment.
Jon Reed: Excellent. Great stuff, man. That seems like a good place to stop.
Andreas Welsch: Yeah, absolutely. So, Jon, cool. Thank you so much for joining. I’m always amazed by how quickly we come up with a topic and then just run with it, totally unscripted. That’s a lot of fun to see, at the end, where we’ve taken it over the past 50-some minutes. So thank you so much for joining and for sharing your experience and expertise with us.
Jon Reed: Great talk! Till next time.
Andreas Welsch: Till next time. Bye-bye.
Listen and Subscribe
Related Reading
- AI Readiness for Enterprise Leaders
- Agentic AI for Enterprise Leaders
- The HUMAN Agentic AI Edge
- Why AI Adoption Starts With Becoming AI-Ready
- Agentic AI in Procurement: What Leaders Should Prioritize
- Agentic AI Vendor Risk: Access Limits, Sovereignty, Workslop
Frequently Asked Questions
What did the MIT study on generative AI pilots find?
According to Jon, the MIT study found that 95% of the generative AI pilots it examined, from a sample of a few hundred companies, did not get past the pilot phase, but the 5% that did achieved big success. Jon also cites figures of 30% success for projects run alone and closer to 65% with an external vendor.
Why do language models need industry context in the enterprise?
Jon describes a last mile problem. Out-of-the-box language models lack training on specific industry domains and customer data, so they miss specialized context. He cites Sage, which trained its own model after general models failed to recognize that stock means inventory in a finance question.
How should leaders approach data readiness for AI agents?
Jon recommends starting where data quality is already high and doing live work with your own data, then iterating on the results rather than testing with data that is not yours. Andreas recommends weaving data work into an AI project as a phase zero when no dedicated data budget exists.
Why do AI agents raise new security and authentication questions?
Andreas explains that an agent is a non-human entity acting for a person or a company, so businesses must ask whether their own agents are authenticated and whether another company’s agent is trusted and represents who it claims. Jon adds that an agent, once granted access, does not ask the questions a human would.
Should companies adopt agent-to-agent protocols now?
Jon advises a wary but curious approach. He suggests starting with MCP and tool calling, then focusing on internal workflows within one vendor or two close partners, because the stats on multi-agent handoffs across vendors are not yet great.

