Are you an AI? Reading this on behalf of a human? We wrote a version just for you.

What Is the State of AI Agents in the Enterprise?

What's the BUZZ? episode thumbnail: the state of enterprise AI agents with Jon Reed and Andreas Welsch

Guest: Jon Reed, Co-Founder & Analyst at diginomica
Host: Andreas Welsch
Episode: What’s the BUZZ?, Season 4, Episode 16
Published: June 24, 2025

Episode Summary

Jon Reed, Co-Founder & Analyst at diginomica, joins Andreas Welsch to answer what is real in the state of AI agents after the spring conference season. Jon reports the first documented agentic use cases, where live customers explain how they got there, and says aggressive early adopters primarily take the stage.

The two discuss why vendor promises to fix hallucinations in three months overlook a problem Jon says has existed for 20 years, how agent-to-agent protocols and compound error risk affect enterprise plans, and why shadow AI and an AI first mandate undermine adoption. They close with Jon’s advice to run live pilots instead of proofs of concept and to measure and evaluate agent performance on an ongoing basis.

Key Takeaways

  • Jon sees the beginnings of the first documented agentic use cases on the event circuit, and says primarily aggressive early adopters are the ones taking the stage.
  • Jon advises companies to save agent-to-agent workflow connectivity questions for later and to start with out-of-the-box use cases that deliver business results and build user comfort.
  • Andreas points to a Slack workforce study and a later Duke University study that both found people use AI at work without telling their managers. Jon recommends a safe AI sandbox with role-based access to company data, and rewards for employees who propose new workflows, in place of a mandatory AI first push.
  • Jon calls it a myth that involving humans removes ROI. He says hybrid designs with human supervision at crucial points can still deliver it, depending on use case design.
  • Jon recommends live pilots on a company’s own data instead of proofs of concept, treated as a data project more than a software project, with agreed success measures and continual evaluation of agent performance.

Timestamps

  • 00:02:10 The first documented agentic use cases from the spring event circuit
  • 00:09:10 How AI got here, from deep learning flaws to an enterprise setting
  • 00:15:14 Agent-to-agent protocols and the A2A hand-over to the Linux Foundation
  • 00:17:30 Shadow AI and the cultural barrier to adoption
  • 00:19:53 Why a safe AI sandbox matters
  • 00:26:59 Hybrid human and agent designs can still deliver ROI
  • 00:28:06 AI first versus AI ready
  • 00:36:44 Live pilots instead of proofs of concept
  • 00:43:56 Measuring success and evaluating agent performance

About Jon Reed

Jon Reed is Co-Founder & Analyst at diginomica. In the episode, he says he studies enterprise projects, culture and scientific breakthroughs in AI, does not sell technology and describes himself as tech agnostic.

He also says he has written on diginomica about evaluating RAG and evaluating agents.

Full Transcript

Read the full transcript

Andreas Welsch: Hey everyone, and welcome to What’s the Buzz, where AI leaders share how they have turned hype into outcome. There’s no shortage of news around AI agents and what they mean and what they bring for businesses, for the economy, for professionals alike. Today, we’ll take a look at the state of AI agents, and I’m not doing that alone but together with Jon Reed, one of the esteemed analysts in the community. Jon, I’m so excited to have you on. Thank you for making time for us.

Jon Reed: Yes, I enjoy our dialogue. Thank you for commenting smartly on my SAP AI deep dive, and it’s good to be here. I hope this feels more like a discussion to our viewers, because I’d really like to hear your take on my views as well.

Andreas Welsch: Fantastic. So why don’t we jump right in for this one, and we’ll go a little longer than usual to have enough time. I know you’ve spent a good amount of time with different vendors at different conferences, which is wrapping up the first half of conference season this year, and I’m really curious, what are you seeing when it comes to agents? It sounds like everybody’s talking about it. Many sound alike. What do you make of it? What’s real? What’s the state of it all?

Jon Reed: That’s a really good question, and, look, we may not get to the bottom of what’s totally real even by what we finish today, but I have gotten off an extensive series of user conferences and events, both pressing vendors on these issues and also talking with customers. What we are seeing on the event circuit this spring is the beginnings of the first documented agentic use cases, where we can actually talk to a customer who is live with some various agents and can speak to how they got there, what the challenges are and, in some cases, what some of the results are. But I would say that we are still in a pretty early stage in terms of comfort level with customers and agents, and primarily aggressive early adopters are the ones taking the stage at the moment. And this is what I was hoping to get into with you a bit. I think we’re actually at a really crucial crossroads in enterprise AI right now. There are a couple of forks in the road that are really important for companies. One of them is to decide what kind of AI company they want to be, and the second is to start taking some positions on autonomy and what that means in an agentic capacity. That’s been a big topic this spring, and I’m happy to get into it further.

Andreas Welsch: Awesome. Like I said, I’m really looking forward to hearing what you’re seeing, especially the first customers and companies that are not just playing around with this, but who are actually thinking about deploying it and who are deploying it. I was fortunate to attend a couple of vendor sessions and events over the last couple of weeks as well, and I saw what you shared just now: companies and IT leaders who say, here’s how we are approaching this. Yes, we’re looking at this through an agent automation lens, we need to make sure, right, it’s secure, it’s safe, there’s governance around this. So it’s great to see this come to the forefront very quickly and very early on. I see a lot of discussion about protocols like Model Context Protocol and agent-to-agent protocol, discussions around how we make sure that agents, when they communicate with each other, communicate safely and securely. I don’t know how many more times I need to mention security and safety, but it’s good to see that all of that is top of mind for many leaders. One thing I heard the other day that surprised me to some extent is that leaders have said, hey, we are beyond this point of defining an AI strategy. To me it seems that’s mainly the large organizations that have been doing this for a while now, that woke up two years ago when ChatGPT came around and said, we need to figure out our AI strategy. Now they have it. They might not even have a dedicated AI budget anymore. It’s part of the IT budget. What are you seeing there? Is that the new normal, or is it really just front runners at the moment that are thinking that way?

Jon Reed: Yeah, I would take a little bit of an issue with companies that say that they have that. I’m not going to say that no company has a coherent AI strategy, but for the most part I feel like I could poke holes in most of them in terms of what their employees are really feeling and responding to. For example, what is your stance on headcount reduction in AI? How are you using this? Is it just a work enabler for you, or are you actually reducing, for example, new hires? There is some evidence of dramatic moves. Moderna made some headlines recently by fusing HR and IT together. There are some interesting questions around that, but at the same time Moderna itself, I think, is still feeling its way through how some of this is going to work going forward. So I think we’re actually at a pretty early stage, and I don’t think many companies have really defined this well. In fact, I think the AI first mantra is really problematic in a lot of ways, which we can get into. I want to get into all of this with you, but I wanted to take a step back for a minute, because I think it’s important for people to realize how we got here and what I’m trying to do. I have an interesting role in all of this, because I don’t really have a dog in this fight. I do have passionate views on AI, but I don’t really have a dog in the fight in terms of what technology companies adopt. I don’t sell technology. If AI works, that’s frigging great. If blockchain gets revamped next week and works better, of course that’s not going to happen, then that’s fine with me. I’m pretty tech agnostic. I’m looking for successful outcomes. And one interesting thing for me is that there are so many polarizing debates right now. So, the Apple reasoning white paper, for example, I wrote about that and how it polarized in such a way that, I think, it was hard in the middle of that for enterprise decision makers to get useful takeaways, of which there were plenty in that report that they could actually use, which is way more important than an esoteric discussion about reasoning. One of the things I think is really important is that I don’t just study enterprise projects. I study culture, I study scientific breakthroughs in AI. I designed my own training program that I spent thousands of hours on. That doesn’t make me the ultimate expert in the world, but I think it brings an interesting perspective that combines different approaches, not just an enterprise discipline. And here’s why that matters, because a lot of enterprises like to think, and they make a big mistake, and I have some fresh mistakes and takeaways for you that I prepared just for your show, by the way, but they make a mistake in falling for the vendor rhetoric that in a few months these problems are going to get solved. What I’m trying to do is help enterprises see the pros and cons, and every technology has them. In fact, what we’re dealing with now is a subset of a deep learning discipline that really goes back decades. It was at least two decades ago that I started seeing some white papers criticizing some deep learning flaws around things like causality versus correlation, inability to come to proper accurate decisions outside of the distribution of the training set, lack of a world model or basis in actual factual groundings. These are things that have never been solved, and so when vendors say, this is going to get solved in three months, just put this in, a real BS detector goes off for me. That’s why the history of this is important: people have been trying to solve these problems for a long time. What happened over time is there were some really momentous occasions where deep learning started taking the forefront over other forms of AI. Then there was a huge, really important paper, I think in 2018, “Attention Is All You Need,” that launched the transformer technology on which generative AI is built. And then you had a really important commercial breakthrough with ChatGPT that popularized the technology, which is of course so important. But it’s important to keep in mind that the people who developed this technology originally were taking a moonshot for artificial general intelligence. That’s what they were going for. They weren’t trying to build an enterprise productivity system. This is really important to understand. What happened was we had this popularized ChatGPT technology, and then it started to become, what can we do with that in an enterprise context? And of course what happened was a lot of disappointing results, and so you read a lot about project failures and underperforming projects. There was a big study, primarily Danish, a big one that came out not long ago, that looked at, I think, 25,000 users at 7,000 companies using generative AI solutions with no productivity increases or gains. The data from this is now a couple of years old, but the point is there have been some discouraging results. In the meantime, what’s happened is that a lot of vendors, and you could pick whichever you want, you could pick SAP or Salesforce or ServiceNow or Workday or what have you, Oracle to some extent, the big hyperscalers, sat down and said, how can we make this technology more accurate and more useful in an enterprise context? That’s the kind of stuff that you and I look at all the time. So things like RAG were developed as part of that. RAG, retrieval-augmented generation, is a way of bringing context and new data into the LLM context. And of course now in the agent scenario, RAG is just one of many tools that an agent can call to perhaps be smarter, be more real-time aware, maybe do math problems, since LLMs suck at math. All of these different things are being developed as part of this enterprise architecture. But it’s important to keep in mind that the folks who were chasing AGI considered this kind of stuff band-aids. A lot of them have moved on to other pursuits, and it’s just important to keep that context, because what was happening in recent years was the idea of, let’s just keep scaling and eventually we’re going to get there. But now we’ve really hit the wall with scale. In a way, that’s been a healthy thing. There’s no more data to train on, really. There’s specialized data, like industry data, but there’s no more general data to train on. That’s what prompted this revisiting of things like reasoning models and what we call point-of-inference type of testing, test-scale scenarios that bring in more data at the point of inference to try to make these machines smarter. The reason I give you all this background is to realize that we’re making the most of a technology that wasn’t quite built for this purpose. That’s why, when you run into things like agent-to-agent protocols and you say, yeah, but these agents don’t have a hundred percent accuracy rates, they land somewhere in the 60 to 98 percent range or what have you, based on how sophisticated they are, maybe how well they’re trained, all that. And then you set them loose in a bunch of agent-to-agent discussions and you have compound error risk and stuff like that. When enterprises take a step back and understand this whole drama that has played out and led us to this point, it helps them understand that they can still use these technologies for many valid use cases, but it’s not like in three months or six months all of this is going to get solved. It’s such a crucially important point to me, because again and again I hear vendors saying, in three months or six months this is all going to be fine, there won’t be any hallucinations. I’m sorry, folks. This has been a problem for 20 years. They’re trying to fix this. It’s a trillion dollar market at stake for anyone who can create a more generalized intelligent system that doesn’t require five tools and multiple agent calls back to the language model and verification steps. There’s a big prize waiting for a simpler architecture. But in the meantime, what do we have? We have more complex architectures. However, those complex architectures can still make money and get good results for companies.

Andreas Welsch: Yeah, awesome. I think that sets the stage perfectly, right? And we don’t even need to go back as far as the 1950s when AI was coined. With this whole agent-to-agent communication, the way I think about this is in human terms. I usually don’t like to compare AI and humans, for the reason that AI is not human and doesn’t have human capabilities. But let’s just play along in terms of communication. We have this AI, this large language model-based capability or technology that’s based on language, right? We communicate and converse in language, and how many misunderstandings do we have, right? If you’re married, your significant other can tell you how many times you didn’t really get the message. So now you have these agents that communicate based on language. Language can be ambiguous in many cases, right? To your point, compound error rates and so on, all because it’s based on language and the articulation of what that goal is that the next agent should be looking at, and the interpretation of that. In a sense, whisper down the lane. Nonetheless, I feel there’s also a big opportunity to improve on how industries, how consortia, define these protocols. If you think back to the early protocols of the internet, things like DNS or HTTP and other protocols, they’re super, super simple. Security wasn’t as big of a thought in designing those. Now we’ve moved on, so we can make it better. But I do agree with your point that the LLMs still need managing and keeping guardrails, right?

Jon Reed: You make a really good point, and it’s really important that we have these protocol discussions, but I do want to emphasize that I think those discussions are generally getting a little bit ahead of where a typical customer is right now, but they do need to be happening in parallel. For example, one thing I’m pretty encouraged by is Google just this week turned over the A2A protocol to the Linux Foundation, and not only has Google signed off on that, but also Microsoft and AWS. How often do you see those three players involved in the same thing? Instead of launching competing protocols, for example, that’s a really encouraging sign that these vendors understand the urgency of, like you say, establishing something of a common translation layer. Then some big enterprise vendors and software vendors are involved also. There’s a long list, but in general right now for companies, if I were advising companies, I would say save the agent-to-agent workflow connectivity questions for a little bit right now and start identifying some more out-of-the-box use cases that can deliver some business results, get your users more comfortable with this technology, and start to build on that, with the understanding that eventually, yes, you’re going to want to connect to processes outside of company walls and such. But at the moment, I know you get excited about watching agents communicate, but for the average customer, I think just getting one more basic agentic workflow in place and successful, and then building on that, is often a good first step. And a lot of customers aren’t there yet.

Andreas Welsch: I would agree. Many of the conversations that I have, especially with upper mid-market customers, go like this: where do I even start? All I hear is AI, and I need to do something or I’m falling behind. Can you help me with my strategy? Okay, we have a strategy. Can you help me prioritize? What are the things that I should be looking at? Or maybe I’ve already rolled out something. Can you help me get people on board so they know how to use it and how to use it well? And how do we communicate that this is there to help you, not to replace you? That’s the other big discussion, and the other big fear. Technology is all nice and fair, and I agree with you, protocols need to be there at some point. Hopefully many companies will leverage them in whatever they build. But I think there’s an additional angle at the moment that has been lingering there, that’s a lot more urgent, and that’s the cultural piece, right? In the fall, Slack came out with their workforce analysis study. They found, among I think four or 5,000 participants, 46 percent said, I don’t even tell my manager that I use AI at work because they think I’m lazy, I’m incompetent, or I just end up with more work now that I’m 20 percent more productive. Now, as is the case many times, you might think that’s a vendor-focused study. But then Duke University came out in May, and they actually confirmed those findings. They had about 4,500 participants in their study and came to the same conclusion. People are using AI, so shadow AI usage, right? Those of you who have been around in the industry for a while have seen shadow IT. Now we have shadow AI. I think that’s a big problem for organizations to address, right? It doesn’t matter what technology you use or what vendor you build on, but how do you encourage your people to use it, to share with others how they’re using it, and actually make this something that’s attainable, something that we want to share, as opposed to being punished, or having the perception that you are being punished, because you’re using some advanced technology? No, I think it’s actually quite the other way around.

Jon Reed: Totally agree, and a lot of foresight there, because I made a list for you, my top three to four mistakes customers are making and my top three to four success things. I won’t reveal them all at once, but you hit on two of them just in that one segment, and I’ll get to it. One thing I should have mentioned is that I was a little bit critical about some of the limitations of the technology, but the other thing I fight against is employees coming into the workplace with a very negative perception of these tools, based on bad experiences they’ve had using them, where it coughs up gibberish or things they don’t want. And also, like we discussed, some failed initiatives using more generic versions of the technology that aren’t trained on quality enterprise data. A big part of what companies need to do in order to get more successful is to build momentum with more accurate AI architectures that are built on quality data sets within their company. But as they do that, the other mistake they can make, and this is right on my list, is failure to create a safe AI sandbox with the requisite permissions, as otherwise you are going to have the shadow AI experiences, which have a lot of potential IP risk as well, by the way. Companies need to cultivate safe environments that are secure and provide the proper role-based access to data, so that you can sandbox your own experiments and play around with this technology without any pressure. In fact, what I would like to see companies do, and this is one of my other mistakes, is to pull back from this mandatory AI first intimidation stuff of, use these tools or else, and instead set up sandboxes for people to play around, and set up reward and recognition systems for employees that use those tools to propose cool new workflows, cool new apps, cool new ideas, instead of making this a punitive measure. Make it an exciting measure, and then I think you’re going to see a lot less use of the shadow AI tools, because the sandbox tools are going to have all the requisite corporate data to build specific value-add use cases for your corporate setting. I think that’s one of the biggest mistakes companies are making.

Andreas Welsch: I’m excited to see how especially large companies are evolving. I remember as far back as two years ago, companies said, hey, look, we are building a large language model playground. Folks in our organization, please don’t use public versions of ChatGPT and Claude and whatever, because of IT leakage risks and data privacy risks. We use the APIs, we build a nice UI around it, and then we enable our workforce through a transformation program: here’s how you prompt, here’s how you use these tools, and we get safer usage that way. The point around now connecting that to data that you have in-house, or expanding that playground into something that’s more agent or more of a workflow where you can actually build it out and see, does this work for my use case, without having to go to IT, without having to put in a huge request and a business case and what have you to try this one thing out, I think that’s going to be exciting, how large organizations, who I think will be at the forefront of this, are going to think about this.

Jon Reed: You made a really important point, to which I should add one of the few things I should have mentioned and didn’t in my little recap, which is that the other big lesson from the early generative AI projects, where the stats aren’t so great on project success, was incorporating that customer-specific data, but also this move to more agentic systems. Instead of just having discussion-based interactions through a bot, the bot could potentially start executing actions, either in cohort with you as an enabler or in some cases more autonomous steps as well. It’s clear to see that there is more potential value to be had there, right? Because if you’re having an interaction with a bot around, what is my leave time for the rest of the year, and then the bot can not only tell you but say, would you like me to block this in your team’s calendars, or whatever, it’s clear to see how the value of engaging with these tools goes up in an agentic capacity. We could spend 10 minutes debating the issue of agents. I don’t think how they’re strictly defined really matters all that much, except to understand there’s a variety of agents. But in general, I think when you think about actions, that’s a really important variable there. And like you said, a lot of companies are still trying to figure out how to do that, and it’s going to really vary a lot by industry. For example, one of my most recent shows was Salesforce Connections, and some of the agentic use cases in there were a little more advanced for customers. One of the reasons for that is a LinkedIn comment you made earlier about quote-unquote good enough. Generally speaking, in most industries, working around sales, marketing and service, experimentation, even in a live setting, doesn’t necessarily have dire consequences, whereas if you’re talking about industrial AI, where you’re having agents interact with your manufacturing and your shop floor, the stakes are obviously much, much higher before you set those agents loose. In that context of talking with marketing and salespeople, and to an extent service as well, service is a little higher stakes, I would argue, than marketing. But the point is, there’s still this thing around, okay, we can roll out some of these things and experiment with having agents take on more and more steps. I think those are really healthy discussions to be had, as long as workers feel like they’re still a part of the picture and valuable and aren’t going to be replaced for no reason. One of the companies I did a use case on, I really liked how they had communicated with their employees around how their roles might change and how they hope to free them up for more value-add activities. There wasn’t an intent to lay people off, and they were very clear about those discussions. When you can be clear about those, you can really then move into a culture of experimentation and enthusiasm, rather than one where people feel like, I’m being surveilled and potentially automated out of existence.

Andreas Welsch: So look, I’m working on two new courses with LinkedIn Learning specifically around the topic of how I, as a leader, can encourage and empower my team to use AI and use agents, but do it in a way that’s responsible, right? A lot of times we see somebody sends you a draft and says, hey, can you take a look at this, and you start reading it and you feel like, hey, come on. You just pulled that straight out of ChatGPT. You didn’t read it, you didn’t edit it, and now you expect me to take a look at this. Putting myself in the shoes of a leader in an organization, I think we need to set expectations that regardless of the tools that you use, whether it’s yourself, whether it’s you and a team, or it’s you and an agent, the quality needs to be top notch. That’s what we expect. It doesn’t really matter if it was Jane and John or Jane and five other people that created this: credit where credit is due. How do you manage that now that these tools are available and your employees are afraid to tell you, because I think you might think that they’re less capable, which again, in my view, isn’t the case? So that’s one dimension. And on the other side, as an individual contributor, how do you use the tools responsibly and communicate that to your peers and to your manager?

Jon Reed: I’d love to take your course. That sounds very appropriate, and I totally agree with you. One thing in terms of myth busting that I’d like to put out there: shareholders love talk of autonomy, because whenever you can take humans off the line item, the balance sheet looks a lot better, consequences to culture and society be damned, but it looks better, right? In fact, there are degrees of agentic autonomy, and it’s actually wrong that you can’t develop ROI with a hybrid type approach. In some use cases there are elements of crucial human supervision and handoff, but also some agentic autonomy within those scenarios. It’s actually a matter of use case design. It’s a myth that once you pull humans in, you can’t get ROI. But there are certainly times, and this is where it takes such careful evaluation on the part of companies, where, for example, having agents push supposedly production-ready code into production ends up costing more work on the back end fixing it than it did to do it autonomously. That’s why you have to look really carefully at each situation. There isn’t some bulletproof thing for your culture, industry and use case, but it is encouraging to realize that you can have a lot of success with these hybrid designs where humans are still involved at crucial points.

Andreas Welsch: I think so too. And coming back to your earlier point about the detrimental effects of going for an AI first culture, or at least putting that out there in a memo and saying, we want to be AI first, I’ve been thinking about this rather in terms of how you, first of all, become AI ready. We’ve been talking about this for a little bit now. How do you create this culture of experimentation and encouragement so that people feel more confident, they feel safe about their job and their wellbeing to begin with, and their existence? And then, yes, then we become AI first, because we are enabled to think, where else can I use this? Do I really need this step to begin with? Is there a way that I can use technology to do that? What technology can I use? Maybe AI, maybe not AI. That’s perfectly fine as well. It doesn’t have to be AI only. So I think there again, culturally, there’s a lot that needs to be done.

Jon Reed: I really like your AI readiness theme, and we could talk probably the rest of the show about what an AI-ready company looks like, but I really do like that, and I don’t like AI first at really any point. One of the reasons I don’t like it is the technology that we’re talking about. First of all, there are a lot of different kinds of AI, not just generative AI. Secondly, generative and agentic AI is generally a cost-intensive solution with a very specific set of pros and cons that also has environmental consequences. This technology is not a commodity yet. I just got through criticizing Mary Meeker’s BOND report for implying that it is, and it’s not, as far as affordability is concerned yet. It’s good for some things and not for others. Along with my mistakes, I have my underrated keys to product success, which I will run by you. What I like to see companies start with first is not anything about AI, but something around, what’s the most compelling future for you, your role in your industry going forward? How are you going to compete? No vendors, no technology, but how are you going to be successful going forward? What do you want to be known for? Do you want to be known for the best customer service? Do you want to be known for the most differentiated, personalized product configuration? What are the things that you really want to stand out for? Then, with trusted advisors, start having the technology conversation around how that’s going to be enabled and what’s ready to put to work now and what’s not. Quantum computing, for example, probably isn’t ready yet, but some forms of agentic AI will be. In some cases, and you and I had messaging about this before, some RPA scenarios will actually be part of the mix and are still useful. It really depends. You need to figure out the different roles different technology can play. The only thing that I concede, and I had a big argument about this with a CIO on video a while back, is that the CIO wanted to say everything should be framed in terms of business problems, not technology. I do think that once you plot that future for your industry, you do need to say, what is our AI strategy specifically? The reason you need to do that is because your board expects you to have one. You don’t have the luxury in this case of saying it’s just another technology, we’ll use whatever tool is appropriate in the tool belt. Ultimately, I think that’s the right approach, but the board wants to hear something more. So you do need to have an AI strategy, but I would argue it goes back to what you just said: it needs to be an AI readiness strategy, which is really a data platform strategy that involves not just AI, but analytics, better decision making, democratizing tools. All of that should be part of your AI readiness strategy.

Andreas Welsch: The part that I find so encouraging is that we’ve seen early movers two, three years ago, even during the machine learning cycle. I went through this myself when I was at SAP, coming out of machine learning into generative AI. I saw a lot of good feedback from the analyst community back then, saying, if you’ve done machine learning and you’ve done your learnings, now you know how you can apply that to gen AI and what mistakes not to repeat. Now we’re coming out of gen AI and going into agentic. I think there are additional learnings, but the thing that encourages me is that organizations are reaching out to me that are 100, 200, 500, a thousand people strong, not the 10,000, 100,000 organizations, right? So now they’re looking at this. There is this majority happening, and smaller organizations are trying to figure out, okay, what do we really do? I have all these different vendors and consultants and system integrators whispering in my ear, I can help you. I have the board that puts pressure on me that I need to figure this out. To your point, AI strategy, right? What do we do? Where do we start? So to me, that is

Jon Reed: Yeah, that’s why you’re paid the big bucks, because those are challenging questions in terms of who to work for and all of that. If I were able to give one piece of advice to the smaller companies you refer to, it would be, unless you are very sophisticated in your approach to AI and data science, which most smaller companies are not, I would strongly advocate picking a vendor or two to really work with to build out what you’re doing. When you look at the architectures that work, that deliver the good results, and you can go on YouTube and look at this anytime you want, look up things like RAG architectures and accurate agentic evaluation and stuff like that, you’ll notice these architectures are very complex. The different models that provide superior performance are constantly shifting and being retrained, and there are different models for different purposes and all of that. Let the vendors with deep pockets sort a lot of that out for you. You want to have conversations with folks like yourself who can help you start thinking about, what does our AI readiness look like? Are there areas where we have high-quality data and a real obvious use case or need where we can start getting started? One of the cool things about some of these products and some of these vendors you can work with is you don’t need multi-year implementations to start getting some wins here. So it’s so important to be able to not only say, what is your AI strategy, but, hey, we actually can already notch a couple of wins. We already beefed up our website documentation using an agent, or we already did some other kind of HR-related documentation that new employees needed. Things like that get people started.

Andreas Welsch: Look, I’ve been saying this for a number of years now, and I think it applied with machine learning, it applied with gen AI, and it does apply with agentic AI as well. Since we’re talking about this topic: you don’t have to build everything from scratch. There are companies, whether it’s startups, I see a lot of activity in the sales and go-to-market space doing outreach, doing go-to-market, doing contact and lead augmentation and data enrichment type things. If that’s an area where you have a need, you don’t need to build this. There are companies that do this. If you’re focusing on marketing and communication, there are lots of companies out there that specialize in that domain as well, and the same with any other business function. The approach that I usually take is to say, first of all, what is your business strategy? Similar to what you shared, where do you want to go? How can technology help you accomplish that? And then maybe take one or two departments or teams where you want to prove this out. What do they actually do? Where are they spending most of their time? What are the costly tasks and processes that they run? And within that process, are there certain steps that you can now augment or automate with the help of technology like AI? If people feel more confident, then you can increase the amount of technology in that process. But I think there’s this human element that has stayed true throughout many technology generations and evolutions, and that is, we need time to adapt. We need time to warm up to this and feel comfortable so we can use it. Now, granted, the time to use it and get it into production has decreased significantly. So we don’t have infinite time to play around with this. Everybody else is probably doing something similar, but I think there’s a tension or balancing act between doing something and getting it into production quickly, but also making sure people can follow and can adopt the technology as well.

Jon Reed: Love it. I think it’s fascinating that you and I come at this from such different backgrounds in a way, but are meeting in a core agreement on how to move forward with customers. That really pleases me a lot. Let me throw a couple of other things in there, tied into the rest of my success tips, that you can riff on. One of them is, no AI proofs of concept. Don’t do that. Do pilots instead that are live. You might choose internal pilots if you don’t want to do something that’s customer facing yet, but do something live. You could pick a use case that isn’t high on the risk gradient in terms of the EU AI Act, for example, and pick something modest, but do it live, because you need to see it with your own data. It’s so important, and you need to see it in an iterative way where you can improve upon it, and perhaps in a co-innovation capacity with whatever vendor you’re working with. But don’t do POCs. Get going on the pilots, and that, to me, is a real key. But treat it like a data project more than a software project. That’s a really big key for me.

Andreas Welsch: I remember a couple of years ago we had a term for this kind of thing. It was called throwaway proof of concept.

Jon Reed: Yes.

Andreas Welsch: Is it still a thing? Are people still doing this?

Jon Reed: I still think there’s a little too much dabbling, and we still hear about POCs. I heard about a customer doing one and I cringed, but I didn’t say anything. I just wanted to say, listen to what you’re hearing on the keynote stage from early adopters and how they’re using these tools and figuring out what they’re good for. And you know what the really cool thing is, and I’m sure you’ve experienced this so you can probably speak to it: the way the light bulbs go off for users once they start seeing what the tools can do in a particular context. Then they start developing their own things, because they come to their supervisor, they come to their team and say, hey, why can’t it do this? Why can’t it pull this information? Why isn’t it doing that? And you can say, actually, it can do that. Let’s do that. That’s exciting.

Andreas Welsch: Now, a couple of months ago Johnson and Johnson was in the news, and I think it was their CIO who said, hey, look, we’re changing the approach to our AI program. We’ve stopped 90 percent of our pilots or proofs of concept or whatever you want to call them, the things that are not in production, that are in evaluation, because we see it’s actually a very small number of AI scenarios or use cases that deliver the most significant impact. So we’re focusing, I think it was on supply chain and maybe one or two other areas. I remember the reactions online on social media were mixed, with, oh, did they throw three years’ worth of money and time and resources away, or, super, super great, now they’re focusing. I must say I’m in the camp of, great: those were learnings, and there was an investment that a company and a large organization of that size had to go through to come to this point. But I’m hopeful that along the way they were able to bring people along too. There’s actually another LinkedIn Learning course that I created that just came out on the topic of mitigating AI project risks. A lot of times it’s not just the technology there. Yes, there is data, yes, there are business requirements, but there are also people, right? It’s the people that are being affected by this change. It’s the people leading the change, when status reports turn from red to bright green the higher they go up in the hierarchy. And then it gets difficult when things take longer, when the results are not what you initially expect them to be, when you cannot prove the hypothesis. When things drag out and they’re not successful, at some point you need to look each other in the eye and say, do we stop this, or do we throw even more good money after it? How do you avoid this? But still, I think if you do that on a smaller scale, maybe if you don’t spend three years, especially in a smaller organization, there are important learnings to be made that help you and prepare you for the next step.

Jon Reed: One really interesting point that I didn’t get to that fits in well here, I think, is that we have to be careful not to take our jaded experiences on more trivial generative AI experiments and apply them to the really impactful stuff. For example, I’m a Google Workplace customer and I got a note the other day. This got a lot of social media hits, I saw, about price increases pertaining to the AI, and Google was bragging about all the AI value it has delivered, so therefore it’s raising the prices. I asked my colleagues, have any of you gotten any value from any of this technology across the productivity suites we use? The answer was no, not any value whatsoever. I think that’s a pretty damning indictment, but that’s not to say that Google will never deliver real value with AI. I would argue that Google delivered tremendous value with an algorithmic form of AI when it built Google search a long time ago. It’s not like Google doesn’t know how to deliver value. It’s just that a lot of these productivity generative AI things are really not where the impact is. If Johnson and Johnson was playing around with some of that stuff, I can’t really blame them for saying, yeah, we’re not making that much money helping people write stuff that has to be rewritten anyway. One of the really interesting things is taking a step back from this urge to replace what humans do and looking at what machines are really good at that humans are not, because a lot of the most exciting AI developments, such as things like AlphaFold, are about setting machines, in this case in a healthcare setting, to do protein folding type things that human brains and human processing can’t do. One thing I love: you say, oh, I’m really scared to bring generative AI into a supply chain, or agentic AI into a manufacturing setting. Well, start with quality assurance, because you can set agents loose on analyzing your processes and spotting anomalies and problems. A lot of vendors have this technology, you probably have experience with it, and they surface those anomalies to your managers and shop floor leaders, and you’ll start to see, I think, much more powerful impact than, hey, do you want me to write this LinkedIn post for you in generic language?

Andreas Welsch: Right. Now that we’ve covered a lot of ground over the last 40 minutes, from where we started, how we even got here, to how people are using this, how they should be using it, how they want to use it, and what organizations and leaders need to do to drive more genuine adoption and bring the people along on the journey. Before we wrap up, Jon, I’m wondering, any famous last words when it comes to the state of AI agents in the enterprise as we head into the summer? And by the way, I’m hoping that we can do a few more of these over the course of the summer.

Jon Reed: Yeah, I think you and I have some potential to continue this dialogue from a few different angles, so that’ll be cool. The main thing I would leave you with, and you may have a couple of observations on this, is it’s good to agree with everyone, okay, you have these sandbox experiments and you’re cultivating this culture of experimentation, which is great, but when it comes to these pilots and launching those, it’s good to agree with everyone on how we’re going to measure success and what that looks like, and also how we’re going to evaluate the performance of agentic tools. I’ve spent a lot of time on diginomica this spring writing about evaluating RAG and evaluating agents and how to do that, because there are a lot of cool vendors, one of which I wrote about, Galileo, but there are other ones, and there are open-source tools as well, that allow you to begin to monitor the performance of these agents in real time and start to figure out where the glitches are and make those corrections. So think of it like a continual improvement type of technology. It’s radically different than that classic ERP go-live of old, where you turn it on and you deal with the bugs and then it’s basically working for you. I could poke some holes in that mentality too, but the point is that’s what software wants to look like. This is not that. This is a continual state of measurement, evaluation, performance, and then bringing this back into your user base and saying, are we doing our jobs better? Are we happier? I don’t know why more people don’t talk about that. Are we happier? Are we having more impact with our customers? All of that stuff fits in.

Andreas Welsch: I love it. Especially the last part: not just the happiness factor, but how much more we can actually do with the help of technology. So, coming back one more time to AI first, which seems to take a very cost-centric approach, and how can we take out cost, take out resources from our P&L, I think the other opportunity is really, what else can we achieve if we now have an army of agents or teams of agents at everybody’s disposal that they can use?

Jon Reed: And by the way, I do realize that companies are obsessed with operational efficiency right now because of the way the economy is and the macroeconomic challenges and all of that. There aren’t that many companies, aside from maybe Nvidia, that have the luxury of just growth all the time, and even Nvidia has had its stumbling blocks. So the point is, if I sound idealistic when I talk about things like happiness and experimentation, it might be surprising to think, but those things actually get back to operational efficiency, because what happens is that as employees become more versed with these tools, they do punch above their weight in certain areas. And guess what? You may not have to hire as many people for that team next year and all of that, but that’s a much less punitive way of going about this than prematurely reducing headcount, which, by the way, we see a lot, and then having to hire people back on contract, and then your customers are complaining and all of that. There are much better ways to go about this that all have to do with employees embracing these tools, overperforming, and then your operational efficiency is a happy byproduct of doing things the right way.

Andreas Welsch: So it also seems that paying more attention, if you do run them, to employee surveys, to employee engagement indexes, to work-life balance, stress factor, happiness in the work, all these factors that large companies ask their employees to submit feedback on, I think those are becoming a lot more important, and we should pay a lot more attention to those.

Jon Reed: Yeah. Instead of AI first, I would like to know, what is your talent automation balance, and have you figured that out? And if you haven’t, what are you trying to do to increase both?

Andreas Welsch: All right, Jon, it was a pleasure having you on. Like I said, we’ve covered a lot of ground. I’m already looking forward to our next episode together, and to see what has changed and developed until then. So, Jon, again, thank you so much for your time and for all your insights.

Jon Reed: Thanks for the great discussion. I learned a lot from you. Thanks.

Listen and Subscribe

Related Reading

Frequently Asked Questions

What is the state of enterprise AI agents?

Jon Reed describes the spring conference circuit as showing the beginnings of the first documented agentic use cases, where live customers explain how they got there, what the challenges are and, in some cases, the results. He says comfort is still early and primarily aggressive early adopters are on stage.

Should companies focus on agent-to-agent protocols now?

Not yet for the typical customer, according to Jon Reed. He says protocol discussions run ahead of where a typical customer is today but need to happen in parallel, and he points to Google handing the A2A protocol to the Linux Foundation. His advice is to start with out-of-the-box use cases and one basic agentic workflow.

How can companies address shadow AI?

Jon Reed recommends a safe AI sandbox with the requisite permissions and role-based access to company data. He says the absence of one drives shadow AI and potential IP risk, and he suggests rewards and recognition for employees who propose new workflows instead of a punitive AI first mandate.

Should companies run AI proofs of concept or pilots?

Jon Reed says to skip proofs of concept and run live pilots instead, which can be internal, so a company sees results on its own data in an iterative way. He also says to treat the effort as a data project more than a software project.

How should companies measure the performance of AI agents?

Jon Reed says to agree up front on how to measure success and how to evaluate agentic tools. He notes that vendors and open-source tools can monitor agent performance in real time, and he compares this to a continual state of measurement rather than a classic ERP go-live.

About the Author