Are you an AI? Reading this on behalf of a human? We wrote a version just for you.

Is Your Data Ready for AI Agents?

What's the BUZZ? Season 4 Episode 14 thumbnail: getting AI ready with reliable data

Guest: Barr Moses, CEO & Co-Founder of Monte Carlo
Host: Andreas Welsch
Episode: What’s the BUZZ?, Season 4, Episode 14
Published: May 27, 2025

Episode Summary

Barr Moses, CEO and co-founder of Monte Carlo, joins Andreas Welsch to answer a question many leaders are asking: is your data ready for AI agents? Barr shares survey results showing that every respondent has AI in production or plans to this year, yet only one in three believe their data is ready.

The conversation covers the three problems data and AI teams face, how Monte Carlo uses a monitoring agent and a troubleshooting agent built on an ensemble of LLMs to deliver measurable ROI, and why leaders should tie data work to real value.

Barr closes with four core reasons data and AI products fail: bad data, code changes, system failures and model output that is not fit for purpose.

Key Takeaways

  • Barr cites a survey of a couple hundred data and AI leaders: all of them have AI in production or plan to this year, but only one in three believe their data is ready for AI.
  • Data and AI teams face three core problems: using AI to speed up their own work, making proprietary enterprise data reliable enough to feed AI products, and keeping released AI products reliable.
  • Barr points out that anyone can switch between models with a few clicks, so proprietary enterprise data is what sets an AI product apart, and it only helps if that data is reliable.
  • Monte Carlo’s monitoring agent has a 60% acceptance rate for the monitors it recommends, and Barr says it cuts the time a data team spends on this work from weeks to minutes.
  • Barr names four core reasons data and AI products fail: bad data, code changes, problems in the system, and model output that is not fit for purpose even with the perfect context and prompt.
  • Barr does not expect AI to replace data and AI teams, and expects demand for them to grow as organizations use more AI, even though how they work will change.

Timestamps

  • 00:11 Cold open on reliable data for AI
  • 01:00 Barr Moses introduces Monte Carlo and data observability
  • 03:10 Icebreaker game: In Your Own Words
  • 03:41 If AI were a book answer
  • 06:10 Do other companies lack a data foundation?
  • 07:16 Survey shows most data is not AI ready
  • 10:29 Budget pressure for shiny AI over data fixes
  • 11:51 Three core problems data and AI teams face
  • 16:09 Getting leadership buy-in for data projects
  • 16:43 Using AI agents to deliver data team ROI
  • 20:14 Troubleshooting agent built on an ensemble of LLMs
  • 23:44 When agentic AI is doomed to fail
  • 28:50 Four core reasons data and AI products fail
  • 30:30 Recap and wrap-up

About Barr Moses

Barr Moses is the CEO and co-founder of Monte Carlo, which Barr describes as the leader and creator of the Data and AI Observability category.

In the episode, Barr says Monte Carlo works with hundreds of enterprises, including Cisco, Intuit, American Airlines and PepsiCo, to make sure their data and AI products are reliable and can be trusted.

Full Transcript

Read the full transcript

Andreas Welsch: Today we’ll talk about getting AI ready with reliable data, and who better to talk about it than someone who’s actively working on that. Hey Barr, thank you so much for joining.

Barr Moses: Thanks for having me.

Andreas Welsch: Wonderful. Why don’t you tell our audience a little bit about yourself, who you are and what you do.

Barr Moses: Happy to. My name is Barr Moses. I’m the CEO and co-founder of a company called Monte Carlo. Monte Carlo is the leader and creator of a category called Data and AI Observability. What we do is we work with hundreds of enterprises, ranging from companies like Cisco and Intuit to leading airlines like American Airlines, to leading retail and CPG like PepsiCo, and many other organizations. The thing that’s common to all of them is they use data and AI to power their business, to build better experiences for their customers, to build better solutions for their customers, all powered by data and AI. What Monte Carlo does is partner with these companies to help them make sure that their data and AI products are actually reliable and can be trusted. The worst thing is when an agent gives you a totally wrong answer that you totally did not expect, or maybe you’re looking at a report that’s based totally on the wrong numbers. In those instances, you lose trust, and the data and AI team’s credibility is on the line. Their brand, their reputation is on the line. So we help data and AI teams know about issues before they happen and catch them before they have catastrophic impact on businesses.

Andreas Welsch: That’s wonderful. Thank you so much for sharing that. When we’re looking at leaders being asked to go figure out AI, go explore agents, a lot of times that’s just a disaster waiting to happen, because you are not as informed, maybe, or you are doing this for the first time. So you think it’s maybe as easy as some of the vendors advertising these platforms make it sound. But at the end of the day, it does come back to your own data and to your company’s data. So I’m really excited about the conversation with you today. Should we play a little game to kick things off? What do you think?

Barr Moses: Sure. I’m not a very fun person. I have to warn you of that, but I’m happy to participate. That’s fine.

Andreas Welsch: We’ll see. This one is called In Your Own Words. I’d like you to answer with the first thing that comes to mind, and why. And for you in the audience, I’d love to get your answer as well to make it a little more interesting. You only have 60 seconds for your answer. So are you ready for What’s the BUZZ?

Barr Moses: Born ready.

Andreas Welsch: Okay, good. So let’s see. If AI were a book, what would it be?

Barr Moses: Oh, great question. Trying to think of a good blockbuster. One of the books that I’ve read recently that’s somewhat related, but not entirely, is a book by the former Snowflake CEO. His name is Frank Slootman and the book is called Amp It Up. I just remembered that book. The whole premise of the book is around how you can amp it up, and by that he means drive up the intensity and the urgency as a culture. And I think if AI has done something, it’s definitely amped it up for everyone. There’s increased urgency, increased intensity, increased innovation cycles. Products are changing on a weekly and monthly basis. New foundation models are being released left and right. So everyone really has to amp it up. Everyone has a harder job. That’s what comes to mind.

Andreas Welsch: I love that. Great answer. And indeed it’s a lot about amping it up. I saw a poll yesterday by Axios where they asked Americans across all different demographics, and the sentiment was, yes, it’s great to have so much AI, but can we slow it down a little bit? Can we tone it down and be more deliberate? So it’s interesting to see how these different demand dynamics play out. A lot of people have had access to generative AI tools and have been able to see this, and there’s been a lot of discourse in the media. So it’s interesting to see how that perception shifts as well. But definitely, we need to act with urgency, and we also need to make sure that we act on the right things. A couple of weeks ago I was meeting with C-suite leaders at an engineering company, and they shared their vision for how AI should help them operate the business better. Throughout that conversation we realized that they first need to have a data foundation in place, which they don’t have yet. It’s actually not so much about an AI transformation, but more about a business transformation. We’d love to do win-loss analysis. Great. How are you capturing the data? In people’s minds, or on spreadsheets, maybe, if we’re lucky. So we need to ask people. There are some fundamental things, and I’m wondering what you are seeing in your discussions. Certainly the companies you mentioned are well underway on their AI, machine learning and generative AI journeys. But I’m sure you’re meeting with others as well. So are engineering companies like the one I met with the only ones facing this, or are others going through similar challenges too?

Barr Moses: I think it’s a great question. Maybe a reaction to that and also to what you said prior. Let’s just start by facing reality. The pressure around AI is real, and that’s happening everywhere, in boardrooms, in the media and with your peer groups. We just did a survey with a couple hundred data and AI leaders and asked them a number of questions about their approach to data foundations and AI, and the results were pretty striking. A hundred percent of leaders currently have AI in production or are planning on having AI in production this year. So clearly everyone is feeling the hype and everyone is acting on it. I don’t think that’s new at all. However, only one out of three respondents actually think their data is ready for AI. In a world where a hundred percent of people are working on AI, but the large majority of people think that their data is not ready for AI, we’re obviously faced with a problem, if you will. What does being AI ready mean for your data? We can unpack that, but maybe just to double click into the implications for many organizations: I was just talking to the CTO of a Fortune 100 company, and he told me, look, by the end of this year I expect us to have over 500 agents out in the wild that we’ve built. And agents are not deterministic systems. I cannot say for certain what the output of a model will be. So I might have 500 agents out in the wild sharing unpredictable outputs that I have no oversight on. That is a scary reality for people, and the implications for revenue, brand and risk are real. To give you a couple of examples, a couple of years ago Citibank was hit with a fine of several hundred million dollars for data quality issues. So there are severe regulatory risks for practices that are not strong enough in data quality, and that’s been the case for a while. In the last five to ten years there have been multiple issues describing catastrophic results and impact when the data and AI platform was wrong. Fast forward to today. Just a couple of months ago a user was able to convince a chatbot to sell a Chevy Tahoe for one dollar. An agent sold a Chevy Tahoe for one dollar because the user was able to convince the chatbot to do that. You can imagine the repercussions in that instance. So organizations far and wide, on the one hand, need to invest a ton in AI or are being tasked with it. On the other hand, a hundred percent with certainty, I can tell you the foundations are not there. And the ability to deliver reliable data and AI products is becoming more important than ever.

Andreas Welsch: I was just at Data Summit in Boston about two weeks ago, where lots of data leaders met, from data management, data governance, many different roles, heads of analytics and so on. They shared a similar thing, but they also shared the concern that they’re not getting the budget for it. My boss says we have lots of budget that we can assign to AI projects, can you come up with an idea for an AI project? But I know that the data isn’t what it needs to be. It’s not complete, it’s not accurate, it’s not fresh, and what have you. What are you seeing there? How are leaders coping with that? On one hand there is budget, but management asks me to invest it in shiny things that we can report up and say, we’re doing AI, when we actually know that we need to fix the foundation before we can do something that is reliable and will not expose us to a significant risk.

Barr Moses: It’s a good question. By and large, from what we can see, most companies have budgets and are willing to invest in AI. A lot of it is experimental or innovation budgets, so there’s a question of what will be sticky and what will still be here in 18 to 24 months. Time will tell, but it’s very rare that I come across companies that don’t invest in AI. When I think about data and AI teams, there are primarily three core problems that I hear they are faced with, and that influences their budget decision making. The first core problem is that, just like every other team and every other function, they’re being tasked with finding ways to accelerate the output and productivity of their team with AI and automation. Every single team, engineering, support, customer success, data and AI teams, is being asked to do more with AI. That means workflows we were already doing, things we were already spending time on, need to be done faster and better using AI. The second core problem is that data and AI leaders own a data platform and a lot of enterprise data, a lot of proprietary data, and that data is feeding AI products. It could be a chatbot, it could be an agent, whatever it is. The data they’re providing to these AI solutions needs to be reliable. Why does this data even matter? When I’m building an AI product, everyone has access to the latest and greatest model. I can always switch between Anthropic and OpenAI and something else. We all have access to that with just a few clicks. I have an API and I’m done. But the thing that I have that my competitor or another company doesn’t have is proprietary data. I have enterprise data, so I know I have more information and can tailor these products to offer a better customer experience. I’ll give you an example. If you work with an airline or a hotel chain, I can offer a recommendation product, for example, maybe you want to have this for lunch or have this experience at the hotel. I can offer a more personalized concierge, if you will, because I know your preferences and I have prior information about them. So I can make more intelligent recommendations for you. If that data is unreliable, if I’m using the wrong data to feed those AI products, then obviously everything crumbles. So that’s the second core issue: make my data AI ready. A lot goes into that. You have to make sure that your structured data is reliable and your unstructured data is reliable. That’s a big thing. The third core issue that data and AI leaders struggle with, or need to divert attention and resources to, is that now that they’ve released AI products, how do they make sure those are reliable? How do we make sure the agent is not selling the Chevy Tahoe for one dollar? You can have the perfect training data, the perfect prompt, the perfect context, but the output of the model will still not be fit for purpose. At a very high level, when I think about decision making and budgets for a company, these three things are honestly table stakes. The first one, making data teams more productive with AI and automation, is really table stakes. The second is more and more table stakes: getting the foundation ready, having AI-ready data. The third category, building reliable AI solutions, is probably an area where people are still in the early days. Many companies are still just moving to the cloud, and there’s a lot that goes into actually delivering AI solutions. But at a high level, these are the three big problems that data and AI leaders have to tackle.

Andreas Welsch: I think that makes it very tangible, and it also shows this progression of how you should think about this in three simple steps. You mentioned that we need to put more emphasis on data and empower data teams more. What are you seeing, how are leaders getting buy-in along the chain of command to do these data projects, when everybody’s asking what our AI strategy is?

Barr Moses: Look, one of the things that is really important and obvious, but easier said than done, is tying what you’re building to real value and real impact, and maybe even finding cost savings along the way using AI. Let me give you a practical example of how you can use AI to save time. At Monte Carlo we offer observability for companies, to help them make sure that their data and AI estate is reliable, and we also use AI ourselves. We are building a suite of observability agents, and the goal of these agents is to deliver real, hard ROI for data and AI teams. I’ll walk you through how they work, because I think it’s super cool, but also because I think it’s valuable for data and AI teams. One of the things that data analysts in particular spend a ton of time on is figuring out what in their data needs to be monitored. I have a lot of tables, sometimes thousands, tens of thousands, hundreds of thousands of tables in my data lakehouse or data warehouse, whether that’s on GCP, Azure, Databricks, Snowflake, AWS, what have you. If I’m a data analyst or a data engineer, I need to make sure that data is accurate, but I have no idea what the requirements are. It’s hard for me to specify, at the table or even the field level, exactly what needs to be accurate, or what the definition of accuracy even means. So oftentimes what data teams do is spend time rigorously and tediously, first of all, profiling the data, understanding the structure of the data and the data itself, digging through metadata, making connections to understand the semantic meaning of fields. And then they come up with specific monitors. I’m just making this up: if this is an airline and I have a column where every row is a flight, I need to make sure that the flight numbers look a certain way and have a certain number of characters. So I need a monitor to make sure that data is always accurate, because I can’t mess up flight numbers. That process is really hard, tedious and manual. What we have done is release a monitoring agent. It’s in production already and used by hundreds of customers, so it’s been live for a while. We have a 60% acceptance rate, which means that 60% of the monitors we recommend are accepted and used. It’s weird to say, but this agent mimics human behavior. It goes through, profiles the data, tries to understand the meaning of the different connections between the data, and then makes recommendations for what you need to monitor. That cuts down the time for a data team from weeks to minutes, if you will. It’s really cool to start seeing what you can offer data and AI teams to make them more productive. What we’re building, and are about to release in the next few weeks, and what I think is a really cool application of LLMs, has to do with troubleshooting. The first thing that data and AI teams do is monitor data, because they need to know when the data is wrong. The second thing is that when the data is wrong, they have a workflow for understanding why: they triage, troubleshoot and try to understand the root cause. Our troubleshooting agent mimics what a data steward or data analyst would typically do, which is start by coming up with hypotheses for what might be wrong. Let’s say I get a notification that there’s a particular issue with a particular report, and that report is used every single day by our field operations team. It’s high visibility, so it’s really important that I look into it. I start coming up with hypotheses for what could have gone wrong, and I go upstream table by table. Let me check if the data arrived on time. Let me check if this upstream data arrived on time. Now let me check whether anyone made a change to the code somewhere, whether anyone broke something. Maybe the ETL job failed and left the data incomplete. I basically have to come up with a list of dozens of hypotheses and start to check and test them to discover what happened. What we’ve done in this troubleshooting agent is build an ensemble of LLMs. There’s a master LLM that comes up with the list of hypotheses, and then it spawns off a new agent for every single hypothesis. Every agent then recursively looks into its particular hypothesis, so you can have up to around a hundred agents running in less than two minutes, all looking into different hypotheses for what could have gone wrong at the same time. Something that could have taken me years, frankly, to go through, a troubleshooting agent does by breaking the problem down into different tasks. Every task is a hypothesis to look into, and many LLMs look into each hypothesis and report back on what they found. Then a bigger LLM summarizes all of that and gives you a TLDR: here’s the root cause, this is the reason for this incident. That is really powerful. You’re taking things that data and AI teams are already spending time on and cutting the amount of time significantly, so the ROI in those instances is very clear, if that makes sense.

Andreas Welsch: Great example, and very tangible too. I love these examples where it would have taken one person or a team of people months or years to do this, and now we can do it in minutes or even less than a couple of minutes. Super powerful, and I think we need more of these examples that show what this can actually do beyond the hype talk. So it sounds like you are already working on some exciting things and are testing this with customers or rolling this out to customers. Super cool. Now, it’s been a few weeks, probably even a few months, since we started talking and setting up today’s session. I remember earlier in the year I came across an article from you where you said agentic AI might actually be doomed to fail. We somehow landed on the topic of getting data AI ready, but I’m wondering why you think AI agents might be doomed to fail. When are they doomed to fail, is probably the better question to ask.

Barr Moses: Good question. We talked about this earlier on this call: when folks are releasing hundreds of agents into the wild and waiting to see what happens, I think we know what the story looks like. I love this example of the troubleshooting agent because, when our team shared that they were working on it, that was a turning point for me. There’s a lot of AI skepticism out there and a lot of questions about what it actually means. Just to be clear, I don’t think AI will replace people or replace data and AI teams. If I look at a corollary like engineering, the more advancements we’ve had for engineers, the higher the demand for engineering teams. I think the same will be true for data and AI teams. The more we use AI, the more we need data, and there’s going to be an increase in demand. It doesn’t mean that we don’t need to change how we work and what we do. I very much believe in that. This application of the troubleshooting agent really brought to life for me how powerful it can be if you put LLMs to use. You can imagine a lot of complex problems you might be working on today that, if we intelligently break them into smaller tasks and use different smaller and larger LLMs in different instances, you could teach LLMs to do really clever things. I think it’s important for me to say that, because figuring out what agents can do for your business is important, and I am convinced today that there is value in that for your company. You have to see it and experience it yourself to believe it. It’s very different from theory. There’s a lot of talk about how, in a couple of years, software engineers will really just be managing a fleet of agents: one agent to build features, one agent to fix bugs, one agent to do PRDs with you. Similarly, you can envision that data and AI teams have a fleet of agents, one agent to monitor their data, one agent to troubleshoot, and we’ll all just sit in our cushy chairs and monitor the agents. I can’t wait for that. It means our world will be very different and how we work will be very different. But the reason I wrote that I think there’s a high chance that agentic solutions will fail is if we are not thoughtful about what it means to build high-trust agents. Let me give you the most popular example, which went viral on X a couple of years ago, or maybe last year. Someone asked one of Google’s LLM solutions what to do if cheese is slipping off their pizza, and the answer was, oh, you should just use organic super glue, no problem, to glue the cheese back onto your pizza. Just to be clear, the training data was good, the prompt was good, everything was good. But the output was totally inappropriate and made no sense for the context. I will continue to use Google, or maybe Perplexity, but Google can get away with it. Most organizations in the world can’t get away with building agents that share responses like that, and continue to have trust. So I think the way we manage our agents and our data and AI platform needs to change drastically. One of the things we’ve done at Monte Carlo over the last several years is work with thousands of enterprises to figure out how to build reliable data and AI products. Based on thousands of customers and conversations, we’ve found that problems with data and AI products can be reduced to four core problems. The first is that data and AI solutions can be running on bad data, meaning the data you’re feeding might be inaccurate or late or just wrong. The second core reason is a code change. Maybe it’s a bad join, maybe it’s a schema change, maybe it’s a code change in the agent itself that changed something dramatically. The third reason things can go wrong is a problem with the system. That could be your ETL jobs, like Airflow or dbt, or orchestrators, or things like LangChain or LangGraph. Anything along your data and AI stack can go wrong, because one thing is for sure: a hundred percent of systems fail at some point. And the fourth thing that can go wrong, which I alluded to earlier, is that you can have all those things work perfectly, the perfect context and the perfect prompt, but the model output will still not be fit for purpose. So for people who are building agents and AI solutions, unless we start thinking about the holistic health of those systems as a function of those four things, data, code, systems and model output, we will fail.

Andreas Welsch: That really hits home and comes down to the core of it. Barr, thank you so much for sharing this. We’ve covered a lot of ground in the last 30 minutes, from helping data teams and understanding that they have a big responsibility in what they can do, to observability, making sure that our data is accurate and correct. We talked about some of the risks, what happens when you don’t do that, and where the failure points are in agentic AI. It was also great to hear what you’re doing at Monte Carlo and how you’re bringing AI and agents to data and AI teams. It sounds like a really exciting opportunity and a really exciting space too.

Barr Moses: Thanks.

Andreas Welsch: Wonderful. Thank you so much for joining us and for sharing your experience with us today.

Barr Moses: Absolutely. Thank you for having me. It’s an exciting time. There’s a lot of exciting stuff happening, and I think the future is bright. I’m excited to be part of it. And thank you to everyone joining.

Listen and Subscribe

Related Reading

Frequently Asked Questions

How many leaders think their data is ready for AI?

In a survey of a couple hundred data and AI leaders that Barr describes, every respondent has AI in production or plans to have it this year. Only one out of three think their data is ready for AI.

What three core problems do data and AI teams face?

According to Barr, data and AI teams are asked to speed up their own output with AI and automation, to make the proprietary enterprise data that feeds AI products reliable, and to keep the AI products they release reliable once they are live.

What does Monte Carlo’s troubleshooting agent do?

Barr explains that it mimics what a data steward or analyst does. A master LLM comes up with a list of hypotheses for what went wrong, spawns an agent for each one, and a larger LLM summarizes the findings into a root cause. Barr says up to around a hundred agents can run in less than two minutes.

What are the four core reasons data and AI products fail?

Barr names bad data, code changes such as a bad join or a schema change, problems in the system such as ETL jobs or orchestrators, and model output that is not fit for purpose even with the perfect context and prompt.

About the Author