How Do MLOps and LLMOps Keep AI Results Consistent? Kristen Kehrer on Production AI

Andreas Welsch: Everyone thinks the hard part is getting AI into your business. But what’s even harder is keeping it there. So today we’ll talk about how MLOps and LLMOps help you drive better results. So join me for this one.

Andreas Welsch: All right, welcome back to another episode of What’s the BUZZ, where leaders share how they have turned hype into outcome. Today we’ll talk about how MLOps and LLMOps help you drive better quality and better results from your AI systems. And who better to talk about it than someone who’s actively involved in that and who’s been working on this topic for a long time. Kristen Kehrer, thank you so much for joining.

Kristen Kehrer: Hi, thank you so much for having me.

Andreas Welsch: Wonderful. Why don’t you tell our audience a little bit about yourself, who you are and what you do?

Kristen Kehrer: Hi, sure. So I’m Kristen Kehrer. I’ve been a data scientist since 2010. I started doing econometric time series analysis and forecasting in the utility industry. I moved into health care doing things like helping motivate patients to get their colorectal cancer screenings, and then I spent six or seven years in e-commerce doing traditional ML e-commerce use cases before going off on my own, where I had the opportunity to work for a company that was specifically in MLOps. I’ve helped build an LLM-powered application for a health care consultancy. I was the head of data science for MoneyGram. I love building other types of things, like the computer vision example I built to detect school buses going by my house. So I absolutely live and breathe this space and couldn’t imagine doing anything besides data.

Andreas Welsch: We have a lot of good topics to talk about today. I just came back from the Gartner Data & Analytics Summit in Orlando a couple of days ago, and I saw the conversation there was still very much about data, obviously, and analytics, as you’d expect. A lot of times when we talk about AI, when we talk about agents these days, it’s usually more at the business level: how do we use agents, what do we use them for? So for me, it was great to see that grounded in the reality of, hey, we actually still need data. It doesn’t go away. But having been to other conferences and vendor conferences, I know there are lots of polished demos. They look very shiny. They’re quickly put together, but really the crunching point is getting AI into production safely, and that’s a whole different thing. What are you seeing? What do leaders need to keep in mind as they embark on this journey of bringing AI, bringing agents, bringing even machine learning and predictive into their organization?

Kristen Kehrer: First of all, I had such FOMO seeing your photos from Gartner. That looked amazing. I think the big issue is that I built my first LLM demo and it took me a couple of hours, right? But when we try to take something like that to production, there is iteration in a way that is more intense than we see with traditional ML, and the amount of collaboration required is a lot higher. I think leaders aren’t necessarily prepared for the amount of collaboration that is involved. They hear that we’re going to do an AI project and they assume it’s going to look like traditional ML, where you scope it, you give it to a data scientist, they’re able to grab the data, and the metrics they’re working with are trying to reduce false positives, trying to improve the AUC or some loss function. And now, with the outputs, we have to be concerned with brand tone, we have to be concerned with accuracy. All of a sudden the data we’re working with is different. It is not data from a database. It might be our internal docs, some knowledge repository that might be text, unstructured data, but then there are also the prompt templates, the retrieved context that we get before we send it to an LLM, and the few-shot examples that we’re including. There are all of these pieces, and all of them, when they change, can change the model behavior. So there’s just so much more to manage and make sure it’s versioned, so that what we’re putting into production is actually auditable, and so that we have some sense of confidence in what we’re putting out there, which is becoming much more likely to be customer-facing than it was before.

Andreas Welsch: I see all these great demos on LinkedIn, and people say, hey, look, this is what I’m building, this is what I wrote code for in an afternoon. But I’m pretty sure it looks a little different if you’re trying to bring this kind of AI into an organization that’s exposed to financial, legal, reputational risk, where it’s not just, let me fix this agent if it doesn’t do what it’s supposed to, let me tweak the prompt right here. If this thing is in production, if it’s live, if you’re even in a regulated industry, financial services, or life science, health care, that’s a whole different ball game, right?

Kristen Kehrer: Oh, yeah. No, it absolutely is. With a demo, or even when I was building something where the goal was to get it live at some point, you can get some fantastic output. I’ve created videos myself where I get fantastic output, and I write it up or I put it up on YouTube, and I am helping others understand how to build a RAG system and what to think about when you’re creating embeddings, when you’re retrieving. So those are useful, but at the same time, if I’m creating something for real, there’s a ton of edge cases. Some of the things are really simple. If I’m going to be sending a communication to a cancer patient, I know it’s a letter that may go off to a cancer patient, and I know I don’t want to put a ton of exclamation points in that, right? As a data scientist, I can make some judgment around how that communication text should feel. But at the end of the day, there’s a lot there where I don’t have the domain expertise. You say you want that going out at a third grade reading level. Well, what is a third grade reading level, right? If you ask the system for a third grade reading level, there are multiple ways of measuring that. The LLM isn’t going to know exactly what that is. So now what are we talking about? Is that the number of syllables in a word? Is it that we’re going to have fewer than 10 words in a sentence? We need to think more deeply, and this requires the subject matter expertise of the business around exactly how they want to communicate, which are often things they’ve learned to do through communicating with each other, but they haven’t thought about how to actually instruct a computer to do that. These are a lower level of instruction that we all need to work out together. So even though I may get some great output when I try it for one use case, it might not work for another. How do we version all of these prompts, so that when it works for some but not others, and I’ve gone off and tinkered some more, and now the other ones aren’t working, we are thinking about this in a methodical way? The versioning is what allows us to get there, to audit it and see what truly gave us the best responses, for something that is very iterative in a way that we haven’t seen before.

Andreas Welsch: So it sounds like that’s the part where you mentioned earlier that collaboration is key. Again, coming back to business and technology teams needing to work together to figure out what it is that we actually need to do, and how we can put the business requirements in technical terms and the technical capabilities to support the business? Now, we’ve obviously talked already quite a bit about MLOps and LLMOps, but what are the key differences that you see between the two? Why do we have two terms, even?

Kristen Kehrer: Okay. Well, LLMOps is an extension of MLOps. Machine learning operations is all of the operations that give you the ability to reproduce your results and audit your results. So this is tracking your data versions and the lineage, right? So that you can see how it was changed over time, and tagging that to a particular model version. Then there’s your experiment tracking. Back in the day, in 2011, I’d build these bespoke models where I’m fitting a polynomially distributed lag of price or whatever, and I’m deciding, hey, if I’m going to include residential housing permits in my model, how many months should I lag that? I’m fitting each variable and I’m determining how to dummy out certain pieces of information to get a more accurate forecast. Over time we aren’t fitting as routinely, but you can imagine that in that case I might have fit that model 30 times before I found the ideal model. And maybe the best model was actually the 20th time that I tried, right? Previously I was writing that in a paper notebook, and the hope was that I could go back and say, okay, this one had the best loss function, I’m going to use that. But nowadays we don’t have to do that, because I add a couple of lines of code at the top of my notebook and now I have all these model versions. At the end I can sort them by what had the best loss function, what had the best false positive rate, and I can choose the model that was actually the best. It’s also looking at CI/CD, so that everyone is deploying the same way, so that we don’t have these error-prone processes where people come up with their own tests to get into production, they’re not standardized, and edge cases get through. And it’s monitoring and feedback loops, and even a little bit of explainability and governance, so that our stakeholders trust our results and, when regulators come along, everything’s auditable. That’s your MLOps, right? And then for LLMOps, there’s an additional layer added. Now we’re not just versioning the data that the model is built on, because often we’re not even training the model, right? We might be using a foundation model, but we still have to track our prompt templates and the retrieved context that we get, because if we have a problem with our output, maybe it’s that our retriever isn’t retrieving the right information. Maybe it’s that after we’ve retrieved that information and sent it up to the model, we’re not getting the right results back, right? We need to be able to diagnose and do root cause analysis on exactly where things are breaking down. We also have few-shot examples in there. We might have fine-tuning data sets. We have user interaction logs, and we want to be able to track all of that. That’s where we have our MLOps, and LLMOps is the extension that takes care of all of the additional pieces that weren’t part of traditional ML but that we now have as part of a RAG system.

Andreas Welsch: So I lost count of all the technical terms that you mentioned.

Kristen Kehrer: It’s a lot of pieces.

Andreas Welsch: False positives, AUC, area under coverage, right?

Kristen Kehrer: Area under the curve.

Andreas Welsch: That sounds awfully complex. I thought it was easy. I thought that agents are taking over the world, putting all of us out of a job, and it’s happening tomorrow. This sounds like real work, and like there’s some plumbing that you need to get right.

Kristen Kehrer: Yeah, that’s always sort of been my thing when people said that AI is coming for jobs. I always wondered if it was people who understood the complexity that said that, or if it was somebody else. Maybe it’s leaders who aren’t actually in the weeds, but somebody does need to know what’s going on and be able to work with these systems and know them intimately.

Andreas Welsch: So if you’re listening or watching, take note of that. It’s gold right there. We obviously need to understand what’s happening underneath the surface, at least to some degree. And I think then many of these claims, many of these systems, don’t seem as magical and mystical anymore when, at the end of the day, we bring it down to crunching numbers and statistics and probabilities and so on. Now, obviously LLMs have been around for a while. Companies are experimenting with that. I even hear many of them are doing that with agents but just aren’t talking about it so much publicly because they’re afraid of backlash. But be that as it may, how do you see LLMs changing the requirements for managing that life cycle when you bring an LLM-based application into production? What do you need to be aware of there? You already mentioned a few things as we were starting out, but what are some of the top things to keep in mind as you bring LLM-based agents or applications into your environment?

Kristen Kehrer: Like I already said, it’s the collaboration piece. But to go into it more, if you’re thinking of doing an AI project, you need to understand upfront that the demo is only going to take a little bit, and that there is going to be time and iteration between the demo and actually getting to production, and you need to budget for that. You also need to set up your team correctly in the beginning and bring in the SMEs, bring in the product stakeholders, get everybody working together, and have a clear understanding of what you want the output to look like. It’s back to when we used to talk about that, because I feel like we lost sight of it for a while and things became about cutting headcount or other stuff. We need to be thinking about the customer. We need to think about exactly what we want that output to look like, and we need to be clear about that from the beginning, because otherwise there’s going to be work upstream to course correct when you’ve got a system that isn’t working well. I have seen companies reduce their headcount and launch an LLM that works poorly, and the customer reps that survived are now inundated with calls. The customers are frustrated because monitoring wasn’t in place. They were evaluating monitoring platforms at the time, but there was nothing in place, right? So I went through myself and looked at about 300 chats, and I used LLM as judge and compared it to what I was doing, and LLM as judge did work pretty well. But you’ve got this system online, you’ve got to take it down, you’ve got to do some manual analysis to understand exactly how bad it is, how we fix it, and where the problem is. A lot of times what I’ve seen is that something gets put into production without a knowledge base that’s big enough, because people get really excited, and instead of really building out the data the way they want it, the data seems to fall through the cracks, right? We need to be able to cover a hundred use cases. We need documents that are going to cover that, and they say, we’ll do 50 and let our old solution handle the other 50, because we want to get this out there, and it all just seems not to be optimal.

Andreas Welsch: Let me ask you a question. You mentioned LLM as judge. Can you talk a little bit more about what that is and why we need it?

Kristen Kehrer: Yeah, so LLM as judge, at least in the way that I was using it, means we have all of these customer chats saved, right? I could go through and see that the LLM said, how are you today, what is your question, and the customer says, my question is this, and then it says, actually we’re not set up to handle that, we’re going to do this, or whatever. As the conversation goes through, you can see somebody getting frustrated. But instead of a human going through, you can just feed that to an LLM and ask, where does it look like customers are getting frustrated? What types of topics are customers getting frustrated on? It would give me back four bullet points: it seems like these topics were handled well, for these topics it looked like people were using very short sentences, and it would list other signs that people are frustrated, like people using four-letter words. I don’t remember them off the top of my head. So you’re able to get output from an LLM judging the output of your original LLM, and it is pretty good, yeah.

Andreas Welsch: So why do we need another LLM to judge what the first one did? Wouldn’t you expect that the first one is good enough to know what you want it to do and do it well?

Kristen Kehrer: No. With traditional ML, we’re able to say, okay, this person was denied a loan. And if we’re using explainability, we also understand what types of factors are going to make it so that this person gets denied for a loan, and their metrics, so their numbers. But now, when you have text, there are metrics for LLMs, but they do not get at the brand tone. They do not get at those nuances of, actually, this person asked a question that the LLM probably should have understood but didn’t. Or, this was not the proper context for what this LLM is supposed to be doing. An LLM can look at that, and it’s going to be a lot faster than a human having to look at it. But in the same sense, we don’t have the number metrics that we are allowed to use in ML that can say this decision was right or this decision was wrong. So it is useful for us there.

Andreas Welsch: I see. Okay, that makes a lot of sense. So you have an evaluation, basically, to say how well did it create this, how well does it align to brand voice, tone and other requirements? All right, well, obviously companies have data. They’ve got lots of data. I think it was 80 to 90% of data is actually unstructured, lives in documents. But what do you do as a leader if you’re being asked to do some AI and you actually find out your data isn’t as clean, isn’t as complete, isn’t as fresh as you need it to be? Can you still work with an LLM? Can you still build a RAG system? What’s your advice to leaders who are facing that situation?

Kristen Kehrer: I think it depends on the use case and what you’re trying to do, right? But in the higher-stakes scenarios, where the output needs to create something that’s going to go out to customers, you need to make sure that your data is clean. I saw this when I was building an LLM application. The company had been creating these types of communications for so long. They had so many cases, but not all of them were general enough to go into a RAG system so that it would work for creating these communications at scale for other companies. What they chose to do was have subject matter experts go through each of the example cases and write them in a way that they knew they’d feel confident feeding to the system. Because if you’re going to put multiple use cases, multiple documents for the same type of thing, in your knowledge base, and they’re not exactly what you want, if they have additional information that you don’t want coming up, you could put in your prompt template that you don’t want that extra thing, but now your prompt template is going to get huge. So you really do need clean data, and it is a different consideration than we’ve had in the past, where you’re getting your data from a database. Now it’s docs, and those docs do need to be updated, and I think that is part of the bottleneck. We see these systems, and I’ve seen systems go live with 18 documents in them as their live POC. But then it takes forever. The cleaning of the data seems to be an afterthought, where, again, leaders need to structure the team so that the subject matter experts and the product people who are working with the data scientist have time carved out in their schedule to make sure that the data going into the knowledge base actually gets done, and that it is a priority, because I do see it as something that has been getting left off. The data scientists are off optimizing on this small knowledge base, and later on all this more data is going to come in. How is that actually going to impact the system? How is the system going to work on that? Or now we’ve overfit for this smaller subset of documents. At the same time, maybe you can feed multiple documents to an LLM, prompt it, and get something that’s really close to exactly what you want to be putting in your knowledge base. But the data does need to be cleaned, and it’s no longer really the data scientist’s job. We spent so much time data cleaning before, and now you still need to spend a ton of time data cleaning, but it’s not in our domain the same way it was before.

Andreas Welsch: That makes sense. Now, before we get to the end of the show, I was wondering if you can summarize the key three takeaways for our audience today. You’ve covered a lot of ground.

Kristen Kehrer: Yeah. MLOps is a necessity. LLMOps is an extension of MLOps. There are ways in which LLMOps is still emerging and will probably be even more useful to us in a couple of years. One thing I want to call out is that with experiment tracking and other areas, sometimes you can put in just a couple of lines of code and you get a lot of tracking out of the box, but then there’s still work to do to log other artifacts and things that don’t come standard. LLMOps is the same. It can feel sometimes like you get a lot out of the box and you’re like, wow, this is so easy. But there is work to get the rest of it built out. And the third piece is really that this is so much more collaborative than it was with machine learning, and you need to budget for that up front.

Andreas Welsch: I love it, especially the part about collaboration, because from my experience too, that’s where the magic happens, when you bring business and technology together and each one learns a little bit from the other side, too. Everyone gets a little smarter, and typically the outcome tends to be a lot better than if it’s only one side that thinks they’ve got all of that figured out. All right, Kristen, it was a pleasure having you on the show. Thank you so much for sharing your expertise with us. And for those of you in the audience, if you’re not following Kristen yet, do give her a follow. She’s awesome.

Kristen Kehrer: Awesome. Thank you so much.

Andreas Welsch: All right, thanks. See you next time for another episode of What’s the BUZZ. Bye-bye.

Kristen Kehrer: Bye-bye.

Kehrer said LLMOps is an extension of MLOps. MLOps is the operations that let a team reproduce and audit results, including data versions and lineage, experiment tracking, CI/CD, monitoring, feedback loops, explainability and governance. LLMOps adds tracking for prompt templates, retrieved context, few-shot examples, fine-tuning data sets and user interaction logs.

Kehrer said her first LLM demo took a couple of hours, but production involves more intense iteration and a lot more collaboration than traditional ML. Prompt templates, retrieved context, few-shot examples and internal documents can all change model behavior, so they have to be versioned and tested with the business, and teams need to budget for that time.

Kehrer described LLM as judge as feeding saved customer chats to an LLM and asking where customers appear frustrated and on which topics. She said it is much faster than a human reviewing every chat, and that numeric metrics for LLMs do not capture brand tone or whether the system should have understood a question.

Kehrer said that when output goes out to customers, the data in the knowledge base needs to be clean. She said unclean or inconsistent documents can lead to a very large prompt template, and that cleaning is often an afterthought. She advised leaders to carve out time for subject matter experts and product people to prepare the documents.

Kehrer said leaders should understand upfront that the demo is only a small part of the work and budget for the iteration between demo and production. She said to set up the team at the start with subject matter experts and product stakeholders, and to be clear about what the output should look like and how it serves the customer.