How Do You Scale Enterprise AI Products Across the Business?

Andreas Welsch: Today we’ll talk about scaling enterprise AI products across the business. And who better to talk about it than someone who’s doing just that. Srujana, thank you so much for joining.

Srujana Kaddevarmuth: Hi, Andreas, excited to be here.

Andreas Welsch: Wonderful. Why don’t you tell our audience a little bit about yourself, who you are and what you do?

Srujana Kaddevarmuth: Sure. Happy to share. Hello everyone. My name is Srujana Kaddevarmuth. I lead the AI portfolio for a Fortune One company working out of the Bay Area here in Silicon Valley. I enjoy being associated with the causes of using AI for good and being an active advocate. I serve on the board of the United Nations Association and focus on using AI to attain the United Nations Sustainable Development Goals. Looking forward to this conversation.

Andreas Welsch: Wonderful. It’s always great having leaders on the show who not only use AI in business, but also who have a deep passion for using it for good. So thank you for that. Excited about our conversation today. Should we play a little game to kick things off?

Srujana Kaddevarmuth: Sure.

Andreas Welsch: Wonderful. All right. This game is called In Your Own Words, and when I hit the buzzer, the wheels will start spinning. When they stop, you’ll see a sentence, and I’d like you to answer with the first thing that comes to mind and why. In your own words. And to make it a little more interesting, you’ll only have 60 seconds for your answer. And for those of you watching this live, also drop your answer in the chat and why. Always curious to see what you come up with. Srujana, are you ready for What’s the BUZZ? Okay, then here we go. If AI were a song, what would it be? 60 seconds on the clock.

Srujana Kaddevarmuth: I think it’s going to be rock and roll.

Andreas Welsch: Okay, why that?

Srujana Kaddevarmuth: Because of how the industry is evolving. I don’t think there could be a better way to describe that. It’s very interesting. There is a lot of hype and practical challenges in terms of deployment, and the industry is evolving at a lightning pace. It’s very difficult to catch up with. So I think that’s the best way to describe it.

Andreas Welsch: So rocking and rolling and lots of moving and shaking. Perfect. Wonderful. So why don’t we talk about the actual topics we want to talk about today. And that’s productizing AI. I know for probably the last two years or so, there’s been a change in mindset and perception that AI isn’t just a project. It’s not just a one-time thing, but we should actually think about it as products. And that’s why I’m really excited to have this conversation with you, because I know you’re approaching it in the same way too. So I’m curious if you can talk a bit about that for our audience. What does productizing AI actually mean and how do you do it?

Srujana Kaddevarmuth: Yeah, absolutely, Andreas. We are in this age of data deluge, wherein 1.1 trillion megabytes of data is getting consumed and generated every single day. Around 70 percent of the world’s GDP is undergoing digitization as we speak today. When such a humongous amount of heterogeneous, non-intuitive, messy data is generated, it becomes a responsibility of everyone involved to use it in a safe and effective manner. And the organizations have an added incentive to use this data to drive value for the large enterprises. So they are focusing on two important concepts here. The first one is around data democratization and the second one is around AI productization. Productizing AI is a journey of translating the insights generated from exploratory analysis into scalable models that can power products. It means developing and deploying these algorithms in the production environment and doing that in an efficient and effective manner. There are multiple benefits the organizations can reap by productizing AI. The top three benefits are, the first one is around human capital efficiency. It helps the organization use the scarce AI resources to do the niche AI work, freeing them up from the mundane repetitive tasks, thereby reducing the attrition and keeping them happy. The second aspect is specifically around ensuring that we are effectively utilizing this AI, but also bringing in and fostering innovation within the organization. Just imagine a lot of bandwidth of these AI experts would get freed up because a lot of processes are going to get automated. Now they can use that bandwidth to run mass experiments and drive some innovative thinking, thereby fostering innovation. And the third aspect is around standardization of the technology capabilities. Just imagine if the large organization is operating across different geographies, across different channels. Building AI as a product is going to help us standardize these technology capabilities across these different channels, which will help us build and monitor these capabilities in a responsible and effective manner. So I think there are a lot of benefits and perks for the organizations to prioritize AI, and that’s why they are doing it.

Andreas Welsch: What’s interesting to me is that the first one you mentioned is data. It all comes back to data at the end of the day. And I think it’s an interesting realization in the industry over the last even 18 months, 24 months that yes, there are things like generative AI, but if you want to get better results, concrete results, relevant results, you still need data. You need your data. This one makes it unique.

Srujana Kaddevarmuth: Absolutely.

Andreas Welsch: Now, when you talk about productizing it, what does it mean from your point of view in terms of process, in terms of mindset, in terms of resources, when you work with the business that’s quote unquote just looking to address a business problem to, I don’t know, be more efficient, be more effective, save a couple percentage points or fractions of a percent here or there. How do you do that? How do you run that process?

Srujana Kaddevarmuth: Yeah. Just imagine, when you’re talking about the large enterprises, there are going to be multiple use cases where we can use AI. However, we cannot be building an algorithm or a model separately for each use case. It’s not efficient at all, and it’s not scalable. So productizing AI helps overcome those challenges and helps us expedite the overall value adoption across the organization. But there are also certain nuances that the business is considerate about and concerned about when deploying these algorithms in the production environment and productizing AI. For example, one of the challenges that productizing AI as a concept would face is around lack of resource investment and leadership championing that. Why? Because in many scenarios, the projects succeed, but there are certain scenarios in the real world when the projects or the initiatives fail. When that happens, the best case scenario is that we have only built a proof of concept. And the worst case scenario is that we have built the entire product end to end and the results are not relevant, they are not good enough. So this loses the credibility of the management and the leadership on these data science or AI teams, thereby blocking the resource investments. Similarly, the second challenge could be around the production systems malfunctioning. When we look at AI and machine learning algorithms, they get smarter over a period of time if they are connected to the constant and new data feed. If they are not, then they degrade in quality and that happens pretty quickly. So if we are not deploying it efficiently in the production systems, and if we don’t understand that these algorithms are very different from software applications, then it’s going to be a challenge. One of the mistakes that most people in the industry make is thinking of these ML algorithms as similar to software applications. When that happens, they don’t understand that there are nuances in how we deploy these algorithms. So that poses a risk. And the third risk that the industry is facing when you look at productizing AI is specifically around legal implications. These AI algorithms are a statistical representation of the world that we live in. So they come up with the outcomes based on the data that they have been trained on. Sometimes that data can have certain kinds of biases, and by productizing that, we are institutionalizing those biases across the organization, which is a challenging and a bad scenario, and it can lead to legal and reputational damage and risks for the organization, and it should be mitigated in an effective manner. So yes, productizing AI helps. There are a lot of benefits, and we could do that efficiently, and that’s why companies are doing that, but there are also certain challenges and risks that we need to be aware of.

Andreas Welsch: That makes sense. So folks, for those of you in the audience, if you have a question for Srujana, please put it in the chat and we’ll take a look in a minute or two and then pick some of those up. Now you’ve already started talking about some of those risks and nuances, but what are some of the additional things that you’re seeing when it comes to deploying AI in the enterprise at scale and how can you mitigate them? I think you already talked about resources as one constraint, but what are some of the other ones you’re seeing?

Srujana Kaddevarmuth: Right. We spoke about some of the risks associated with productizing AI, but if we identify those risks early on in the life cycle, it will be easy for us to mitigate them. And we can mitigate them by primarily focusing on three different principles. The first one is around model flexibility. The perfect model or algorithm does not exist in this universe. However, we can build algorithms that are flexible enough to consider certain real life scenarios and accommodate them without having to undergo major architectural redesign, because the technology is evolving so rapidly. Whatever was relevant when I started my career is not relevant today, and whatever is relevant today would not be relevant tomorrow. So the algorithms need to be flexible enough to adapt to these technological changes, so that we don’t have to do a tech stack revision. So that’s one. The second aspect is around computational efficiency. We can mitigate the risks of productizing AI by focusing on the computational efficiency, the functional usage, the runtime, because one of the mistakes that people make is focusing a lot on accuracy of the models. While accuracy of the models is important in the proof of concept stage, in the production stage deployment it’s equally important to focus on the functional usage, computational efficiency and runtime, lack of which would lead to risks. So ensuring that there is the right precedence, especially when deploying the algorithms in the production environment, is going to be super important. And the third aspect is around building model wrappers. As I mentioned, machine learning algorithms operate very differently from software applications. They need to be connected to the constant and new data feed. And that has to be done through building strong and effective data wrappers or model wrappers. If that does not happen, then there would be risks of project failures as well as initiative failures, primarily because any deviation that is happening due to the broken data feed is very difficult to detect as compared to outright application failures. So focusing on these aspects and mitigating them along the way is going to help us productize AI in an efficient and effective manner.

Andreas Welsch: Thanks for sharing that. I’m curious, when you say you need a wrapper, certainly I understand the part about data needing to be fresh. There has to be a stream of new data. But what do you mean by a wrapper around the model? What does it mean? What does it look like?

Srujana Kaddevarmuth: It’s more about connecting the data feeds in and out. For the inputs, we look at eclectic data sources, which could be first-party or third-party data sources, and connect those to the model. And as an output, we take the outcomes or the outputs of the model and keep creating a feedback loop so that the algorithms can constantly learn as well. That happens when we create the data wrappers, both as an input as well as the output. There would also be an inferencing layer, and some of the outcomes would go to the visualization layer. Creating those effective wrappers is going to be important across the overall value chain or across the ecosystem, so that we have robust algorithms, robust models, as well as a data flow that is current and efficient across the ecosystem.

Andreas Welsch: Okay. Thank you for explaining that. Now, we’ve talked about machine learning. We’ve talked about data, obviously, quite a bit. I know the industry is still super excited about generative AI. From your perspective, when should you use machine learning? When should you use generative AI? When should you use them individually or together? What is it that you’re seeing or that you would recommend?

Srujana Kaddevarmuth: Yeah, so I feel, Andreas, that there is a lot of hype around generative AI. It’s not that generative AI algorithms do not have a lot of potential. They have a lot of potential. However, when I say hype, I mean that there has been a lot of investment that has happened in generative AI across industries over the last year, but the return on investment has not been proportionate. So I see just a lot of craziness across the industry around using generative AI algorithms for every sort of use case, which should not be the right approach. The right approach should be, yes, there are going to be certain use cases which require deep language understanding, where generative AI would be the best solution. Focus on using generative AI for those use cases. But there are several other use cases, classical regression and classification problems, where conventional AI can solve for that and even help with optimization. So we should be using conventional AI in that context. This is primarily important because generative AI as a field requires a lot of compute infrastructure. There is a lot of cost associated with generative AI deployments. It’s not just cost to the companies, but also cost to the environment, cost to the society. So one needs to be really mindful of when and how to use it. For example, there are three important costs associated with using generative AI algorithms. The first one is the compute cost. There is an infrastructure cost. And then there is an energy cost. It requires a significant amount of energy to run. So being mindful of when to use generative AI algorithms is going to be super important, rather than just using them for the sake of it. Look at the true use cases which require deep language and intent and context understanding and use generative AI to solve them. And when we are doing that, we can still use various methodologies like load balancing, because not all the resources are consumed at all phases of the project life cycle. So focusing on scaling resources up and down dynamically is going to be helpful, as well as using both GPU and edge computing and looking at distributed ways of computing. So finding different avenues to ensure that we efficiently use these generative AI algorithms is going to be important, but we should start with asking whether this use case actually qualifies for generative AI deployment. That’s how I would approach this.

Andreas Welsch: Awesome. And as I’m listening to you, I’m getting really excited about this whole topic of edge AI and doing your inference and also your language processing on device and then only reaching out to the cloud if needed. By the way, I was watching some reviews again of the Rabbit R1 personal assistant and of the Humane AI Pin over the weekend and just seeing what some of that latency is if you need to make a call to the cloud every time. Certainly at the moment, it isn’t as user friendly as you would like it to be, as you’re used to having things on device. But anyway, I’m digressing.

Srujana Kaddevarmuth: No, I believe, Andreas, that’s important. The reason why I say that is that even the World Economic Forum made a prediction in terms of the gallons of fuel or the energy that’s going to be used to cool down these processes and servers. And we are looking at quantum computing to help us mitigate the situation in the coming decade. But overall, I think it’s really important for us to not reach the situation wherein we have to desperately look for remedial measures, but to do that in a conscious manner right from the beginning.

Andreas Welsch: That’s a great point. Absolutely. I think that also goes back to your point about machine learning, statistical algorithms, regression and so on. If you can use those, and we know that they use less power and less energy, with machine learning, even better results in many cases, then you don’t need a hammer for every problem. Now what are maybe some examples that you are seeing around machine learning and traditional, more classical ways of doing things in business?

Srujana Kaddevarmuth: Yeah, for the traditional algorithms, there are a lot of supply chain related use cases, such as road optimization and inventory management. We can use traditional AI algorithms in the customer domain and marketing domain too. Multi-touch attribution models and single-touch attribution models can be built using conventional AI algorithms. Some recommender systems around explore-exploit algorithms use conventional AI models. In terms of generative AI, there is a huge focus on using these algorithms for content creation, which could be creating product reviews, creating content on behalf of customers and taking their consent to post it, so that people can have a realistic opinion about what the product is and its key capabilities, and it can make the overall discovery and search process more efficient and useful. There are certain opportunities around using generative AI to enhance computer vision applications around drone deliveries and autonomous vehicles. So there are multiple other avenues as well, especially around content creation for, let’s say, smart creatives, wherein we are talking about creating images and assets for the organization that would otherwise take a lot of manual intervention and processes, which can be expedited as well. So there are multiple avenues and multiple applications that different organizations are thinking of in terms of using conventional AI as well as generative AI.

Andreas Welsch: Thanks for sharing those. I think it’s really important to keep in the back of our minds what those examples are and where opportunities are across the business, in addition to generating, summarizing, translating language. Now I also know that when we look at building AI products, like you had already said, we rely on data, data that’s derived from real world events. Maybe some of it is created in a lab as synthetic data. But the idea is that we have data that has been created or observed in the real world. And you already mentioned bias and biases as one example of data points in the data that represent the real world that we might not want to have in there, because we want to make it more inclusive. We want to add more diversity, more fairness in these kinds of things. So I’m wondering from a product leadership point of view, how does diversity play into building AI products? And by the way, is it just the technology that we need to focus on? Or is there more than data and technology to it?

Srujana Kaddevarmuth: There is definitely more to that, Andreas. Today we are seeing most of the government organizations and corporate organizations delegating the decision making power to these algorithms. And these decision making powers are the ones that are impacting people’s lives and livelihoods. There are some life altering decisions that we are delegating to the algorithms, like legal sentencing, job applications, loan approval, college admissions, and even the less important and less vital decisions around what to eat, what to wear, whom to make friends with on social media, where to travel, and whom to date are all being delegated to the algorithms. So when these algorithms are influencing different facets of our lives, transforming all the industries that we could think of and being consumed by different sections of the society, it only makes sense to have equal representation amongst the people who are developing these technologies, isn’t it? But we don’t see that happening, and there is a lot of gap in terms of ethnic gap, in terms of neurodiversity gap, as well as gender gap. As per Women in Tech statistics, nearly 80 percent of the technology related jobs are held by men, with only 20 percent being held by women, and only 26 percent of the overall computing related jobs are held by women. And this situation has further exacerbated and the gap has widened. It’s really important for us to think about how we can make sure that the workforce is expanded, especially because of the caregiving responsibilities that women had to take during the pandemic. Many people left the workforce, around 750,000 women in the United States left the workforce during the pandemic. And the majority of them were in their prime working age of between 33 and 45 years. So it’s really important for us to think about bringing in more diversity in the domain. And very recently, McKinsey published an article which indicated that corporate America is at the crossroads and the decisions that the corporate leadership makes today will have implications on diversity for generations to come. And this is going to be reflected in the technology space, because technology is defining how a lot of things shape up today. While the opportunities are numerous, I believe the future of women in tech or the future of diverse ecosystems in the technology space is primarily going to depend upon the tech industry’s ability to attract more girls to pursue STEM education and then enter STEM careers, and to create a holistic environment for them to thrive, not only in the corporate sector, but also in the development sector, to create a greater socio-economic impact.

Andreas Welsch: Thank you for sharing that. I couldn’t agree more. It starts with providing these opportunities. It starts with creating and finding those opportunities and also starting at an early age, like you mentioned, because only then, and if you have role models that you can look up to and that you see in these roles today, you see someone that you want to become. So thank you for sharing that. Now we’re getting close to the end of the show and I was wondering if you can summarize the key three takeaways for our audience today.

Srujana Kaddevarmuth: Yeah. Because there’s a humongous amount of data that is getting generated, the first key point that I would want the audience to take away from this is understanding that data democratization and AI productization are here to stay, and most of the large organizations are going to focus on that. So it’s really important for us to bring in that kind of a product mindset when building and deploying AI algorithms. The second key takeaway is that there are a lot of risks associated with productizing AI, so doing that in a responsible and conscious manner is going to be super important. And the same holds true with generative AI algorithmic deployment. Not just using generative AI algorithms for the sake of it, but being mindful of the use case and using the right algorithms for the right use cases is going to be really important. It will save time and money for the enterprise as well as make an individual successful in the corporate sector. The third aspect is that we need to celebrate and intentionally build diversity and inculcate that within the organization, because diversity helps us build products that can be useful and acceptable to the different sections of society, as well as it helps foster innovation within the organization. So these are the three takeaways that I want the audience to take from this particular conversation.

Andreas Welsch: Wonderful. Thank you so much for sharing. Also, Srujana, thank you so much for joining us today and for sharing your experience with us. It’s been a pleasure having you on the show.

Srujana Kaddevarmuth: Same here. I totally enjoyed this conversation, Andreas. Thank you for having me here.

Productizing AI is a journey of translating the insights generated from exploratory analysis into scalable models that can power products, according to Srujana Kaddevarmuth. It means developing and deploying algorithms in the production environment efficiently and effectively, instead of building a separate model for each use case.

Srujana names three top benefits: human capital efficiency, which frees scarce AI experts from repetitive tasks; innovation, because freed bandwidth lets them run experiments; and standardization of technology capabilities across geographies and channels, which makes them easier to build and monitor responsibly.

Srujana names three risks: lost leadership credibility and resource investment after failed projects, production systems that degrade without a constant data feed, and legal and reputational damage from institutionalized bias. She recommends model flexibility, attention to computational efficiency and runtime beyond accuracy, and strong model wrappers.

Per Srujana, a model wrapper connects data feeds in and out of a model. Inputs come from first-party and third-party data sources. Outputs feed a feedback loop so the algorithms keep learning, with an inferencing layer and a visualization layer. It keeps the data flow current and efficient across the ecosystem.

Srujana recommends generative AI for use cases that need deep language, intent and context understanding, and conventional AI for regression, classification and optimization problems. Generative AI carries compute, infrastructure and energy costs, so she suggests first asking whether a use case actually qualifies for it.