How can a knowledge graph improve your RAG app?

Andreas Welsch: Today we’ll talk about how you can supercharge your RAG app with a knowledge graph. And who better to talk than someone who’s doing just that, Anthony Alcaraz. Anthony, thank you so much for joining.

Anthony Alcaraz: Thank you. Thank you for having me.

Andreas Welsch: Wonderful. Why don’t you tell our audience a little bit about yourself, who you are and what you do?

Anthony Alcaraz: Yes. Currently I am a Chief AI Officer. What we do is that we automate CV screening and job matching for human resources processes. So we are a startup, we are raising funds right now. And I work with an engineering team to leverage AI for human resources. I have a somewhat surprising career, because I started as a CFO, so Chief Financial Officer, and I fell in love with data. Since day one, I coded my first Python program, I couldn’t stop. That’s why I’m here.

Andreas Welsch: Excellent. Thank you so much for being on the show. I’m really looking forward to our conversation and what you’ll share. What do you say, play a little game to kick things off?

Anthony Alcaraz: Why not?

Andreas Welsch: Okay, perfect. All right. This game is called In Your Own Words, and when I hit the buzzer, the wheels will start spinning. When they stop, I’d like for you to answer with the first thing that comes to mind and why, in your own words. Are you ready for What’s the BUZZ?

Anthony Alcaraz: Let’s go.

Andreas Welsch: Alright. Awesome. So here we go. If AI were a movie, what would it be? 60 seconds on the clock. Go!

Anthony Alcaraz: I really like, because I am a Tarantino guy, I don’t know why, but I thought about Pulp Fiction. It’s creative. You never know what will happen in the scene. So it’s generative. It makes you think. It’s funny. It’s fun, and I hope it’s useful. I don’t know if Pulp Fiction was useful, but yes, I don’t know why I think about this topic.

Andreas Welsch: And I think it can get pretty messy sometimes as well. Awesome. You did good. I love the answer. Fantastic. Now, look, if we talk about more of the subject of our episode today, and I think some of the AI can get pretty messy too, especially if you look into some of the technology and there’s so much terminology being thrown around these days. One of the ones that we’ve seen emerge in the last couple of months is RAG, retrieval augmented generation, to get better results, more tailored results. But then I also see that it has its shortcomings. It’s supposed to help you with hallucinations and generic output, make that more specific, reduce hallucinations. But where do you see RAG fall short, for example?

Anthony Alcaraz: Yes. So RAG, what it is. We currently have models, large language models, that can generate an answer from examples. With RAG, you bring the model the right information to generate an answer. To make it simple. And you can bring this information in many ways. Where it falls short right now is that in some systems, for example with vector similarity search, a vector database, the information that you give to the model can get messy. You can get maybe misleading information. It’s very hard to bring the right information at the right moment to the model. So what you can get is a model that is confused about what you give it. One of the stakes is to control the food that you will serve to the model. And this talk will be about that. Maybe giving the best food that varies within your business data.

Andreas Welsch: We talked a lot about data quality, fresh development data. Data is proprietary. But then if you look at knowledge graph, that’s the main topic of the episode today, maybe for folks in the audience that are not that familiar with the topic yet, what does it help you with when you have all other sources?

Anthony Alcaraz: Yes, I have a definition. A knowledge graph is an interlinked set of facts that describe real-world entities and their relationships in a machine-readable format. What it does is that it provides organizing principles over your data to add structure and context to support your knowledge discovery and reasoning. To make it simple, it’s based on a graph structure with nodes that represent entities, for example a chicken, connected with a relationship edge. A chicken is part of a farm. This is a triplet in a knowledge graph. You can have billions of such relationships, and all these relationships are connected in some way. So you can apply mathematical and graph algorithms within this structure. What we found recently is that applying this structure of data is good for the accuracy, transparency and explainability of retrieval augmented generation systems.

Andreas Welsch: Speaking of knowledge graphs, I’m reminded of things like social networks. Many years ago there was this statement that within two or three hops you’re connected with the entire world or something like that. That reminds me of that: entities, relationships, and then the closeness or distance between them.

Anthony Alcaraz: Pretty interesting, because currently we are on LinkedIn. Behind the hood within LinkedIn, you have a huge knowledge graph. Most social media are built on a knowledge graph database, on which AI is trained.

Andreas Welsch: Obviously social media is one of the examples, but what are maybe some other examples of using knowledge graphs in a business?

Anthony Alcaraz: This kind of relationship between your data can be found in every field. Recently, I had a call with a friend that works on biology data, where they are building a knowledge graph. In my company, we use a knowledge graph for human resources, with skills, job, occupation, et cetera. For every data that there is in the world you have an ontology, so relationship rules. Because behind your businesses there are rules, there are relationships between your clients, between your suppliers, et cetera. Every data can be translated into a knowledge graph. That’s the beauty of it.

Andreas Welsch: Building on the understanding of what a knowledge graph is and what it can help with to some extent, how do you build it? Do you need a large amount of data? Can you do it with a little bit of data? Do you need to be a deep expert in data science? Do you need to have a PhD or can anybody do it? How do you build a knowledge graph?

Anthony Alcaraz: You have a specialty, knowledge graph engineers. You have languages to build knowledge graphs. For Neo4j, for example, they are called Cypher queries. You have SPARQL queries. So you have a coding language. Before LLMs, it was technical. You needed to know the programming language to build it. You needed the people skills. Now it’s still the case. You need people that know what we are doing. But something interesting is that LLMs can help build up a knowledge graph. How do you build it? You start from your data. 80% of your data is unstructured. It comes from your emails, from your reports, from all the stuff that you have written and that you use in your everyday business. This can be transformed into a knowledge graph. It can also be your relational database, numbers that you have in your database. This can also be translated into a knowledge graph. Right now you can empower your knowledge graph engineer with the capabilities of LLMs. It helps speed up the process and reduce cost, because from text you can use LLMs with many techniques to extract the necessary information to build your knowledge graph.

Andreas Welsch: So how does it work together with RAG then? For example, if you upload documents or if you have data in your own data structure already, how does a concept like a knowledge graph enhance retrieval augmented generation and get better results?

Anthony Alcaraz: For a start, I mentioned the stakes of controlling the information that you feed to your artificial intelligence. With a knowledge graph, you have a huge control of what is feeding the LLMs. For example, you can apply mathematical operations on your knowledge graph. I will give you an example. You can look for a node for a chicken. This species of chicken is part of how many farms? This is a mathematical operation. You can do that to feed your LLM, and the LLM will have this kind of information to answer your customer that eats chickens. This is one example. The second example, and the most interesting one, is that you can train other artificial intelligence models with data that is present in your knowledge graph. For example, you can train models that look at the most important nodes in your knowledge graph. In social media there is a phenomenon where a few accounts get most of the likes, most of the comments. It’s called a power law. You can have graph algorithms that automatically detect this kind of node, the important influencer. Maybe for your marketing campaign, I want to know who I will pay for my marketing, and I will look at the most influential nodes. All those relationships and mathematical operations can help the LLM think, and it will reduce considerably, by a huge percentage, the amount of accuracy, and you can build up a system where the LLM actually reasons on your data.

Andreas Welsch: That’s exciting, because a lot of times when we hear about large language models and some of their shortcomings and deficiencies, on one hand it’s their huge understanding of language, or at least being able to predict the next word in a sentence, which is a hotly debated topic. But if you can feed it more relevant information and context about either your business or the business function that you’re trying to solve, then you use the LLM for natural language understanding and generation, but what it actually generates comes from the knowledge graph.

Anthony Alcaraz: Exactly. I think the most important feature right now for LLMs, the real revolution behind LLMs, is what we call in-context learning. They can leverage something that they didn’t see in their training, and they can reason about this new context to give a personalized answer, for example. But to leverage this in-context learning, you need to feed it the proper data, the right data, and the format of the data will be important. I have another point I want to make. Over time, we will have more and more powerful models. For example, yesterday Anthropic released a model that is called Claude 3, with three different models. We will have more and more powerful models. The only key differentiator for companies will not be the model that they use. It will not be the AI system that we build. It will be their data, and the quality of their data. If you compare each database that exists currently, the most qualitative data that you can have are knowledge graphs, by their interconnected nature and their curated nature when they are built correctly. So I think it’s a strategic investment, a strategic asset, to have this kind of data to leverage the future AI models that will appear.

Andreas Welsch: I think that’s an important point, because every company has a good amount of data about their business, about their products, about their customers, how they are doing business with each other, what products they’re buying and so on. So putting that in the machine then.

Anthony Alcaraz: Another point I wanted to make. We think most of the time of LLMs as chatbots. You have a chatbot, you ask a question and it answers. I am more interested in automation, in building, behind the hood, document understanding and document processing in an automated way. LLMs are just one part of the equation, but an important part. It needs to be built within a system that leverages proper data. You can create a lot of value from this, when you automate, when you don’t have many errors, et cetera.

Andreas Welsch: I think that is really powerful together with the data that you mentioned. If 80 percent of our data is stored in documents and so on, on one hand being able to extract it, but then as a next step put it into a relationship with some other data, maybe data you have in your database or other instructions, and then reason over it. I’m taking a look at the chat here real quick, and I think there are two interesting questions. One is about the originators of that approach of using knowledge graphs for better semantic understanding by LLMs. And the second one is, Claude 3 shows some advancements in ontological understanding of finance concepts. Do you know if Anthropic used knowledge graphs to augment the LLM to achieve this? I don’t know how much has been published. I haven’t gone through all the news in the last 24 hours yet myself. But maybe you have some insights.

Anthony Alcaraz: The originators of the approach, I don’t know precisely. There is a paper that is called a roadmap about unifying LLMs and knowledge graphs. It’s from February 23, I believe, something like that. There are many people that are studying the fusion of both approaches. Both approaches come from different fields, because the knowledge graph is a field in curriculum and NLP is another field. The fusion of both is something recent, I think. It comes from this idea of combining neuro and symbolic. Maybe this notion is too complicated, but it’s a neuro-symbolic approach. For Claude 3, I don’t know. But for the question about ontological understanding, it’s interesting, because what you can do is train your LLM, an open-source LLM, on a particular ontology. There are papers, for example OntoGPT. There are techniques to make sure that your LLM understands an ontology. It’s something you can use.

Andreas Welsch: Yeah. If you have questions for Anthony, please put them in the chat in a couple of minutes, but thank you for answering those on the spot. There are so many new papers on a daily basis, new announcements like the ones yesterday that already supersede the one from a couple of days ago. It’s mind-blowing, the pace.

Anthony Alcaraz: I believe we are living through some sort of technological jump. The amount of papers and the amount of techniques is challenging to follow, even in a narrow field. So saddle up. We are living, I think, in an information revolution and an accessibility-of-knowledge revolution, many revolutions at the same time. I wouldn’t be afraid, because I am optimistic. I think those technologies will empower people to learn better and in a more personalized way when they are built right. So I wouldn’t be so afraid of what is happening. I think it’s for the better. And I am a huge believer in open source.

Andreas Welsch: There’s a lot of advancement and development happening there for sure. And with open source, it definitely gives a good amount of transparency into what’s actually happening. Maybe one question then: we’ve talked so much about knowledge graphs, what they are, how they improve your large language model type applications and so on. But from your perspective, when should people even use a knowledge graph, say in a business? Is that something that you now need to know? Is that the next thing after RAG that everybody now needs to put in? Does it only make sense at a certain amount of data, a certain complexity of the problem? When should you even use a knowledge graph to begin with?

Anthony Alcaraz: It’s a tough question. If you have a customer or department that is linked with a finance department, et cetera, you need a relationship in your data. You need to connect the dots. A good friend of mine, Jeremy Ravenel, is working on this subject in his company. What you need to think is, it has been demonstrated that it improves the RAG system. Retrieval augmented generation is better with a knowledge graph. But like I said, investment in your data will help you train AI models that are performant. It will help you leverage generative AI with no hallucination or better accuracy, et cetera. So it is a strategic investment to get a proper database that fits your needs. And I think a knowledge graph can serve any use case, whether it’s a recommendation system, analytics or business intelligence. It’s a powerful paradigm in databases right now, and it’s more easily understood than a relational database, for example, or a vector database, because it’s intuitive. It’s the way our minds work. We connect concepts and relationships between concepts. So yes, but I am biased.

Andreas Welsch: I’m glad you are. It makes you the perfect show today. Now, when you look at large language models themselves and what they’re able to do, how are they becoming a source for creating a knowledge graph? And do you need more of an agent-type framework to even use a knowledge graph, or is that independent of that? What do you say?

Anthony Alcaraz: Okay, there are many approaches to leverage this, because your question is how to leverage the information that is in the knowledge graph with the LLM. Yes, you can have an agent system. Currently, one approach that I have, because you have many ways to interact with a knowledge graph, I won’t be technical, but there are currently three main ways. You can imagine each of those ways as a tool, some kind of a key, for the LLM to interact with and to leverage. Yes, an agent framework, so an autonomous LLM that thinks about what it’s doing and which key to use, can be leveraged. But you can also have a simple system where, for example, for a customer chatbot, you know what kind of questions will be asked, so you can constrain the LLM to this part of the graph. It’s very flexible in terms of what types of information you want to feed the LLM. There are many approaches. It will depend on your problem. You need to think first of your problem and find the solution. In every data science project this is the approach. If you start from the tool and go back to your problem, you will have problems.

Andreas Welsch: I love that. Thank you for re-emphasizing that. It all starts with a clear point of view. Awesome. We’re getting close to the end of the show, and I was wondering if you can summarize three takeaways for our audience before we wrap up.

Anthony Alcaraz: Yes. GenAI can create value for your business, but you need qualitative data, and I believe a knowledge graph is a huge asset to have. The second key takeaway is that GenAI is not the only thing. You need other AI tools and AI models, and you also need qualitative data to train those models. So it’s circular, it goes with what I just told you. The third point would be to plan for this. You need some kind of strategy and you need to start from your problems. You need to understand correctly your business needs, and build up from those business needs to see what the solution would be in a data science way. And whatever the technology that will happen in the future, you will need data and qualitative data. It should be your priority, a data-centric approach. It’s not a tool-centric approach, it’s not a model-centric approach, it’s a data-centric approach.

Andreas Welsch: It sounds like that need for good and clean data is not going away anytime soon.

Anthony Alcaraz: No, no, no, because one funny fact is that a model is trained at a point in time. If your business is a business that evolves a lot, no model will understand it without access to your data. You will need clean data.

Andreas Welsch: Awesome. Anthony, thank you so much for joining us today and for sharing your experience with us. And for those of you in the audience, thanks.

Anthony Alcaraz: Thank you. Bye bye.

Anthony says that in some systems, such as vector similarity search, the information given to the model can get messy or misleading. It is hard to bring the right information at the right moment, so the model can be confused about what it receives.

Anthony defines a knowledge graph as an interlinked set of facts that describe real-world entities and their relationships in a machine-readable format. Nodes represent entities, such as a chicken, and edges represent relationships, such as being part of a farm.

Anthony says a knowledge graph gives control over what feeds the language model. Mathematical operations and graph algorithms on the graph can supply the model with structured information and support accuracy, transparency and explainability.

Anthony says a knowledge graph fits when relationships between data matter, such as between customers, departments and suppliers. Anthony treats it as a strategic investment in data, and advises starting from the business problem rather than the tool.