--- title: "How to Use Multimodal RAG to Extract Text, Images, & Tables (with Demos)" source_url: https://youtu.be/jDFpEnJeSVg captured_at: 2025-05-18 author: "AWS Developers" contributor: "ชิดชนก" platform: youtube --- # How to Use Multimodal RAG to Extract Text, Images, & Tables (with Demos) youtube video by **AWS Developers**, 11m38s, uploaded 2024-12-11. URL: https://youtu.be/jDFpEnJeSVg Tags: aws developers, technical tutorials, github, amazon web services, developer, hands-on, cloud computing, data analyst, software development, application development, full stack, cloud community, computer science, educational, amazon cloud ## Shared in LINE Note+ - 2025-05-18 09:59 by ชิดชนก ## Description In this video, you'll learn how to use Multimodal RAG (Retrieval Augmented Generation) to extract information from documents containing text, images, and tables. First, we'll extract these different data modalities using Python libraries like PyMuPDF. Then, we'll create embeddings for the extracted data using the Titan model from Amazon Bedrock. After storing the embeddings in a vector database, we can use a language model to retrieve relevant information and generate responses to queries. You'll see examples of asking questions related to text, images, and table data, showcasing Multimodal RAG's capability to handle multimodal inputs intelligently. Whether your data contains just text or a mix of modalities, this technique enables effective information retrieval and question answering. 🛠️ GitHub: Follow AWS Developers! 📺 Instagram: 🆇 X: 💼 LinkedIn: 👾 Twitch: Follow Suman! 💼 LinkedIn: 00:00 Intro 00:36 Multimodal RAG with Amazon Bedrock demo 11:23 Learn more #MultimodalAI #rag #amazonbedrock ## Transcript (captions:manual:en-US) Hello, everyone. Are you aware of RAG? If yes, it's highly likely that you have been able to extract text from your vector database. But in the real world, most of the time, our data contains not only text, but also images, tables, and so on. Hi, I'm Suman Debnath. I'm a Principal Developer Advocate for AI and machine learning at AWS. And today I'm gonna show you how you can use RAG, which has the capability to extract images, text, and tabular data from your database using multimodal RAG. All right, so let's jump into the demo. So I have already created a Jupyter Notebook, and I'm running it locally from my laptop. And if you see here, when we talk about multimodal RAG, our text, which is on the left-hand side, as you can see here, the document, it can contain PDF, images, and tables. So for this example, we will consider that this document contains images, tables, and text. And our first task will be to extract this information from the data. And we can use different frameworks. We are going to use one framework called PyMU. But we can use any other framework like unstructured. Once that data is there, where we have extracted images, tables and text, we are going to use a multimodal model to create the embeddings. And once that is done, we can store all the information in a vector database or any index. And once that is over, we are ready for retrieval. So this is option number one, and there are a couple of other options available. In this demo, I'm going to show you option number one. But there are some alternate options, option number two and option number three. And here what we are doing is even more interesting. Here, what we can do is after you take the images, tables, and text data from the document, we can create a summary. So we can use any model to create the summary of the information and then store them in the vector database. So in this vector database, we are not storing the raw data, raw table, and raw images. What we are storing is the summary. And this is going to be the main factor or the main difference between the first option and this option. So in this case, we are going to ask question, and we are going to get the retrieved information in the form of summary. And once we know the summary, we know that summary is coming from what type of data because we have stored that in a hash table where we have images, tables, and the raw text, with its ID and the summary. But we are not going to talk about this option two and option three. We are gonna talk about option number one. So let's get started. So at first, we have to import a few of the libraries that we are going to use to process the data. And once that is done, we have to load the data. So we have taken the PDF of the research paper, which is one of the famous research paper of our time, "Attention Is All You Need," which has actually changed the dynamics of how we develop software in today's world. So this is the research paper for transformer. And we are going to use and load this data. And the reason that we are taking this data is because if I scroll down, you will see that this data contains few images. This also contains few tables like this one, and so on, right? What we are do first is we are going to extract this table data, text data, and image data from this PDF. So for this, as I was mentioning a while back, we are going to use a Python framework called PyMuPDF. And then we are going to define different folders. And these are the folders where we are going to store our process data. So we need to have the text data separate, table data separate, and the image data separate. So this is a small function which we are using to extract the table data using tabula. And then we are here extracting the text information using LangChains, our text_splitter. And lastly, we are extracting the images from this PDF file. And if you can see here, we are reading all the images from each of the pages from this PDF. All right, so once we have all this information, what we are gonna do is we are going to store all of this in a local list, local element called elements. And we are going to add all this information in this list because at the end of the day, what we're gonna do is we are gonna take that list which would contain text, PDF, text, images, and tabular data, and we will store it in a vector database. So now that our data is processed, if you want to see how the data looks like, we can say that the type is text. If the type is text, how the data would look like? Because this is how we have stored the information. So this is the first chunk of the PDF. And it shows the text information and the path. And the type is text. Similarly, if you can see a few of the tables, and we can see that the type is table and the data is in this format. And a path of this raw data is here. And if you remember, we have shown that we have created four different folders for text, tables, and images. All of this, you can see here. So if I just click on this one, you can see the images. So if you see this, this is the image that got extracted from page number two. This is the image which got extracted from page number three, and so on. So this is what we have done so far. So we have extracted the information from the PDF, and we have stored it in our local list. So let's go back to our notebook. All right, so now that we have our data, the next task would be to create the embeddings. And here we are going to use Bedrock, and we are going to use Titan model for creating the embeddings. So if you see here, we are using Bedrock, we are creating a Bedrock client, and using the Titan embedding model. And since we have different types of data, we need to make sure that we pass the right type of data so that the Titan model can embed both image as well as the text. So this small function, or this five lines of code, actually creates those embedding for us and storing it again back to my lists. So once everything is done, you can see that in the same items dictionary or the list, I have now the text as well as the embedding, right? So this is the embedding that we are adding. So this is just the first element. If I just run for the third element, you will see the third chunk of the text, right? So now we are done with creating the embeddings, and we have also extracted all different types of data. The last thing that we need to do is store that in a vector database or index it. In this case, I'm using faiss. But you can always use any vector database of your choice. At AWS, you can use OpenSearch or Aurora. All of these databases supports vector database. So once we store all these embeddings, in this case, we have created this index, we are ready to ask questions. And for this, for this retrieval, we are going to use Claude 3. You can always use Claude 3.5 or any model of your choice. And we are going to use Amazon Bedrock to use this large language model. And since we have different types of data, we need to make sure that we get context from different types. So in this function, what we are gonna do is we are going to ask the question, and this is the prompt template that we are using. We are passing the extracted retrieved information from the vector database and the actual question. So if we now try to test this pipeline, we can ask one question. So let's say, how is the scaled-dot-product attention is calculated? So this is a formula which is there in the PDF file. So now, if you see, the first thing that we need to do is we need to find out the relevant chunks for this. And that is what we are doing here by calling this generate_multimodal_embedding function. So for this, we have to first create the embedding of this particular query, and then try to find out …