--- title: "VideoRAG: ✈️Advanced Retrieval-Augmented Generation with Videos 📺" source_url: https://www.youtube.com/watch?v=LtcHVLkkxjk captured_at: 2025-10-19 author: "Prompt Engineer 48" contributor: "ชิดชนก" platform: youtube --- # VideoRAG: ✈️Advanced Retrieval-Augmented Generation with Videos 📺 youtube video by **Prompt Engineer 48**, 12m28s, uploaded 2025-01-19. URL: https://www.youtube.com/watch?v=LtcHVLkkxjk Tags: #AI, #RAG, #VideoRAG, #LVLM, #MachineLearning, #ArtificialIntelligence, #MultimodalAI, #VideoAI, #DeepLearning, #Innovation ## Shared in LINE Note+ - 2025-10-19 09:30 by ชิดชนก — nearby chat: บ้านบ้าน ## Description Discover the cutting-edge innovation of VideoRAG, a groundbreaking framework that redefines Retrieval-Augmented Generation by incorporating the rich, multimodal knowledge from videos. Unlike traditional RAG approaches that focus on textual or limited visual content, VideoRAG dynamically retrieves relevant videos based on query relevance and leverages both visual and textual information for more accurate and context-aware responses. Powered by the latest Large Video Language Models (LVLMs), this framework unlocks the full potential of videos to enhance factual accuracy and depth in AI-generated outputs. Watch now to explore how VideoRAG is pushing the boundaries of AI-powered knowledge generation! #AI #RAG #VideoRAG #LVLM #MachineLearning #ArtificialIntelligence #MultimodalAI #VideoAI #DeepLearning #Innovation Link: ------------------------------------------------ Learn More: Try Out Gloud GPUs on Novita AI (Affiliate Link): ------------------------------------------------- CHANNEL LINKS: 🕵️‍♀️ Join my Patreon for keeping up with the updates: ☕ Buy me a coffee: 📞 Get on a Call with me at $50 Calendly: 💀 GitHub Profile: 🔖 Twitter Profile: Other videos that you w ## Transcript (captions:auto:en-orig) so this new video is on video rag retrieval augmented generation over video corpus now we all know rag retrieval augmented generation we put in a query then it goes and get the retrieval query asking a database and we get back the retrieve text and then we construct the full prompt and we get the response note that we have retrieved text here so instead of text what this paper is trying to do is give in videos so retrieved videos versus retrieve text so this is the Innovation this is the tldr if you're interested let's go through this entire paper I'm going to go to the most important things that is necessary for putting up the argument now we all know rag applications now this is an example of a textual rag for example how to tie a tie after I cross the white end over the narrow end now this is a simple question by the user and after having received this question the retriever goes ahead and finds something so just a normal retriever which is a textual rag it goes to Wikipedia and draws out some content which is not that relevant and ultimately you get some answer like this which is not very relevant to the question that we have asked so this is an example of textual rag now conventional multimodal rag this is where you would get an image and some text as well so we get a retrieved image of a tie but that doesn't help to answer this question so how to tie a tie after I cross the white and over the narrow end so this question is not answered by this form of retrieval which is the photo and the text as well so for example the answer comes out like this neck ties are traditionally worn on top shirt button fastened and the tie not resting between the collar points doesn't help much now this is the Innovation that you can see here here we see that after we ask this question it goes ahead and get the video content for you it goes through the video it it understands the video and brings that particular video along with the subtitles if necessary okay and uses that video and the text if there are no subtitles then this audio is converted to video sorry if there are no subtitles the audio is converted to subtitles or text so using the video and the text we get this answer so the answer is W one wide end behind the narrow end bringing him back to the front the opposite side now you can see the importance of of and the benefit of using this video rag now let's see one more example before we go into this paper so this is the example of doing this so we have explain how to bake cookies on your card dashboard now generated answer is I'm sorry but this is not possible to bake cookies on your card dashboard the dashboard is not designed for cooking and it is not safe to be used as a heating source Additionally the fumes from baking could be harmful to your car Interiors so this is what the naive without any rag system would answer but when we have this video rag it explains like this explain how to bake cookies on your card dashboard it gets the video it gets the video it gets the text it gets the video so in this case it gets the video only because that's a video rag V we will see in just a second in just a minute maybe so it fetches the video content and using that video it generates the answer now if you look at this answer and compare with the ground truth you will see a lot of similarity between the generated answer and the ground truth which means it is very much useful to have video stored in the vector database and get those videos to answer the question so let's get started from the start again now retrieving external knowledge is very essential in the case of rag applications and we want to incorporate in the generation process rag is retri Belong by the generation largely based on textual information but there Overlook largely the video sources which are a rich source of multimodal knowledge capable of representing events processes and contextual details more efficiently than any other modalities so you have seen it is useful now we will convert the videos into textual descriptions as well and they are presenting forward video rag which is a noble framework that not only dynamically retrieves relevant videos based on their relevance with queries but also utilizes both the visual and the textual information of videos and the output generation that's interesting so this is an example of the multimodality richness of the video data for example consider this query how does the expression of the dog change when it is angry while texal descriptions might describe the dogs barking or growling the fail to capture the visual cues of bearing teeth raised heckles and narrowed eyes it's very difficult to describe with text which are needed for accurate interpreting the emotional state of the dog as well as formulating the answer to the query therefore we have this Innovation we're going to see the results but the method is that we are presented with the video rack which performs retrieval of query related videos over video Corpus and generate the response grounded in them this is pretty amazing now this they have described how the rag is performed I'm sure you know this so rag retrieves a set of documents or knowledge elements known as K which is the retrieve documents based on the query and this K is a subset of the external Corpus so this K is a retriever of the query from the Corpus C so once we have these K items these K will be used to form the the final reply so Y is this so so so final output is goes to the model uh and you can see that it depends on the query and it depends on the K the model is any llm that uses the K or the retrieve contents so this is the mathematical representation of the retrieval augmented generation now let's go to this part auxiliary text generation sometimes not every video in the Corpus comes with subtitles so they propose generating auxiliary textual data by extracting audio from the video and converting it to the text so both the text and the video will be good now the aser the data sets they're using two data sets first is the wiki how QA and then we we have the howto 100 million data set these are a comprehensive collection of instruction videos sourced from YouTube now they have tested with different models Baseline models such as naive here text rag then we have text rag here DPR and text rag R so video rag which is their contribution they have made three variants the first variant uses only text so video r t integ only the transcripts or auxiliary textual data obtained from the retrieve videos so we are talking about videos as well video rag so we're talking about the video Corpus but from that video Corpus if we just retrieve the textual format only that model will be called video rack T if it only utilizes the video frames then we say it video rack V and if it jointly utilizes everything we are calling this video rack VT so these are the three contributions and then we have the Oracle version as well video rag which is the ground truth so we are giving the actual answer that we expecting so this Oracle version is you know your dream version or the original version now let's go and see the performance of this so evaluating the performance we're using this different matrixes Rogue L play Bird score G well and you can see the results so that's pretty good on the x-axis on the on the rowwise you can see that we have different models here on the column wise we can see that different tests has been done and you can see that video rack V performs pretty good it's the best in all these cases and the second best is video rack VT we have this and here we have the second best is text video rag But ultimately we can say that video rag V performs good for all these cases so this Oracle version is our Dream version because it is based on the ground truth okay so we see that these are the results so when we use just the visual cues versus when you use the textual cues and when we use everything we use emble we …