---
title: "True Multimodal RAG - Audio/Image/Video/Text"
source_url: https://youtu.be/qCAvqsBbN2Y
captured_at: 2025-05-22
author: "Adam Lucek"
contributor: "ชิดชนก"
platform: youtube
---
# True Multimodal RAG - Audio/Image/Video/Text

youtube video by **Adam Lucek**, 21m40s, uploaded 2024-07-15. URL: https://youtu.be/qCAvqsBbN2Y
Tags: artificial intelligence, OpenAI, AI, Gemini, Mistral, Llama, DALLE, Open Source, HuggingFace, Machine Learning, Deep Learning, AI Trends, AI Innovations, AI Applications, AI for Beginners, AI Tutorial, AI Research, AI Solutions, AI Projects, AI Software, AI Algorithms, Artificial General Intelligence, AGI, AI Skills, AI Strategy, AI Integration, AI Development, ElevenLabs, AssemblyAI, Multimodal, 

## Shared in LINE Note+
- 2025-05-22 22:09 by ชิดชนก

## Description
Everyone knows general text based vector databases, and text based RAG for LLM applications, but as it turns out thats just the beginning! Taking advantage of CLIP & CLAP models along with some fancy tricks, we embed 25,000 text entries, 1999 pictures, 2000 audio files, and 99 videos into a single vector database, allowing us to run direct text to text/audio/image/video retrieval!

Resources:
Multimodal Image RAG Video: 
Code: 
Colab Notebook: 

Chapters:
00:00 - Intro
01:04 - CLIP Model Review
02:08 - CLAP Model Overview
02:35 - Modality 1: Audio Setup & Dataset 
03:45 - Modality 1: Custom Audio Embedding & Loader Functions
05:40 - Modality 1: Audio Embedding & Testing Retrieval
07:38 - Modality 2: Image Setup & Dataset
08:52 - Modality 2: Image Embedding & Testing Retrieval
09:46 - Modality 3: Text Setup & Dataset
10:24 - Modality 3: Text Embedding 
12:22 - Modality 3: Testing Text Retrieval
13:06 - Modality 4: Video Setup & Methodology
15:06 - Modality 4: Video Dataset & Embedding
16:22 - Modality 4: Testing Video Retrieval
17:10 - Full Multimodal Retrieval!
18:34 - RAG: Setup
19:26 - RAG: Prompt Setup
20:25 - RAG: Full Multimodal Retrieval Augmented Generation
21:15 - Outro

#a

## Transcript (captions:auto:en-orig)
hello everybody Adam LK here and today we're going to be setting up truly multimodal rag a vector database that can store images audio video and text all at once that will allow us to search through all four modalities with just a text query and run Vector similarity search between text in audio text in video text in images and text in text so ever since my last video where we checked out chroma DB's multimodal integration with open clip embedding functions allowing us to use the open clip model to embed both images and text the same way which allows us to generate embeddings for images store them in a vector database and then run similarity searches from the same embedding function over text allowing us to do direct text to image querying without the use of any metadata or alt text querying that would rely on actual texttext retrieval I've been interested in how to actually apply this idea to the other two modalities that are lacking video and audio and so when I came across this the contrast of language Audio pre-training model I knew I had to put it to the test because this would allow us to do what the clip model was able to do with images for audio so just to give a quick refresher on what the clip model is before we show how it's applied to audio the clip model originally came out of open ai's research and it's essentially their pre-training of an image encoder and a text encoder to predict which images were paired with which texts in their data set of 32,768 randomly sampled text image pairs what this essentially allows it to do then is do direct zero shot classification of images where it can look at an image embedding and the text embedding and compare how similar the two are without ever having seen the text and without ever having seen the image before and its data set in our prior example when we were looking at clip embeddings we applied this to a bunch of fashion photos where we embedded a thousand fashion photos and then were're able to do plain text queries using the clip embeddings over those photos to directly retrieve the photos so we can see here that a query of black skirt returned five photos of models in black skirts and this is great because it allows us to use the same embedding function the model to embed all of the images into our Vector database and then also be able to embed the text to run the similarity search using the same embedding function hopping back to the clap model La essentially took all of the same ideas of the clip model and images but now applying them to audio so using their contrastive language audio pre-training on an embedding model we're now able to do exactly what clip was able to do with images using clap and audio and so this means using the clap model we can perform zero shot audio classification using the same embedding function to generate the embeddings for the audio files and then also for the text that we'll be searching the audio files over so let's see how that's all set up we're going to be using chrom ADB again since they have this multimodal capability already built in and then using and taking advantage of some of its open source nature we're going to be plugging in a few of our own features into chrom ADB to then support the clap model for audio retrieval the data set that we'll be embedding and using for this example is going to be the ESC 50 data set which contains 2,000 different environmental sounds and should be perfect for this demonstration ation because I really wanted to show how it's able to do the sound classification that doesn't rely solely on the transcription when doing a little bit of the pre- resarch for this I saw that a lot of the embeddings that claim to be able to embed audio really just ran it through something like the whisper model to retrieve the text and then embedded the text for textto text retrieval that then path back to the audio file this is not what I wanted to do because I wanted to do direct text to audio retrieval similar to what we did in clip so using this data set this will be a perfect example we're just going to be in that through hugging face and then saving all of the files locally and the reason we're saving these files locally is because chrom ADB doesn't actually store all of these files within its Vector database it will just store the embeddings and then the path on your file directory to those files and so if you look at chroma DB's multimodal documentation you can see that what you need to add a collection with multimodel capabilities is a name an embedding function and a data loader the data loader goes back to what I was saying about chroma DB not actually storing the file in the database itself but it'll actually use a data loader to path over to the data and then load it when need be when it's retrieved or during the embedding process the embedding function is then of course going to be the function that will be able to take both the text and the modality of choice and be able to generate the embeddings from it which in its built-in functionality is the clip embedding function so what we need to do is create a custom data loader for audio and a custom embedding function for audio and so starting with with the data loader what I have here is the custom audio loader built out essentially what it'll do is it'll take in an audio file do a little bit of pre-processing to ensure that everything's in a standardized format and then return it in a form that the embedding function can actually handle and so what it'll do is it'll convert it to 48 Kilz if it's not already convert it to mono and then return from the audio file a waveform tensor from pytorch the waveform tensor is then going to be passed into our clap embedding function which we have here and the clap embedding function takes in a few different things it can either take in the waveform or it can take in a text document and this is how we're going to be able to use the same embedding function to embed both the audio and the text that we want to run the similarity search between the text and the audio from and so we path over to the larger clap General model to load it and then within this class there are two encoding functions we check if it's an audio tensor if it is then it'll process that and return the audio embedding but if it's text it'll just process the text and return the text embedding and so using the same embedding function with the larger clap General model this is how we're going to be able to generate effective embeddings that will allow us to run the similarity search between audio and audio or text in audio in the retrieval step later and so then setting up our audio collection is as simple as using the get or create collection method from our chroma DB client passing in a name which I've said audio collection and then we can pass in our embedding function and our data loader which are our custom clap embedding function and our custom audio loader here and so then after we've added all of the files to this we can test it foret retrieval so I have a quick testing function right here that'll do just that essentially all we need to do is path over to our collection and then use the query method where we can then pass in texts how many results we want to return and then including the uis and the distances it will always return Five results so I wanted the distances so that we can do a little bit of distance-based filtering and then the uis are necessary because that will be the paths to the original audio files so that we can actually use them to display so let's try this out by searching for water droplet with a max distance of 1 .5 over our ESC 50 audio database that's returned five audio files here which they don't have great names like 188 293 so let's listen to them and see if it sounds like water droplets sounds like water droplets that's great so using the clap model we were able to embed all of the audio files into our Vector database and then because we're able to use the …