---
title: "Turn ANY Website into LLM Knowledge in SECONDS"
source_url: https://youtu.be/JWfNLF_g_V0
captured_at: 2025-10-26
author: "Cole Medin"
contributor: "ชิดชนก"
platform: youtube
---
# Turn ANY Website into LLM Knowledge in SECONDS

youtube video by **Cole Medin**, 18m44s, uploaded 2025-01-13. URL: https://youtu.be/JWfNLF_g_V0
Tags: ai, artificial intelligence, ai agents, software engineering, software development, coding, automation, saas, development, crawl4ai, web scraping, web crawling, ai web scraping, ai web crawler, rag, ai rag, llm rag, open source ai, open source llm, website rag, llm with website data, playwright, playwright ai, rag ai agent, pydantic, pydantic ai, pydantic ai agent, pydantic ai rag, rag web crawler

## Shared in LINE Note+
- 2025-10-26 10:20 by ชิดชนก

## Description
One of the biggest challenges we face with LLMs is their knowledge is too general and limited for anything new. That’s why RAG is such a huge topic when it comes to AI right now - it’s a method for providing an LLM with external knowledge you curate so it can become an expert at something it wasn’t before - a specific AI framework, your ecommerce store, you name it. The problem is, that “curate” step can be very difficult and slow.

That is where Crawl4AI comes in! Crawl4AI is an open source web crawling framework specifically designed for scraping websites and formatting the output in the BEST possible way for an LLM to understand. The best part is it solves a LOT of problems we typically have with systems that crawl websites - usually they are slow, resource intensive, and complicated. But Crawl4AI is VERY fast, intuitive, easy to set up, and extremely memory efficient.

In this video, I show you how to use Crawl4AI to super easily crawl websites for LLMs in just seconds, and at the end I even show you a RAG AI agent I’ve built to be a “Pydantic AI” framework expert using Crawl4AI to build the knowledgebase. And you could really take this and use it for any website you want. Next

## Transcript (captions:auto:en-orig)
one of the biggest challenges we face with large language models right now is their knowledge is too General and it's very limited for new things because of its training cut off for example my favorite new shiny framework for building AI agents is pantic AI but if I go over to Claude right now and ask it what pantic AI is it has no clue and even if I use an llm that can search the web the information that I get back is still going to be very Bare Bones but on the other hand if I take all of the framework documentation for pantic AI and put it in a knowledge base for the llm and I ask it the exact same question the answer that I get is spot on that is why rag is such a huge topic when it comes to AI right now which that by the way stands for retrieval augmented generation it's a method for giving external knowledge that you curate yourself into an llm basically to make it an expert at something that it wasn't before like an AI agent framework you're using your e-commerce store you name it the problem is that Cur R step can be very difficult and slow for example if you want to ingest an entire website into a knowledge base for your llm how do you actually do that and get it done fast so that it's not 2027 by the time your knowledge base is ready and AI has taken over the world anyway that is where crawl for AI comes in crawl for AI is an open-source web crawling framework specifically designed to scrape websites and format the output in the best way for llms to understand and the best part is it's it solves a lot of problems that we typically see with these systems for website scraping usually they're very slow overly complicated and super resource intensive but crawl for AI is the complete opposite it's super intuitive very very fast easy to set up and extremely memory efficient so in this video I'll show you how to super easily use crawl for AI to scrape any website for an llm in just seconds and at the end of this video I'll even quickly showcase a rag AI agent that I built basically to be an expert at the pantic AI framework of course using crawl for AI to curate all the framework knowledge into my knowledge base and really you could take what I built and use it for any website super exciting stuff so let's go ahead and dive right into it so the two big things that we're going to focus on right now is crawl for AI which this is their GitHub right here completely open source and then also pantic AI now this is not a pantic AI video but it's just a very good example of full documentation that we can scrape with crawl 4 Ai and bring into a knowledge base for our llm now back over to crawl for AI the first obvious question when we visit this page is what is the point of crawl for AI why even use it and luckily at the top of their readme here on their homepage they have a very concise description for why you should care about crawl for AI and the first big thing is when you visit a website and you extract the raw HTML from it I mean this looks like a mess and as a human looking at this it's very hard for us to actually extract useful information from it and a good general rule of thumb is if it's hard for us as a human to understand something it's probably harder for a large language model as well and so one of the most important things that crawl for AI does is it takes this ugly HTML that we get when we visit the raw content of a web page and it turns it into markdown form which is actually even human readable format like a super clean way to represent a web page and all the information that we see when we typically visit a UI like this just in something that's all text space so we can give it to a large language model for reg and just for llms to understand it better in general so that's the first thing and it does it very very efficiently it's super super fast it handles a ton of things under the hood like proxies and session management things that are not easy to handle so it's not like you can go and make your own version of crawl for AI very easily as well just because you know how to use the request module in Python to pull HTML from websites and it's also completely open- source and very easy to deploy they even have a Docker option which I'll probably cover in another video on my channel later on they're doing major updates soon to their Docker deployment so I don't want to focus on that now but just know that that is available as well and there's a ton of other things that they do that are super valuable as well like one really important thing is removing irrelevant content I mean when we go to the h ml here we have a ton of script tags and there's probably a lot of information um that we can view on the page that is very obviously redundant or just not useful to us and that they take care of removing that as well So eventually what we get back from scraping the site with crawl for AI contains just what we care about actually ingesting into our knowledge base and getting started with crawl for AI is so easy all you have to do is PIP install the python package and then run this setup command which is going to install playright under the hood that is the tool that crawl for AI uses under the hood to scrape websites and basically have this browser running in the background that can visit these sites and playright is a fantastic also open source tool that I use for a lot of testing for my web applications as well so very very familiar with it I can recommend this as well so I think it's a great choice for actually having that web scraping functionality under the hood and getting started is so so easy it's just as simple as this little script right here and so I actually have my own version of it if I go into my source control here I've got my own version of it just scraping the homepage of pantic AI their documentation so let's go ahead and test this out and see what kind of output we get and by the way all of the code that I go over in this video I'll have in a GitHub repository that I will link below and so right now we're starting with just these basic examples right here starting with this one to pull the homage of the documentation for pantic AI and then we'll get to my rag AI agent later and I'll show showcase that a little bit as well and so yeah I've already installed crawl for AI it took like 30 seconds to install everything including play right under the hood and so now I can just go ahead and run our script and within seconds we're going to have the entire page printed out in the terminal here so so fast very very impressed with this and this isn't like the perfect format for us as humans to understand cuz we have all this little markdown syntax and stuff but this is definitely a great format for an llm to understand this entire page especially compared to what we get if I just go back to the pantic documentation here I'll just open this back up if I inspect the page Source this is the HTML that we got the markdown from and imagine pasting all of this into an llm prompt I mean it's just a mess it's definitely going to hallucinate when you try to ask a question with all of the HTML tags and everything in there and so it's just so much better when we have something that looks like this all right so we saw a basic example using for AI to get the markdown for a single file but obviously we have to take this a lot further to do something that is actually useful for our llms now the first thing that we want to do is make it possible to ingest every single one of the pages in the pantic AI documentation so you want to get the markdown for the introduction page installation getting help contributing …