A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG
markdown
html-to-markdown
markdown-parser
web-scraper
rag
html2md
llm
llmops
html-to-markdown-converter
markdown-scraper
markdown-crawler
md-crawler
-
Updated
Aug 13, 2024 - Python