Scrapy, a fast high-level web crawling & scraping framework for Python.
-
Updated
Aug 28, 2024 - Python
Scrapy, a fast high-level web crawling & scraping framework for Python.
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Python scraper based on AI
Pythonic HTML Parsing for Humans™
🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.
LinkedIn_AIHawk is a tool that automates the jobs application process on LinkedIn. Utilizing artificial intelligence, it enables users to apply for multiple job offers in an automated and personalized way.
Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)
List of libraries, tools and APIs for web scraping and data processing.
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
Declarative web scraping
Collection of useful data science topics along with articles, videos, and code
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
Up-to-date simple useragent faker with real world database
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
Add a description, image, and links to the scraping topic page so that developers can more easily learn about it.
To associate your repository with the scraping topic, visit your repo's landing page and select "manage topics."