Scrapy Cheatsheet Type: Fast, high-level web crawling and scraping framework for Python — used to build custom spiders that extract links, emails, subdomains, and other data from target sites
Installation # Via pip (recommended) pip3 install scrapy # Via apt sudo apt install python3-scrapy # Via conda conda install -c conda-forge scrapy # Verify scrapy version Basic Usage # Quick one-off crawl from the shell (no project needed) scrapy shell <url> scrapy shell https://example.com # Fetch a single page and dump to stdout scrapy fetch https://example.com # View the page as Scrapy sees it (opens in browser) scrapy view https://example.com # Run a standalone spider file scrapy runspider myspider.py Command-Line Tools Command Description scrapy startproject <name> Create a new project skeleton scrapy genspider <name> <domain> Generate a new spider from template scrapy crawl <spider> Run a spider inside a project scrapy runspider <file.py> Run a self-contained spider file scrapy shell <url> Interactive scraping shell (test selectors) scrapy fetch <url> Download a page using Scrapy’s downloader scrapy view <url> Open the fetched page in a browser scrapy parse <url> --spider=<name> Parse a URL with a spider’s callback scrapy list List available spiders in the project scrapy settings --get <KEY> Print a settings value scrapy bench Run a quick benchmark crawl Common crawl Flags Flag Description -o <file> Output scraped items to file (.json, .jsonl, .csv, .xml) -O <file> Same as -o but overwrites instead of appending -a <name>=<value> Pass an argument to the spider (e.g. -a domain=example.com) -s <KEY>=<value> Override a setting at runtime -L <level> Log level (DEBUG, INFO, WARNING, ERROR) --logfile <file> Write logs to a file --nolog Disable logging -t <format> Output format when not inferred from extension Common Commands # Create a project scrapy startproject recon # Generate a spider scoped to a domain cd recon scrapy genspider links example.com # Run the spider and export results scrapy crawl links -o results.json # Run a standalone spider with output scrapy runspider spider.py -o output.jsonl # Pass arguments into a spider scrapy crawl links -a domain=example.com -a depth=2 # Override settings on the fly (respect robots.txt off, set delay) scrapy crawl links -s ROBOTSTXT_OBEY=False -s DOWNLOAD_DELAY=1 # Limit log noise scrapy crawl links -L WARNING -o out.csv # Test CSS / XPath selectors interactively scrapy shell "https://example.com" Inside the Scrapy Shell # After: scrapy shell "https://example.com" response.url # Current URL response.status # HTTP status code response.headers # Response headers # CSS selectors response.css('a::attr(href)').getall() # All link hrefs response.css('title::text').get() # Page title # XPath selectors response.xpath('//a/@href').getall() # All link hrefs response.xpath('//img/@src').getall() # All image sources # Follow a link fetch('https://example.com/about') # Regex over the body (e.g. emails) import re re.findall(r'[\w.+-]+@[\w-]+\.[\w.-]+', response.text) Minimal Recon Spider # save as spider.py, run with: scrapy runspider spider.py -o out.json import scrapy from urllib.parse import urlparse class ReconSpider(scrapy.Spider): name = "recon" start_urls = ["https://example.com"] def parse(self, response): # Collect all links and follow same-domain ones for href in response.css('a::attr(href)').getall(): yield {"link": response.urljoin(href)} if urlparse(response.urljoin(href)).netloc == urlparse(response.url).netloc: yield response.follow(href, callback=self.parse) Useful Settings (settings.py / -s overrides) Setting Description ROBOTSTXT_OBEY Whether to honour robots.txt (default True) DOWNLOAD_DELAY Seconds between requests (politeness / rate limiting) CONCURRENT_REQUESTS Max simultaneous requests DEPTH_LIMIT Max crawl depth (0 = unlimited) USER_AGENT Custom User-Agent string RETRY_TIMES Number of retries on failed requests HTTPCACHE_ENABLED Cache responses locally to avoid re-fetching AUTOTHROTTLE_ENABLED Auto-adjust delay based on server load # Example: stealthier crawl scrapy crawl recon \ -s DOWNLOAD_DELAY=2 \ -s CONCURRENT_REQUESTS=2 \ -s AUTOTHROTTLE_ENABLED=True \ -s USER_AGENT="Mozilla/5.0" Notes Active — Scrapy makes real HTTP requests to the target; only crawl in-scope assets. Set ROBOTSTXT_OBEY=False only when authorised; by default Scrapy respects robots.txt. Use DOWNLOAD_DELAY / AUTOTHROTTLE to avoid hammering targets and tripping WAFs. Great base for custom recon crawlers — [[reconspider]] is a Scrapy-based spider built exactly for this. Export to JSON/JSONL then post-process with jq to extract emails, subdomains, and links.