Scraping principles

• Prefer official APIs when available
• Respect Terms of Service
• Rate-limit requests
• Identify your scraper (User-Agent)
• Don’t scrape private data

robots.txt basics

robots.txtrobots.txt is a convention that tells crawlers which paths are allowed/disallowed.

It’s not a security feature, but a strong signal.

Rate limiting

Use delays and backoff:

import time
import random
 
 
def polite_sleep(base=1.0):
    time.sleep(base + random.random())

import time
import random
 
 
def polite_sleep(base=1.0):
    time.sleep(base + random.random())

Avoid getting blocked

• keep concurrency low
• cache responses
• handle 429/503
• rotate proxies only if permitted and ethical