Rate Limiting

What is Rate Limiting?

Rate limiting is a technique used by websites to control how many requests a single client can make within a specific time window. It prevents abuse, protects server resources, and ensures fair usage among all visitors. Common implementations include “100 requests per minute” or “5 requests per second.”

Rate limiting is the traffic cop of the web. Respect it, or get blocked. Simple as that. Even the most legitimate scraping operation will get IP-banned if it hammers a site with 500 requests per second. Speed kills — in web scraping, literally.

Rate Limit Response Codes

Implementing Polite Scraping

import time
import requests
from ratelimit import limits, sleep_and_retry

@limits(calls=30, period=60)  # 30 calls per minute
def polite_request(url):
    response = requests.get(url, headers=headers)
    response.raise_for_status()
    return response.text

# Or handle 429 manually
def request_with_backoff(url, max_retries=5):
    for attempt in range(max_retries):
        resp = requests.get(url)
        if resp.status_code == 429:
            wait = (2 ** attempt) * 1  # Exponential backoff
            time.sleep(wait)
            continue
        return resp
    raise Exception("Max retries exceeded")

Adaptive Rate Limiting

Golden rules for production scraping:

1. Start slow — Begin with 1 request/second, observe
2. Monitor responses — 429s mean you’re too fast
3. Randomize intervals — Add jitter: sleep(1 + random.random())
4. Respectrobots.txt — The Crawl-delay directive matters

Pro tip: Some sites rate-limit by endpoint. Product pages might allow 10/sec while search results only allow 2/sec. Map out rate limits by testing systematically, then configure your scraper accordingly.