User-Agent

What is User-Agent?

The User-Agent is a string that identifies your browser, operating system, and version to the websites you visit. Every HTTP request includes this header, telling servers “I’m using Chrome on Windows 11” or “I’m a Python script.” Websites use this info to serve appropriate content, but also to identify and block scrapers.

The User-Agent string is your browser’s ID card. Master it to avoid detection and access restricted content. Using the right User-Agent won’t make you invisible, but using a suspicious or missing one will definitely get you blocked fast.

Anatomy of a User-Agent

Mozilla/5.0 (Windows NT 10.0; Win64; x64) 
AppleWebKit/537.36 (KHTML, like Gecko) 
Chrome/120.0.0.0 Safari/537.36

Breakdown:
- Mozilla/5.0    : Historical compatibility prefix
- Windows NT 10.0: Operating System
- Win64; x64     : 64-bit architecture
- AppleWebKit    : Rendering engine
- Chrome/120.0.0.0: Browser version
- Safari/537.36  : WebKit compatibility

Common User-Agent Strategies

1. Real Browser Profiles (Best for Stealth)

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'
}

2. Rotate Realistic UAs

# Keep a pool of recent browser UAs
ua_pool = [
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36...',
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36...',
    'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36...',
]

User-Agent Checklist

Reality check: User-Agent is just one signal. Sophisticated anti-bot systems check JavaScript execution, canvas fingerprints, WebGL renderer, and hundreds of other data points. UA spoofing alone won’t fool Cloudflare or Akamai.