Brotli Compression

Technical Definition

Brotli is a compression algorithm developed by Google that provides 15-25% better compression ratios than gzip for web content. Modern websites increasingly serve assets and API responses Brotli-compressed, especially over HTTPS connections. For high-volume scraping operations, supporting Brotli decompression significantly reduces bandwidth costs and improves extraction speed—fetching compressed responses and decompressing locally uses less network capacity than transferring uncompressed content, particularly for large HTML pages and JSON APIs.

Business Use Case

Media monitoring services scraping news sites with heavy Brotli-compressed content see substantial bandwidth savings by supporting decompression. Price comparison engines accessing thousands of product pages per minute reduce data transfer costs by 20%+ with Brotli support. API aggregators calling JSON endpoints that return large payloads benefit from both faster transfer times and reduced bandwidth costs.

Pro-Tip

Most HTTP libraries support Brotli via optional modules—ensure your scraper’s HTTP client includes Brotli decompression. In Python, add brotli or brotlicffi to your dependencies. In Node.js, use the brotli package with fetch or axios. Verify Brotli support by checking Accept-Encoding headers—servers indicate Brotli capability, and properly configured clients should include ‘br’ in Accept-Encoding to receive compressed responses.