Cloudflare cache compression cuts storage needs by 2.8× using Zstandard

Cloudflare has prototyped a cache storage optimization called Cache Transcoding that applies Zstandard (zstd) compression to eligible web assets, reducing their disk footprint by approximately 2.8 times. The system, developed during an intern project, targets uncompressed text content like HTML, JSON, CSS, and JavaScript—which comprised 67.3% of requests but only 22.3% of bytes in sampled traffic—while excluding already-compressed media. Processing occurs in Pingora-based proxies at zstd level 3, adding minor CPU overhead for encoding while preserving origin content fidelity. Initial testing focused on responses exceeding 4 KiB with unset Content-Encoding headers, covering 71% of eligible text assets. Compressed objects remain in zstd format during Tiered Cache transfers between data centers, decoding only when serving end users. This reduced inter-DC bandwidth usage and increased effective storage capacity—critical given that uncompressed text representations previously occupied disproportionate cache space. The 4 KiB threshold excluded tiny files where compression overhead outweighed benefits, affecting just 1% of compressible bytes. Technical validation involved over one million test requests across 10 cache servers, tracking metrics via Prometheus and Jaeger. While the 2.8× compression ratio reflects a controlled corpus of ~195-272 KiB assets, broader deployment will assess variable content types. Future iterations may adjust zstd levels or expand eligibility to pre-compressed origin responses. The approach demonstrates how lossless compression algorithms can optimize infrastructure-scale storage without modifying origin content.