Long before I touched a real data pipeline, TV Series Silicon Valley’s Pied Piper made me think compression ratios were the most dramatic thing in tech. Turns out, they kind of are. So let’s dig into compression codec ZSTD (Zstandard) and understand why it is becoming such an important, preferred codec in the data engineering world.
How ZSTD Works
I am not going to walk you through the full history of compression theory, mostly because nobody needs that to make a good decision at work. Here is the short version of what is actually happening under the hood.
Snappy, the codec most of us grew up with, was built for one priority only: speed, even at the cost of ratio. ZSTD was designed with the benefit of hindsight, and it works in two stages instead of leaning on just one trick.
Pattern matching. ZSTD scans your data for repeated sequences and replaces them with short references back to where that sequence already appeared, the same core idea most general purpose compressors use. What sets it apart is that it searches a much larger window of previously seen data than Snappy does, so it catches repeated patterns that Snappy would simply miss because they are too far back to look at.
Entropy coding. After the pattern matching pass, there is still some redundancy left in how often certain bytes or symbols show up. ZSTD squeezes that remaining redundancy using a technique called Finite State Entropy, a modern variant of Asymmetric Numeral Systems. It gets very close to the theoretical best possible compression for that data, and it does it fast enough to not become a bottleneck.
A dial instead of a fixed setting. Because ZSTD does not commit to one trade off, it exposes compression levels from 1 to 22. Lower levels lean on faster, looser pattern matching. Higher levels search harder for matches and spend more effort on entropy coding, at the cost of speed.
📖 Recommended Reading: RFC 8878: Zstandard Compression
Here are the Write and Read path steps:
There is also a layer people forget about. Formats like Parquet already apply lightweight, type aware encoding to columns before any general purpose codec even runs. So ZSTD is only squeezing what is left over after that first pass, which is part of why the gains from switching codecs on already encoded Parquet data can look smaller than raw file compression benchmarks would suggest.
Benchmarks From the Field
Rather than trust one number in isolation, here is what a few independently published benchmarks found.
A hands-on Parquet benchmark by Jerónimo López compressed the same 19.6 million row NYC Taxi Trips dataset with six different codecs and published the raw file sizes. Snappy, LZ4, and LZO wrote at roughly the same speed as leaving the data uncompressed, ZSTD added modest overhead, and Gzip was the slowest, worsening write time by about 50 percent.
A separate independent benchmark built specifically to compare these codecs found Snappy and LZ4 decompressing at over 3.5GB/s, ZSTD holding steady around 1GB/s across compression levels, and Gzip trailing well behind both.
At a high level, ZSTD compresses noticeably smaller than Snappy and LZ4, and lands close to Gzip's ratio. On speed, it's slower than Snappy and LZ4 to decompress, though real Parquet read performance only loses a few percent. Worth reading both linked articles for more detailed analysis.
Where ZSTD Is Actually Gaining Ground
Snappy has been the default compression codec for most of the big data ecosystem for years, and it still is in a lot of places e.g. Spark defaulted to Snappy for Parquet since version 2.1.0, and that has not changed in Spark 4.0. So this is not a story about ZSTD having already taken over everywhere. It is a story about where it is winning first, and why.
💡Worth mentioning: GZIP was the default in multiple tools including Spark, but it's been phased out of the big data world. The reason is that distributed engines need splittable files, and GZIP can only be decompressed by a single core which leads to bottleneck at scale.
Apache Iceberg is the clearest example of ZSTD’s rise. Its default Parquet compression codec was GZIP, until the project switched that default to ZSTD starting with version 1.4.0, a real signal of where the ecosystem is heading, even if some engine integrations are still catching up.
📖 Pull Request: Default to zstd compression for Parquet
ClickHouse tells a similar story from a different angle. It defaults to LZ4 for hot query paths but recommends ZSTD for cold storage, trading some decompression speed for a smaller footprint once speed is no longer the priority.
📖 Recommended Reading: Database compression: encodings, codecs and ratios
That shows ZSTD is getting popular but does not mean it has replaced everything overnight. It does seem like it is slowly getting there.
Decision Flow: When You Should Actually Reach for ZSTD
The biggest real-world payoff from any of this is usually the storage bill, not raw speed. A 15 to 20 percent smaller footprint adds up fast once you are paying for it across petabytes of S3.
If your workload is latency sensitive, like streaming ingestion or a hot query path, stick with Snappy or LZ4.
If not, but the data is read very frequently, like a table scanned repeatedly by dashboards or queries, stay at ZSTD level 3 so decompression speed does not become the bottleneck.
If the data is cold or archival and rarely read again, push ZSTD up to level 9 or higher, since the extra compression time barely matters for data written once.
Otherwise, use ZSTD in the level 3 to 9 range and benchmark read latency on your own workload.
The ZSTD compression level itself is something you configure for fine tuning: Spark exposes it via a property parquet.compression.codec.zstd.level, while Iceberg exposes it as a table property, write.parquet.compression-level, which defaults to level 3.
Whatever you land on, I would recommend to benchmark it on your own data. Compression behavior depends heavily on the shape of your data, and the gap between a published benchmark and your actual production table can be significant.
The Bottom Line
ZSTD isn’t magic, and it’s definitely not the right choice for every workload. Its popularity makes sense, though. It gives you a larger match window, strong entropy coding, and, importantly, a bit more control over the balance between compression and performance.
Snappy isn’t going anywhere either. For a lot of write-heavy workloads, it’s still the simpler and faster choice. The trick is knowing when the extra compression from ZSTD is actually worth the additional CPU cost.