Use standard libary Zstandard for Python 3.14+ #3613

mollymorphous · 2025-07-25T19:01:24Z

Summary

PEP 784 add a Zstandard implementation to the Python standard library under compression.zstd, and is scheduled for release in Python 3.14. This PR adapts ZStandardDecoder to work with either the standard library implementation or the implementation from the zstandard package.

This has the implication that Zstandard content decoding is available by default on Python 3.14 and later, without the need to install the zstd extra.

Checklist

I understand that this PR may be closed in case there was no previous discussion. (This doesn't apply to typos!)
I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
- Testing this requires Python 3.14, but the existing zstd unit tests pass with the standard library implementation. I'm always happy to add more tests if needed!
I've updated the documentation accordingly.

lovelydinosaur · 2025-07-27T21:25:32Z

Ooh interesting, thanks.

Any idea on how widely zstd is currently supported? (Wrt both browsers and servers.)

Related to this... compression is one of the currently unimplemented features in the httpx 1.0 prerelease... https://www.encode.io/httpnext/

lovelydinosaur · 2025-07-27T21:25:56Z

httpx/_decoders.py

+    def _new_decompressor(self) -> None:
+        decompressor = zstandard.ZstdDecompressor()
+        if hasattr(decompressor, "decompressobj"):
+            self.decompressor = decompressor.decompressobj()  # prgama: no cover
+        else:
+            self.decompressor = decompressor  # pragma: no cover


Could you explain this part?

This is admittedly a little awkward. Python upstreamed the pyzstd package into compress.zstd because it's API was closer to existing standard library compression APIs. The zstandard package provides a ZstdDecompressObj facade that implements the standard library style API.

An alternative would be to import the libraries under separate names so the method would look more like this:

def _new_compressor(self) -> None: if compression_zstd is not None: self.decompressor = compression_zstd.ZstdDecompressor() else: self.decompressor = zstandard.ZstdDecompressor().decompressobj()

What do you think?

if the requirement in pyproject.toml is changed to the 3.9+ backport at backports.zstd published by the pyzstd maintainer (which got moved into cpython 3.14 standard library):

backports-zstd==1.0.0 ; python_version < "3.14"

this diff would get much simpler right?

maybe with a simpler diff, this PR has a better chance of landing?

reading from chunked streams gets streamlined like that: aio-libs/aiohttp@df8ad83

lilydjwg · 2025-07-28T04:28:25Z

Any idea on how widely zstd is currently supported? (Wrt both browsers and servers.)

At least crates.io, packagist and sourceforge support it. (My tests blow up because I cache http responses and there are zstd responses from other Python versions; the "zstandard" module doesn't support 3.14 yet.)

mollymorphous

Thanks for the review! Firefox and Chrome currently support zstd (caniuse). Safari does not yet, but plans to: WebKit/standards-positions#168

I wasn't able to find a hard number on server-side deployments, the answer seems to be not a lot, but CloudFlare recently added support to the CDN.

mollymorphous · 2025-07-28T16:31:02Z

httpx/_decoders.py

+    def _new_decompressor(self) -> None:
+        decompressor = zstandard.ZstdDecompressor()
+        if hasattr(decompressor, "decompressobj"):
+            self.decompressor = decompressor.decompressobj()  # prgama: no cover
+        else:
+            self.decompressor = decompressor  # pragma: no cover


This is admittedly a little awkward. Python upstreamed the pyzstd package into compress.zstd because it's API was closer to existing standard library compression APIs. The zstandard package provides a ZstdDecompressObj facade that implements the standard library style API.

An alternative would be to import the libraries under separate names so the method would look more like this:

def _new_compressor(self) -> None: if compression_zstd is not None: self.decompressor = compression_zstd.ZstdDecompressor() else: self.decompressor = zstandard.ZstdDecompressor().decompressobj()

What do you think?

lovelydinosaur · 2025-07-29T19:52:55Z

From a bit of time reviewing this I've not been able to track down good examples of URLs to use for comparison purposes here.

Eg...

CloudFlare blog pages don't appear to use compression for hosted images.
GitHub pages sites don't appear to use compression for hosted images.
WikiPedia supports gzip compression throughout.

Here's one example of a URL that does support zstd, though in this particular case it appears less efficient than gzip...

$ curl https://help.netflix.com/en/node/30081 -H "Accept-Encoding: br" --output netflix.br 
$ curl https://help.netflix.com/en/node/30081 -H "Accept-Encoding: gzip" --output netflix.gzip
$ curl https://help.netflix.com/en/node/30081 -H "Accept-Encoding: zstd" --output netflix.zstd
$ curl https://help.netflix.com/en/node/30081 -H "Accept-Encoding: identity" --output netflix
$ wc -c netflix*
   84920 netflix
   84920 netflix.br  # Not supported on this URL
   12602 netflix.gzip
   16761 netflix.zstd

The CloudFlare pitch for zstd https://blog.cloudflare.com/new-standards/ isn't neccessarily convincing... gzip is essentially just as fast, and looks to have slightly less efficient though notably more stable compression ratios.

I'm reviewing this for the purposes of httpx 1.0, and I'm expecting that only supporting gzip might be a reasonable default.

Does anyone have some useful real world examples that'd help verify if that is / isn't a good policy?

cclauss · 2025-08-31T15:51:34Z

I suggest adding Python 3.14 to the test suite as in:

GitHub Actions: Add Python 3.14 to test matrix #3645

tuffnatty · 2025-09-08T11:58:59Z

I would suggest dropping zstandard and switch to stdlib-compatible backports.zstd on Python 3.9-3.13, as httpx has dropped Python 3.8 support already.

tuffnatty · 2025-09-08T12:13:51Z

The CloudFlare pitch for zstd https://blog.cloudflare.com/new-standards/ isn't neccessarily convincing... gzip is essentially just as fast, and looks to have slightly less efficient though notably more stable compression ratios.

gzip is definitely not just as fast (without hardware offloading), it's just that they measure the whole response time, where the compression speed difference does not seem to matter much on average.

I'm reviewing this for the purposes of httpx 1.0, and I'm expecting that only supporting gzip might be a reasonable default.

Does anyone have some useful real world examples that'd help verify if that is / isn't a good policy?

I had used this to stream huge JSONL chunks over HTTP, and while it does not make traffic much less or a great overall speed improvement, the CPU load situation has improved a lot (relative to gzip).

cclauss · 2025-09-17T05:33:32Z

@tuffnatty Would you be willing to create an alternative pull request that uses the backport and adds automated tests on Py3.14 like

GitHub Actions: Add Python 3.14 to test matrix #3645

lovelydinosaur · 2025-09-17T10:35:00Z

Okay, so my review of this was that supporting gzip only would be a sensible policy.
That's what we'll go for in 1.0. Let's not spend any more time rejigging zstd here.

lilydjwg · 2025-09-17T11:04:24Z

Okay, so my review of this was that supporting gzip only would be a sensible policy.
That's what we'll go for in 1.0. Let's not spend any more time rejigging zstd here.

Would there be a mechanism to add support for other compression methods via third-party code then? I'm a bit worried about interoperability with not-so-good servers and proxies.

lovelydinosaur · 2025-09-17T11:38:41Z

That's a good question... ...I can't answer that fully at the moment.

There might not be any API explicitly for that purpose. Here's how interop. with the streams class would be...

# A custom stream on top of the streams API...
class ZstdStream(httpx.Stream):
    def __init__(self, stream: httpx.Stream):
        self._wrapped = stream

    # Implement `.read()` and `.close()`

# Usage...
stream = ZStdStream(response.stream)
body = stream.read()

We probably don't want specific dials to "support for other compression methods via third-party code", since the less config the better. However we do want the tooling itself to be flexible enough to support customisation.

There's a current example in the docs demo'ing a custom client which is vastly more simple than with httpx 0.28. Perhaps building out that example to also demo custom response classes would be a good way to point users towards adaptability without increasing API surface area.

tuffnatty · 2025-09-18T09:57:59Z

@tuffnatty Would you be willing to create an alternative pull request that uses the backport

@cclauss Thanks but the issue has been resolved in another way.

ddelange · 2025-10-03T03:29:22Z

Okay, so my review of this was that supporting gzip only would be a sensible policy. That's what we'll go for in 1.0. Let's not spend any more time rejigging zstd here.

fwiw, the major alternatives to httpx support zstd (and for sure brotli):

aiohttp
urllib3
- requests

supporting only gzip/deflate might be insufficient in 2025. especially brotli accounts for 33% of compressed http responses already in 2021 (Figure 22.4) and 44% of compressed javascript versus 41% gzip in 2024 (Figure 1.21)

cclauss · 2025-10-03T04:39:04Z

the issue has been resolved in another way.

How was the issue resolved?

tuffnatty · 2025-10-03T16:45:10Z

the issue has been resolved in another way.

How was the issue resolved?

#3613 (comment)

ddelange · 2025-10-08T16:14:14Z

disregard my last comment, I see that they're all supported 👍

mollymorphous added 2 commits July 25, 2025 14:34

feat: Use standard library zstd (3.14+) if available

b312e20

docs: Default zstd support for Python 3.14+

824f3f5

lovelydinosaur reviewed Jul 27, 2025

View reviewed changes

mollymorphous commented Jul 28, 2025

View reviewed changes

cclauss mentioned this pull request Sep 8, 2025

Use backports.zstd instead of zstandard #3662

Closed

3 tasks

lovelydinosaur closed this Sep 17, 2025

anuraaga mentioned this pull request Oct 9, 2025

Support Python 3.14 connectrpc/connect-python#30

Merged

Uh oh!

Use standard libary Zstandard for Python 3.14+ #3613

Use standard libary Zstandard for Python 3.14+ #3613

Conversation

mollymorphous commented Jul 25, 2025

Summary

Checklist

Uh oh!

lovelydinosaur commented Jul 27, 2025

Uh oh!

lovelydinosaur Jul 27, 2025

Choose a reason for hiding this comment

Uh oh!

mollymorphous Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

ddelange Oct 12, 2025

Choose a reason for hiding this comment

Uh oh!

lilydjwg commented Jul 28, 2025

Uh oh!

mollymorphous left a comment

Choose a reason for hiding this comment

Uh oh!

mollymorphous Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

lovelydinosaur commented Jul 29, 2025

Uh oh!

cclauss commented Aug 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tuffnatty commented Sep 8, 2025

Uh oh!

tuffnatty commented Sep 8, 2025

Uh oh!

cclauss commented Sep 17, 2025

Uh oh!

lovelydinosaur commented Sep 17, 2025

Uh oh!

lilydjwg commented Sep 17, 2025

Uh oh!

lovelydinosaur commented Sep 17, 2025

Uh oh!

tuffnatty commented Sep 18, 2025

Uh oh!

ddelange commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cclauss commented Oct 3, 2025

Uh oh!

tuffnatty commented Oct 3, 2025

Uh oh!

ddelange commented Oct 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

cclauss commented Aug 31, 2025 •

edited

Loading

ddelange commented Oct 3, 2025 •

edited

Loading