Another piece is that PyPI has no index— it's just a giant list of URLs [1] where any required metadata (eg, the OS, python version, etc) is encoded in the filename. That makes it trivial to throw behind a CDN since it's all super static, but it has some important limitations:
- there's no way to do an installation dry run without pre-downloading all the packages (to get their dep info)
- there's no way to get hashes of the archives
- there's no way to do things like reverse-search (show me everything that depends on x)
I'm assuming that a big part of pyx is introducing a dynamically served (or maybe even queryable) endpoint that can return package metadata and let uv plan ahead better, identify problems and conflicts before they happen, install packages in parallel, etc.
Astral has an excellent track record on the engineering and design side, so I expect that whatever they do in this space will basically make sense, it will eventually be codified in a PEP, and PyPI will implement the same endpoint so that other tools like pip and poetry can adopt it.
For sdists, this is impossible until we can drop support for a bunch of older packages that don't follow modern standards (which is to say, including the actual "built" metadata as a PKG-INFO file, and having that file include static data for at least name, version and dependencies). I'm told there are real-world projects out there for which this is currently impossible, because the dependencies... depend on things that can't be known without inspecting the end user's environment. At any rate, this isn't a PyPI problem.
> there's no way to get hashes of the archives
This is provided as a URL fragment on the URLs, as described in https://peps.python.org/pep-0503/. Per PEP 658, the hash for the corresponding metadata files is provided in the data-dist-info-metadata (and data-core-metadata) attributes of the links.
> This is provided as a URL fragment on the URLs, as described in https://peps.python.org/pep-0503/. Per PEP 658, the hash for the corresponding metadata files is provided in the data-dist-info-metadata (and data-core-metadata) attributes of the links.
Also, this is an implementation detail, but the archives are stored in a content-addressable store with the path being based on the (hexadecimal) blake2b/256 hash of their contents.
Ah interesting, thanks for that! I was frustrated once again recently to note that `pip install --dry-run` required me to pre-download all packages, so I assumed nothing had changed.
You could do worse than to start using --only-binary=:all: by default. (It's even been proposed as default behaviour: https://github.com/pypa/pip/issues/9140) Even if you can't actually install that way, it will point out the places where sdists are needed.
In principle, separate metadata availability should still at least be possible for most sdists eventually. But I'm not the one calling the shots here.
Should I expect that to download only metadata and not whole wheels/sdists for everything? Or does that depend on everything in my requirements file being available as a wheel?
- there's no way to do an installation dry run without pre-downloading all the packages (to get their dep info)
- there's no way to get hashes of the archives
- there's no way to do things like reverse-search (show me everything that depends on x)
I'm assuming that a big part of pyx is introducing a dynamically served (or maybe even queryable) endpoint that can return package metadata and let uv plan ahead better, identify problems and conflicts before they happen, install packages in parallel, etc.
Astral has an excellent track record on the engineering and design side, so I expect that whatever they do in this space will basically make sense, it will eventually be codified in a PEP, and PyPI will implement the same endpoint so that other tools like pip and poetry can adopt it.
[1]: Top-level: https://pypi.org/simple/ Individual package: https://pypi.org/simple/pyyaml/