Pip has been a flag bearer for Python packaging standards for some time now, so that alternatives can implement standards rather than copy behavior. So first a lock file standard had to be agreed upon which finally happened this year: https://peps.python.org/pep-0751/
Now it's a matter of a maintainer, who are currently all volunteers donating their spare time, to fully implement support. Progress is happening but it is a little slow because of this.
Python has about 40 keywords, I say I would regularly use about 30, and irregularly use about another 5. Hardly seems like a "junkyard".
Further, this lack of first class support for lazy importing has spawned multiple CPython forks that implement their own lazy importing or a modified version of the prior rejected PEP 690. Reducing the real world need for forks seems worth the price of one keyword.
False await else import pass
None break except in raise
True class finally is return
and continue for lambda try
as def from nonlocal while
assert del global not with
async elif if or yield
Soft Keywords:
match case _ type
I think nonlocal/global are the only hard keywords I now barely use, for the soft ones I rarely use pattern matching, so 5 seems like a good estimate
> I had been hoping someone would introduce the non-virtualenv package management solution that every single other language has where there's a dependency list and version requirements (including of the language itself) in a manifest file (go.mod, package.json, etc) and everything happens in the context of that directory alone without shell shenanigans.
Isn't that exactly a pyproject.toml via the the uv add/sync/run interface? What is that missing that you need?
A lot of the tools you list here are composable, mypy checks type hints, black formats code, because of the purpose and design ethos of those two tools they would never think to merge.
So which is it that you want, to just reach for one tool or have tools that have specific design goals and then have to reach for many tools in a full workflow?
FWIW Astral's long term goal appears to be to just reach for one tool, hence why you can now do "uv format".
> There is one part of Python that I consider a design flaw when it comes to packaging: the sys.modules global dictionary means it's not at all easy in Python to install two versions of the same package at the same time. This makes it really tricky if you have dependency A and dependency B both of which themselves require different versions of dependency C.
But it solves the problem that if A and B both depend on C the user can pass an object from A to B that was created by C without worrying about it breaking.
In less abstract terms, let's say numpy one day changed it's internal representation of an array, so if one version of numpy read an array of a different version of numpy it would crash or worse read it but misinterpret it. Now if I have one data science library produces numpy arrays and another visualization library that takes numpy arrays, I can be confident that only one version of numpy is installed and the visualization library isn't going to misinterpret the data from the data because it is using a different version of numpy.
This stability of installed versions have allowed entire ecosystems build around core dependencies in a way that would be tricky without that. I would therefore not consider it a design flaw.
I wouldn't mind a codebase where numpy objects created by dependency B can't be shared directly with dependency A without me first running some kind of conversion function on them - I'd take that over "sorry you want to use dependency A and dependency B in this project, you're just out of luck".
> I wouldn't mind a codebase where numpy objects created by dependency B can't be shared directly with dependency A without me first running some kind of conversion function on them
Given there's no compiler to enforce this check, and Python is dynamic language, I don't see how you implement that without some complicated object provenance feature, making every single object larger and every use of that object (calling with it, calling it, assigning it to an attribute, assigning an attribute to it) impose an expensive runtime check.
You let people make the mistake and have the library throw an exception if they do that, not through type checking but just through something eventually calling a method that doesn't exist.
> You let people make the mistake and have the library throw an exception if they do that, not through type checking but just through something eventually calling a method that doesn't exist.
Exceptions or crashes would be annoying, but yes, are manageable, although try telling that to new users of the language that their code doesn't work because they didn't understand the transitive dependency tree of their install and it automatically vendored different versions of a library for different dependencies, and how did they not know that from some random exception occurring in a dependency.
But as I explain in my example, the real problem is that one version of the library reads the data in a different layout from the other, so instead you end of with subtle data errors. Now your code is working but your getting the wrong output, good luck debugging that.
The advantage is type hints can be fixed without needing to release a new version of Python. The disadvantage is there's a lot of code in the standard library that doesn't really consider how to represent it with type hints, and it can be really tricky and not always possible.
I'm surprised to see so many people moving to pyrefly, ty, and zuban so quickly. I was going to wait until some time in 2026 to see which has matured to the point I find useful, I guess some users really find existing solutions actually unworkable.
Hmm. Presumably mypy and pyrefly use the same ones, but then I don't understand why pyrefly is complaining and mypy isn't:
ERROR Argument `Future[list[BaseException | Any]]` is not assignable to parameter `future` with type `Awaitable[tuple[Any]]` in function `asyncio.events.AbstractEventLoop.run_until_complete` [bad-argument-type]
--> xxx/xxx.py:513:33
|
513 | loop.run_until_complete(tasks.gather(*x, return_exceptions=True))
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(tbh this is rather insignificant compared to the noise from external packages without type annotations, or with incomplete ones… pyrefly's inferences at the existence of attributes and their types are extremely conservative…)
> Hmm. Presumably mypy and pyrefly use the same ones, but then I don't understand why pyrefly is complaining and mypy isn't:
> …where is it even puling "tuple[Any]" from…
Perhaps it's a bug in pyrefly, perhaps mypy or pyrefly is able to infer something about the types that the other isn't. I would strongly suggest checking their issues page, and if not seeing a report already report it yourself.
While there is an ongoing push to more consistently document the typing spec: https://typing.python.org/. It does not actually cover all the things you can infer from type hints, and different type hint checkers have decided to take different design choices compared to mypy and will produce different errors even in the most ideal situation.
This is one of the reasons why I am waiting for these libraries to mature a little more.
> it does not actually cover what rules you can check or infer from type hints
Indeed this is the cause of maybe 30% of the warnings I'm seeing… items being added to lists or dicts in some place (or something else making it infer a container type), and pyrefly then refusing other types getting added elsewhere. The most "egregious" one I saw was:
def something(x: list[str]) -> str:
foo = []
for i in x:
foo.append(i)
return "".join(foo)
Where it complains:
ERROR Argument `str` is not assignable to parameter `object` with type `LiteralString` in function `list.append` [bad-argument-type]
--> test.py:4:20
4 | foo.append(i)
Edit: now that I have posted it, this might actually be a bug in the .join type annotation… or something
Edit: nah, it's the loop (and the LiteralString variant of .join is just the first overload listed in the type hints)… https://github.com/facebook/pyrefly/issues/1107 - this is kinda important, I don't think I can use it before this is improved :/
I assume in your example if you update the foo declaration to the following it solves the complaint:
foo: list[str] = []
If so this a type checking design choice:
* What can I infer from an empty collection declaration?
* What do I allow further down the code to update inferences further up the code?
I don't know Pyrefly's philosophy here, but I assume it's guided by opinionated coding guidelines inside Meta, not what is perhaps the easiest for users to understand.
It is a purely subjective design decision, but I personally prefer the stricter rules that don’t do backwards type inferences like this… type hints shouldn’t follow duck typing semantics. Otherwise you’re not providing nearly as much value IMO. Typescript is really the model organism here. They took the most cursed mainstream programming language, and made it downright good.
Today, the “: list[str]” is 11 wasted characters and it’s not as aesthetically pleasing. Tomorrow, you do some refactor and your inferred list[str] becomes a list[int] without you realizing it… I’m sure that sounds silly in this toy example, but just imagine it’s a much more complex piece of code. The fact of the matter is, you declared foo as a list[any] and you’re passing it to a function that takes an iterable[str] — it ought to complain at you! Type hints are optional in Python, and you can tell linters to suppress warnings in a scope with a comment too.
That being said, perhaps these more permissive rules might be useful in a legacy codebase where no type annotations exist yet.
Really, it’d be extra nice if they made this behavior configurable, but I’m probably asking for too much there. What’s next, a transpiler? Dogs and cats living together?!
> Today, the “: list[str]” is 11 wasted characters and it’s not as aesthetically pleasing. Tomorrow, you do some refactor and your inferred list[str] becomes a list[int] without you realizing it… I’m sure that sounds silly in this toy example, but just imagine it’s a much more complex piece of code.
Hmm. I'm looking at a codebase that is still in a lot of "early flux", where one day I might be looking at a "list[VirtualRouter]" but the next day it's a "list[VirtualRouterSpec]". It's already gone through several refactors and it kinda felt like the type hints were pretty much spot on in terms of effort-benefit. It's not a legacy codebase; it has reasonably good type hint coverage, but it's focused on type hinting interfaces (a few Protocol in there), classes and functions. The type hinting inline in actual code is limited.
I do understand your perspective, but tbh to me it feels like if I went that far I might rather not choose Python to begin with…
I agree, but as a type checker it is a subjective choice, whether to be explicit and not make assumptions or whether to infer from certain certain patterns as probably correct and not be noisy to the user. Very glad to see they plan to fix this.
Are there any reliable decentralized package distribution systems operating at within 2 orders of magnitude of that scale? How do they handle administrative issues such as malicious packages or name squatting? Standards updates? Enforcement of correct metadata? And all the other common things package indexes need to handle.
I'm clearly skeptical, but would be very interested in any real world success stories.
There is, the web. The web distributes code directly to end users at a much larger scale. To distribute the bandwidth costs, the web is federated: to depend on a script you refer to its url, and whoever hosts this url foots the bill.
Deno is a Javascript implementation for the backend that attempts to mimic this pattern (it later introduced a more npm-like centralized repository, but afaik it's optional). Deno is of course less popular than Python, but its url-centered model can really scale imo.
> There is, the web. The web distributes code directly to end users at a much larger scale. To distribute the bandwidth costs, the web is federated: to depend on a script you refer to its url, and whoever hosts this url foots the bill.
But the Web is notorious for the problems I listed, you end up with standards around not following standards. It leaves almost all the responsibility on the client tool (browser or whatever) to do validation to stop malicious sites, name squatting, accepting and "fixing" poorly constructed metadata etc.
> Deno is a Javascript implementation for the backend that attempts to mimic this pattern (it later introduced a more npm-like centralized repository, but afaik it's optional). Deno is of course less popular than Python, but its url-centered model can really scale imo.
I was not familiar with Deno, I've done some shallow reading on this now and it's certainly interesting. I don't know enough about the JavaScript world to make a comment on the pros or cons.
But I don't think can work for Python, as transitive dependencies would immediately conflict as soon as dependencies required a different version of the same transitive dependency. And the guarantee of Python packaging is you only have a single version of a library installed in an environment, while it can cause some dependency solver headache, it also solves a lot of problems as it makes it safe to pass around objects.
What has been the community reaction? Has allowing scripts been scalable for users? Or could it be described as people blindly copying and pasting allow commands?
I am involved in Python packaging discussions and there is a pre-proposal (not at PEP stage yet) at the moment for "wheel variants" that involves a plugin architecture, a contentious point is whether to download and run the plugins by default. I'd like to find parallels in other language communities to learn from.
In my experience, packages which legitimately require a postinstall script to work correctly are very rare. For the apps I maintain, esbuild is the only dependency which benefits from a postinstall script to slightly improve performance (though it still works without the script). So there's no scaling issue adding one or two packages to a whitelist if desired.
> However, pip has some really gnarly internal infrastructure that prevents it from taking advantage of a lot of uv's good ideas (which in turn are not all original).
FWIW, as a pip maintainer, I don't strongly agree with this statement, I think if pip had the same full time employee resources that uv has enjoyed over the last year that a lot of these issues could be solved.
I'm not saying here that pip doesn't have some gnarly internal details, just that the bigger thing holding it back is the lack of maintainer resources.
> For just one example: uv can quickly install previously installed packages by hard-linking a bunch of files from the cache. For pip to follow suit, it would have to completely redo its caching strategy from the ground up, because right now its cache is designed to save only download effort and not anything else about the installation process.
I actually think this isn't a great example, evidenced by the lack of a download or wheel command from uv due to those features not aligning with uv's caching strategy.
That said, I do think there are other good examples to your point, like uv's ability to prefetch package metadata, I don't think we're going to be able to implement that in pip any time soon due to probably the need for a complete overhaul of the resolver.
> FWIW, as a pip maintainer, I don't strongly agree with this statement, I think if pip had the same full time employee resources that uv has enjoyed over the last year that a lot of these issues could be solved.
Fair enough. I'm sure if someone were paying me a competitive salary to develop my projects, they'd be getting done much faster, too.
> I actually think this isn't a great example, evidenced by the lack of a download or wheel command from uv due to those features not aligning with uv's caching strategy.
I guess you're talking about the fact that uv's cache only stores the unpacked version, rather than the original wheel? I'm planning to keep the wheel around, too. But my point was more that because of this cache structure, pip can't even just grab the wheel from its cache without hitting the Internet, on top of not having a place to put a cache of the unpacked files.
> uv's ability to prefetch package metadata,
You mean, as opposed to obtaining it per version, lazily? Because it does seem like the .metadata file system works pretty well nowadays.
> I don't think we're going to be able to implement that in pip any time soon due to probably the need for a complete overhaul of the resolver.
Ugh, yeah. I know the resolution logic has been extracted as a specific package, but it's been challenging trying to figure out how to actually use that in a project that isn't pip.
> I guess you're talking about the fact that uv's cache only stores the unpacked version, rather than the original wheel? I'm planning to keep the wheel around, too. But my point was more that because of this cache structure, pip can't even just grab the wheel from its cache without hitting the Internet, on top of not having a place to put a cache of the unpacked files.
I'm talking about the fact there is no `uv pip download` or `uv pip wheel`.
I'm sure we'll discuss when you add this feature to uv, but I personally find uv's cache already grows too big too fast, so adding more to it makes me concerned.
> You mean, as opposed to obtaining it per version, lazily? Because it does seem like the .metadata file system works pretty well nowadays.
Yeah, one of the way uv speeds up resolving is it pre-downloads .metadata files and checks if their requirements are identical to versions it had already checked, so it can quickly rule them out. It's a clever use of a collector and an advanced resolver.
>FWIW, as a pip maintainer, I don't strongly agree with this statement, I think if pip had the same full time employee resources that uv has enjoyed over the last year that a lot of these issues could be solved.
No. This something people tell each other, that it's a lack of resources, but in reality almost all OSS projects with long standing flaws don't have a resources problem. They have a prioritization problem, where they actively ignore and refuse to work on things that affect users every single day of usage.
There are features in FreeCAD that are straight up just broken that you hit every single day of using FreeCAD. When a simple low cost fix is suggested you get immense pushback, because of how "impure" the fix is despite being entirely cosmetic, also being reversible and having no long term impact on maintenance.
Progress happens when you sidestep those who block progress. That's what realthunder was doing and that's what Astral is doing with uv. No more bullshit excuses.
This is what forks are for. Alternately, money would probably help. Otherwise it's a bit rude to impose your priorities on others.
Perhaps some larger projects lose track of what they intended to prioritize. This is also a resources problem. There's nobody available to do management (and probably also not a social structure internally that would allow for it).
> because of how "impure" the fix is despite being entirely cosmetic, also being reversible and having no long term impact on maintenance.
The developers will always be in a better position than end users to assess the long-term maintenance impact.
> The developers will always be in a better position than end users to assess the long-term maintenance impact.
At best developers may be in a better position to assess long-term consequences. The history of major software projects is also the history of many projects making technical decisions that look good in the short-term but which turn out to impede actually achieving the desired features/performance/etc, and thus lead to major refactors/rewrites.
imtringued is right though. Obstinate maintainership is a real failure mode of open source (and other) projects and it's often true that patch rejections involving phrases like "maintenance burden" are often pretextual. It's a shame to have to resort to creative destruction to improve things. Forks have a low success probability.
> Obstinate maintainership is a real failure mode of open source
I agree, but I also think sometimes that people think it's obstinate maintainership and it's not. For example, large PRs are hard to merge from a contributor who is fresh to the project because there is so much careful review that needs to be done, it's very resource intensive.
That said, one of my goals of becoming a maintainer has to be to try and making submitting a PR more friendly. Feel free to submit a PR to pip, I will be happy to help you get over any teething issues.
> No. This something people tell each other, that it's a lack of resources,
Well uv, as an OSS project, has come along with around 2 orders of magnitude more manpower than pip and has solved a lot of problems
> but in reality almost all OSS projects with long standing flaws don't have a resources problem. They have a prioritization problem, where they actively ignore and refuse to work on things that affect users every single day of usage.
I wasn't making a statement about OSS projects in general, but I agree that pip has prioritization problems but I would argue that it stems from the lack of resources.
The people who do donate their spare time are in fact only likely to donate that time on things that interest them. If there were resources to have someone act like a project manager, and other to follow their lead, then the prioritization problems could be fixed, but there aren't those resources.
> No more bullshit excuses
Then contribute your full time hours to pip, or other OSS that needs fixing, rather than arm chair commenting on hacker news.
Pip has been a flag bearer for Python packaging standards for some time now, so that alternatives can implement standards rather than copy behavior. So first a lock file standard had to be agreed upon which finally happened this year: https://peps.python.org/pep-0751/
Now it's a matter of a maintainer, who are currently all volunteers donating their spare time, to fully implement support. Progress is happening but it is a little slow because of this.
reply