as someone fairly new to larger python projects, working in a team too - why does it seem like I’m always in dependency hell? Sometimes things work with venv until I forget and a notebook needs something else. Or a diff type of proj because mlflow wanted another kind of env manager?
It would be similar hell in for example Matlab or C/C++ if install of external packages were made so easy.
Some systems that are designed more with the concept of maintenance challenges (Windows and others) make it possible to have different versions installed simultaneously.
The need for the whole venv thing fundamentally underscores the problem. How many versions of libc do you have installed simultaneously? (docker users pls don’t respond)
i reckon that’s mostly it - I get an idea, wanna try something and don’t “reset” to a fresh venv. or I do then forget to reinstall the tools I want to use cause I was “just” using them in this same terminal.
it’s me not python. though it’s funny getting a few replies with different solutions. like in PERL, tmtowtdi.
6 years of python and I’ve never really had that problem, also working on larger projects. Use poetry or uv and you’ll probably be fine.
Unless you’re doing something strange with your dependencies.
The only thing I would say is non trivial is updating the dependencies. And if a library has a bug or something you have to downgrade for.
You can specify dev dependencies for notebooks and such.
I’ve not heard of mlflow having a problem with a manager. Perhaps you’re in a cloud environment and don’t have access to poetry for example?
never used poetry. just venv, virtualenv and such. I guess I just don’t know the current era’s idiomatic way of doing things. I’m more familiar with java/mvn, rust, etc. It seems like every manning book on a pythonic tool has a different way the author setup the env. to be expected sure. I just need to grok and settle into my own. :)
Poetry/uv is similar to Rust’s cargo. You specify your direct dependencies in a TOML file and their version constraints and the tool takes care of the rest.
That’s one of the main pain points of Python. Something like poetry or uv can help but the python ecosystem is just unstable. It’s very hard to update a dependancy. You project might only run with very specific versions of libraries and the language itself.
I honestly hate that about python and I don’t think python is good for developing large software. Even js/ts is a much better ecosystem. (I’m bracing for the hate)
hrm. yeah. I am “trying on” some basic ml models (just simple classifications) into our workflow and while I can whip one up with scikit, a notebook, and serve it with flask/fast - there is no way I can build a workflow tenable for others who are more “data analytics” than coders. The ecosystems for low/no code model generation (knime etc) with handoff to engineers like me to serve is … young.
If you use conda, I suggest using pixi as a project manager. It has lock files which will fix dependencies, and it can activate your enviroments with scripts and variables defined in your pyproject.toml. It has been so much better than using conda directly: https://pixi.sh/
wreck is a dependencies manager, which is venv aware, BUT is not a venv manager nor does it put dependencies into pyproject.toml. Sticks with good ol’ requirement files.
Assumes, for each package, it’s normal to be working with several venv.
Syncs the dependencies intended for the same venv.
Across many packages, unfortunately have to resort to manually sync’ing dependencies.
Lessons learned
have very recently ceased putting build requirements into requirement files. The build requirements’ transitive dependencies should be in the requirement files. wreck does not support this yet. i’m manually removing dependencies like: build wheel setuptools-scm (setuptools and pip already filtered out) and click.
syncing across packages is really time consuming. Go thru this pain, then no dependency hell.
Wrote wreck cuz all the other options were combining requirements management with everything including the bathroom sink. build backends … venv management … everything goes into pyproject.toml. All these ideas seem to just compound the learning curve.
Less is more especially when it comes to learning curve.
You’ll likely also want to check out ruff for linting and formatting, by the same company that makes uv. It doesn’t enable a lot of lints by default, but there’s a long list of checks to enable.
They also have a typechecker, ty, which is still in early alpha. If it’s as good as their other tools I expect it to become the standard for typechecking Python. Currently you’ll likely want to go with pyright for that.
as someone fairly new to larger python projects, working in a team too - why does it seem like I’m always in dependency hell? Sometimes things work with venv until I forget and a notebook needs something else. Or a diff type of proj because mlflow wanted another kind of env manager?
It would be similar hell in for example Matlab or C/C++ if install of external packages were made so easy.
Some systems that are designed more with the concept of maintenance challenges (Windows and others) make it possible to have different versions installed simultaneously.
The need for the whole venv thing fundamentally underscores the problem. How many versions of libc do you have installed simultaneously? (docker users pls don’t respond)
What are you doing to get into dependency hell? Never had that problem. Are you running “pip install” in your venv at will?
Anti Commercial-AI license
i reckon that’s mostly it - I get an idea, wanna try something and don’t “reset” to a fresh venv. or I do then forget to reinstall the tools I want to use cause I was “just” using them in this same terminal.
it’s me not python. though it’s funny getting a few replies with different solutions. like in PERL, tmtowtdi.
thanks for your reply!
6 years of python and I’ve never really had that problem, also working on larger projects. Use poetry or uv and you’ll probably be fine. Unless you’re doing something strange with your dependencies. The only thing I would say is non trivial is updating the dependencies. And if a library has a bug or something you have to downgrade for. You can specify dev dependencies for notebooks and such. I’ve not heard of mlflow having a problem with a manager. Perhaps you’re in a cloud environment and don’t have access to poetry for example?
never used poetry. just venv, virtualenv and such. I guess I just don’t know the current era’s idiomatic way of doing things. I’m more familiar with java/mvn, rust, etc. It seems like every manning book on a pythonic tool has a different way the author setup the env. to be expected sure. I just need to grok and settle into my own. :)
just learning is all. :) appreciate the reply!
Poetry/uv is similar to Rust’s cargo. You specify your direct dependencies in a TOML file and their version constraints and the tool takes care of the rest.
That’s one of the main pain points of Python. Something like poetry or uv can help but the python ecosystem is just unstable. It’s very hard to update a dependancy. You project might only run with very specific versions of libraries and the language itself.
I honestly hate that about python and I don’t think python is good for developing large software. Even js/ts is a much better ecosystem. (I’m bracing for the hate)
hrm. yeah. I am “trying on” some basic ml models (just simple classifications) into our workflow and while I can whip one up with scikit, a notebook, and serve it with flask/fast - there is no way I can build a workflow tenable for others who are more “data analytics” than coders. The ecosystems for low/no code model generation (knime etc) with handoff to engineers like me to serve is … young.
If you use conda, I suggest using pixi as a project manager. It has lock files which will fix dependencies, and it can activate your enviroments with scripts and variables defined in your pyproject.toml. It has been so much better than using conda directly: https://pixi.sh/
wreck is a dependencies manager, which is venv aware, BUT is not a venv manager nor does it put dependencies into
pyproject.toml. Sticks with good ol’ requirement files.Assumes, for each package, it’s normal to be working with several venv.
Syncs the dependencies intended for the same venv.
req fix --venv-relpath='.venv' req fix --venv-relpath='.doc/.venv'Across many packages, unfortunately have to resort to manually sync’ing dependencies.
Lessons learned
have very recently ceased putting build requirements into requirement files. The build requirements’ transitive dependencies should be in the requirement files. wreck does not support this yet. i’m manually removing dependencies like: build wheel setuptools-scm (setuptools and pip already filtered out) and click.
syncing across packages is really time consuming. Go thru this pain, then no dependency hell.
Wrote wreck cuz all the other options were combining requirements management with everything including the bathroom sink. build backends … venv management … everything goes into pyproject.toml. All these ideas seem to just compound the learning curve.
Less is more especially when it comes to learning curve.
thanks! I will check it out!
You’ll likely also want to check out
rufffor linting and formatting, by the same company that makesuv. It doesn’t enable a lot of lints by default, but there’s a long list of checks to enable.They also have a typechecker,
ty, which is still in early alpha. If it’s as good as their other tools I expect it to become the standard for typechecking Python. Currently you’ll likely want to go withpyrightfor that.