I love decoupling. This makes the project maintaining easier. We have 2 main ways to do it:
This article about creating python package without pain.
Most python packages which you are using contain setup.py file in its root directory. This file describes package name, version, requirements (required third-party packages), package content and some optional information. Just call
setuptools.setup(...) with this info as kwargs. It’s enough for package distribution. If you have setup.py then you already can distribute it. For example, upload into pypi.org.
Pip – de facto standard approach for installing python packages in your system. Simple and well known.
By default, pip installs all packages for all users in the system and requires root privileges for it. Don’t sudo pip. Use virtualenv to install packages into isolated environments. Besides security troubles, some packages may have incompatible required versions of some mutual package.
Sometimes you want to get actual package version directly from your other repository. This is very useful for non-distributable projects. Setuptools doesn’t support it, but you can do it via pip:
pip install -e firstname.lastname@example.org:...git@master#egg=package_name
And you can pin this and any other requirements into requirements.txt:
-e email@example.com:...git@master#egg=package_name ... deal Django>=1.11 ...
Also, pip supports constraints.txt with the same syntax for pinning versions for optional dependencies:
To install these dependencies just pass them into pip:
pip install -r requirements.txt -c constraints.txt
Requirements.txt is very useful when you don’t want to create setup.py for your internal projects.
In most commercial projects you have at least 2 environments:
Pip-tools provide some tools for this conception.
Pip developers decided to improve requirements.txt for grouping dependencies and enabling native support for version locking. As a result, they have created Pipfile specification and pipenv – a tool for working with it. Pipenv can lock versions in [Pipfile.lock], manage virtual environments, install and resolve dependencies. So cool, but for distributable packages, you must duplicate main dependencies into setup.py.
pyproject.tomlgenerating. Yeah, if you fork and improve some project, you must make sdist for any changes and bump version for all projects that depend on it. Or manually convert project’s
For backward compatibility, you can generate setup.py and requirements.txt from pyproject.toml via poetry-setup.
If pyproject.toml is cool, why only poetry use it? Not only. Flit supports pyproject.toml too. This is a very simple tool with only 4 commands:
That’s all. And enough in common cases. Flit uses pip for packages installation. And flit listed in alternatives by PyPA. As in poetry, you need to manage virtual environments by other tools. But this package has a significant disadvantage: flit can’t lock dependencies.
All solutions below have some problems. Let’s fix it!
As we remember, with pipenv we need to duplicate all requirements in old format for setup.py. Let’s improve it! I’ve created install-requires project that can help you convert requirements between formats. But which format should you choose?
requirements.txt. This is the most popular requirements format for projects. Anyone can use it as they want.
Pipfile.lock. Pipenv has better requirements lock than pip-tools, and you should use it for better security. But if you want to create a package from a project, then you plan to use this package into other projects. So, if this project depends on more than one package with locked requirements, pipenv can’t resolve these dependencies. For example, one package lock
Django==1.9version, but other uses
Django==1.11. Do not use it for distributable packages: PyPA recommends to place not locked versions into install_requires.
Pipfile. This is our choice. Modern format with some problems, but also with many features. And, most importantly, simple and comfortable. I recommend using it for internal python packages in your company.
Install-requires repository contains example how you can convert requirements from Pipfile to setup.py compatible format on the fly.
Many developers (me too) love poetry because it uses beautiful project metadata describing format as setup.py alternative. But setuptools allows you to use setup.cfg instead of setup.py and it’s also beautiful. Furthermore, isort and flake8 supports setup.cfg too.
So, what’s wrong with setuptools? I think this tool has some problems:
pip freeze) can lock dependencies from its own files. Setuptools can’t. This is because of setuptools for packages, not projects.