Skip to content

Trying Python pipreqs and pip-tools#

Relative to pipenv, and poetry, if you're searching for some lightweight python package managers for a small project, I will introduce 2 handy tools for you: pipreqs and pip-tools.

pipreqs#

pipreqs github

Suppose you are onboarded to an existing project where only pip is used. The requirements.txt file is generated by pip freeze. and it contains more than 30 lines of requirements, in addition the team cannot remember the original basic requirements anymore. One of the simple ways to rebuild the basic requirements (the project dependency but not all the underlying dependencies) is to use pipreqs.

How to use:

# First of all, just backup your current requirements.txt
$ mv requirements.txt{,.bck}

# at this moment, there's no more requirements.txt, then run pipreqs
$ pipreqs /home/project/location
Successfully saved requirements file in /home/project/location/requirements.txt

Let's use the --debug option to see what it does in background

# I'm running pipreqs from the root path of a Flask project
$ pipreqs . --debug
DEBUG: Found packages: {'json', 'flask_webtest', 'time', 'requests', 'webtest', 'sys', 'flask', 'os', 'pathlib', 'setuptools', 'unittest', 'werkzeug'}
DEBUG: Found imports: Flask, flask_webtest, Requests, WebTest, Werkzeug
DEBUG: Getting packages information from Local/PyPI
DEBUG: Starting new HTTPS connection (1): pypi.python.org:443
DEBUG: https://pypi.python.org:443 "GET /pypi/flask_webtest/json HTTP/1.1" 301 122
DEBUG: Starting new HTTPS connection (1): pypi.org:443
DEBUG: https://pypi.org:443 "GET /pypi/flask_webtest/json HTTP/1.1" 301 221
DEBUG: https://pypi.org:443 "GET /pypi/Flask-WebTest/json HTTP/1.1" 200 2155DEBUG: Starting new HTTPS connection (1): pypi.python.org:443
DEBUG: https://pypi.python.org:443 "GET /pypi/WebTest/json HTTP/1.1" 301 122DEBUG: Starting new HTTPS connection (1): pypi.org:443
DEBUG: https://pypi.org:443 "GET /pypi/WebTest/json HTTP/1.1" 200 12870

$ cat ./requirements.txt
Werkzeug==1.0.1
Flask==1.1.2
requests==2.25.1
flask_webtest==0.0.9
WebTest==2.0.35

pipreqs has also some other useful options(--no-pin, --force).

pip-tools#

pip-tools github

Another missing feature of the native pip is that pip freeze doesn't provide the packages dependencies. All the packages installed in the venv are listed in a single requirements.txt file, in the same top level with only the version info. Pipenv and Poetry resolve this issue, and introduce some lock system. But they're not the native requirements.txt way. By using pip-tools, we can resolve this issue too and at the same time keep using requirements.txt. I found this tool occasionally by checking Flask project requirements.

The idea of pip-tools is to maintain a project basic dependency in a file called requirements.in, then use pip-tools to generate the requirements.txt file with all the dependencies including the underlying dependencies info inside but in the comments part.

Please be aware that: pip-tools = pip-compile + pip-sync

pip-compile#

I'm running a small Flask project, the only package I need is just a single Flask. Let's see an example of pip-compile by using requirements.in file without setup.py,

(venv)$ cat requirements.in
Flask

(venv)$ pip-compile requirements.in

(venv)$ cat requirements.txt
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile requirements.in
#
click==7.1.2
    # via flask
flask==1.1.2
    # via -r r.in
itsdangerous==1.1.0
    # via flask
jinja2==2.11.3
    # via flask
markupsafe==1.1.1
    # via jinja2
werkzeug==1.0.1
    # via flask

The Solution for with setup.py is here.

pip-sync#

Let's see an example of pip-sync:

# Upgrade Werkzeug to a rc version that is newer that the once listed by requirements.txt
(venv)$ pip install Werkzeug==2.0.0rc2
Collecting Werkzeug==2.0.0rc2
  Downloading Werkzeug-2.0.0rc2-py3-none-any.whl (284 kB)
     |████████████████████████████████| 284 kB 3.3 MB/s
Installing collected packages: werkzeug
  Attempting uninstall: werkzeug
    Found existing installation: Werkzeug 1.0.1
    Uninstalling Werkzeug-1.0.1:
      Successfully uninstalled Werkzeug-1.0.1
Successfully installed werkzeug-2.0.0rc2

# Use pip-sync to downgrade Werkzeug in order to make the venv to have exactly the same version of dependencies listed in requirements.txt
(venv)$ pip-sync
Collecting Werkzeug==1.0.1
  Using cached Werkzeug-1.0.1-py2.py3-none-any.whl (298 kB)
Installing collected packages: Werkzeug
  Attempting uninstall: Werkzeug
    Found existing installation: Werkzeug 2.0.0rc2
    Uninstalling Werkzeug-2.0.0rc2:
      Successfully uninstalled Werkzeug-2.0.0rc2
Successfully installed Werkzeug-1.0.1

pipdeptree for dependency tree#

The pip-tools' github page introduces also a dependency tree generation tool: pipdeptree which is also very nice:

(venv)$ pipdeptree
Flask==1.0
  - click [required: >=5.1, installed: 7.1.2]
  - itsdangerous [required: >=0.24, installed: 1.1.0]
  - Jinja2 [required: >=2.10, installed: 2.11.3]
    - MarkupSafe [required: >=0.23, installed: 1.1.1]
  - Werkzeug [required: >=0.14, installed: 1.0.1]
pip-tools==5.5.0
  - click [required: >=7, installed: 7.1.2]
  - pip [required: >=20.1, installed: 20.2.3]
pipdeptree==2.0.0
  - pip [required: >=6.0.0, installed: 20.2.3]
setuptools==49.2.1

Comments