Python Lint And Format

Lint

pylint

As pylint has too many options, it recommends to use the pylint config file:

# file ~/.pylintrc, can be generated by pylint --generate-rcfile

[MESSAGES CONTROL]

# Enable the message, report, category or checker with the given id(s). You can
# either give multiple identifier separated by comma (,) or put this option
# multiple time.
#enable=

# Disable the message, report, category or checker with the given id(s). You
# can either give multiple identifier separated by comma (,) or put this option
# multiple time (only on the command line, not in the configuration file where
# it should appear only once).

# C116 for missing-function-docstring
disable=C0116

But we can also ignore some warnings directly in the pylint command:

pylint . -j 0 --disable=C0116,C0330,C0326

flake8

# ignore W503 because of black format. BTW, flake8 also has W504 which is in contrary to W503.
flake8 . --exclude=venv --extend-ignore=E203,W503 --max-line-length=88 --show-source --statistics --count
flake8 [a_file_path]

bandit

The bandit config file format is not well documented, I passed a lot of time to test the config.

$ cat .bandit
# https://github.com/PyCQA/bandit/issues/400
exclude_dirs:
  - "./venv/*"

# https://github.com/PyCQA/bandit/pull/633
assert_used:
  skips:
    - "*/*_test.py"
    - "*/test_*.py"

# without specifying -c ./bandit, it doesn't work
$ bandit . -r -c ./.bandit

mypy

For projects having sqlalchemy, we often install the sqlalchemy-stubs plugin as sqlalchemy uses some dynamic classes.

And also django-stubs

mypy config file:

[mypy]
ignore_missing_imports = True # We recommend using this approach only as a last resort: it's equivalent to adding a # type: ignore to all unresolved imports in your codebase.
plugins = sqlmypy # sqlalchemy-stubs

running mypy:

mypy .
mypy . --exclude [a regular expression that matches file path]
mypy . --exclude venv[//] # exclude venv folder under the root

Format

isort

isort . --profile=black --virtual-env=venv --recursive --check-only
isort . --profile=black --virtual-env=venv --recursive
isort [a_file_path]

Be very careful with isort, it’s not uncompromising, especially for some codes that dynamically import some modules inside a function instead of from the beginning of a file. People use often this to avoid circular import problem. Always run the tests after the isort.

black

black . --check
black .
black [a_file_path]

Using black with other tools: https://black.readthedocs.io/en/stable/guides/using_black_with_other_tools.html

VSCode

Just my 2 cents, try the errorlens extension in VSCode, it will lint all the warnings/errors on live when coding, it’s really cool.

And don’t forget to install the official SonarLint extension, it will give you extra lint. It eats a lot of memory with its java processes nevertheless.

Git pre-commit

https://pre-commit.com/

“Git hook scripts are useful for identifying simple issues before submission to code review. We run our hooks on every commit to automatically point out issues in code such as missing semicolons, trailing whitespace, and debug statements. By pointing these issues out before code review, this allows a code reviewer to focus on the architecture of a change while not wasting time with trivial style nitpicks.”

Online examples

pylint github pre-commit-config.yaml

Create a file named .pre-commit-config.yaml to the root of your project

You could also add pytest (or unittest, or nose, etc.) hook in the pre-commit to make sure all the tests (mostly the unit tests only) are passed before each commit.

repos:
  - repo: local
    hooks:
      - id: pylint
        name: pylint
        stages: [commit]
        language: system
        entry: pylint --disable=C0116 --ignore=venv
        types: [python]

      - id: flake8
        name: flake8
        stages: [commit]
        language: system
        entry: flake8
        types: [python]

      - id: bandit
        name: bandit
        stages: [commit]
        language: system
        entry: bandit
        types: [python]

      - id: mypy
        name: mypy
        stages: [commit]
        language: system
        entry: mypy
        types: [python]

      - id: isort
        name: isort
        stages: [commit]
        language: system
        entry: isort --profile=black --virtual-env=venv
        types: [python]

      - id: black
        name: black
        stages: [commit]
        language: system
        entry: black
        types: [python]

Install the git hook scripts

$ pre-commit install
pre-commit installed at .git/hooks/pre-commit

$ pre-commit install --hook-type post-merge
pre-commit installed at .git/hooks/post-merge

$ pre-commit install --hook-type pre-merge-commit
pre-commit installed at .git/hooks/pre-merge-commit

You could also run pre-commit install --hook-type pre-push to register pre-push hooks.

(optional) Run against all the files

“it’s usually a good idea to run the hooks against all of the files when adding new hooks (usually pre-commit will only run on the changed files during git hooks)”

pre-commit run --all-files

Git commit

Each time we use git commit to stage some files, these files will be sent to pre-commit to be checked against to the hooks defined in .pre-commit-config.yaml.

Temporarily disabling hooks

The official doc gives the example how to disable explicitly hooks by hooks’ ids: SKIP=flake8 git commit -m "foo", but if you want to disable completely all the hooks, an easy way might be found here by using git commit --no-verify or its shortcut git commit -n. If you use pre-commit during push, you can disable pre-commit during push by git push --no-verify or git push -n.

Automatically enabling pre-commit on repositories

https://pre-commit.com/#automatically-enabling-pre-commit-on-repositories

Usage in continuous integration

https://pre-commit.com/#usage-in-continuous-integration