Python Lint And Format
Lint
pylint
As pylint has too many options, it recommends to use the pylint config file:
# file ~/.pylintrc, can be generated by pylint --generate-rcfile
[MESSAGES CONTROL]
# Enable the message, report, category or checker with the given id(s). You can
# either give multiple identifier separated by comma (,) or put this option
# multiple time.
#enable=
# Disable the message, report, category or checker with the given id(s). You
# can either give multiple identifier separated by comma (,) or put this option
# multiple time (only on the command line, not in the configuration file where
# it should appear only once).
# C116 for missing-function-docstring
disable=C0116
But we can also ignore some warnings directly in the pylint command:
pylint . -j 0 --disable=C0116,C0330,C0326
flake8
# ignore W503 because of black format. BTW, flake8 also has W504 which is in contrary to W503.
flake8 . \
--exclude=venv \
--extend-ignore=E203,W503 \
--max-line-length=88 \
--max-complexity=7 \
--show-source \
--statistics \
--count \
--jobs=auto
flake8 [a_file_path]
There’s a very nice flake8 plugin called flake8-cognitive-complexity which checks the Cognitive Complexity in addition to the Cyclomatic Complexity provided by flake8 out of the box. We dont need to add extra parameter to use the Cognitive Complexity in flake8, it’s set to --max-cognitive-complexity=7 by default once the plugin is installed. By the way, Sonar sets the Cognitive Complexity threshold to 15 by default.
bandit
The bandit config file format is not well documented, I passed a lot of time to test the config.
$ cat .bandit
# https://github.com/PyCQA/bandit/issues/400
exclude_dirs:
- "./venv/*"
# https://github.com/PyCQA/bandit/pull/633
assert_used:
skips:
- "*/*_test.py"
- "*/test_*.py"
# without specifying -c ./bandit, it doesn't work
$ bandit . -r -c ./.bandit
ossaudit
ossaudit uses Sonatype OSS Index to audit Python packages for known vulnerabilities.
It can check installed packages and/or packages specified in dependency files. The following formats are supported with dparse:
- PIP requirement files
- Pipfile
- Pipfile.lock
- tox.ini
- conda.yml
# check installed packages and packages listed in two requirements files
$ ossaudit --installed --file requirements.txt --file requirements-dev.txt
Found 0 vulnerabilities in 214 packages
Github has already provided, free of charge, the vulnerable dependencies alert.
mypy
For projects having sqlalchemy, we often install the sqlalchemy-stubs plugin as sqlalchemy uses some dynamic classes.
And also django-stubs
[mypy]
ignore_missing_imports = True # We recommend using this approach only as a last resort: it's equivalent to adding a # type: ignore to all unresolved imports in your codebase.
plugins = sqlmypy # sqlalchemy-stubs
exclude = (?x)(
^venv
| ^build
)
running mypy:
mypy .
mypy . --exclude [a regular expression that matches file path]
mypy . --exclude venv[//] # exclude venv folder under the root
ignore lint error in one line
| linter | ingore in one line |
|---|---|
| pylint | (2 spaces)# pylint: disable={errorIdentifier} |
| flake8 | (2 spaces)# noqa: {errorIdentifier} |
| bandit | (2 spaces)# nosec |
| mypy | (2 spaces)# type: ignore |
| multiple linters | (2 spaces)# type: ignore # noqa: {errorIdentifier} # pylint: disable={errorIdentifier} |
Format
isort
isort . --profile=black --virtual-env=venv --recursive --check-only
isort . --profile=black --virtual-env=venv --recursive
isort [a_file_path]
Be very careful with isort, it’s not uncompromising, especially for some codes that dynamically import some modules inside a function instead of from the beginning of a file. People use often this to avoid circular import problem. Always run the tests after the isort.
black
black . --check
black .
black [a_file_path]
Using black with other tools: https://black.readthedocs.io/en/stable/guides/using_black_with_other_tools.html
VSCode
Just my 2 cents, try the errorlens extension in VSCode, it will lint all the warnings/errors on live when coding, it’s really cool.
And don’t forget to install the official SonarLint extension, it will give you extra lint. It eats a lot of memory with its java processes nevertheless.
Git pre-commit
“Git hook scripts are useful for identifying simple issues before submission to code review. We run our hooks on every commit to automatically point out issues in code such as missing semicolons, trailing whitespace, and debug statements. By pointing these issues out before code review, this allows a code reviewer to focus on the architecture of a change while not wasting time with trivial style nitpicks.”
Online examples
pylint github pre-commit-config.yaml
Create a file named .pre-commit-config.yaml to the root of your project
You could also add pytest (or unittest, or nose, etc.) hook in the pre-commit to make sure all the tests (mostly the unit tests only) are passed before each commit.
repos:
- repo: local
hooks:
- id: pylint
name: pylint
stages: [commit]
language: system
entry: pylint --disable=C0116 --ignore=venv
types: [python]
- id: flake8
name: flake8
stages: [commit]
language: system
entry: flake8
types: [python]
- id: bandit
name: bandit
stages: [commit]
language: system
entry: bandit
types: [python]
- id: mypy
name: mypy
stages: [commit]
language: system
entry: mypy
types: [python]
- id: isort
name: isort
stages: [commit]
language: system
entry: isort --profile=black --virtual-env=venv
types: [python]
- id: black
name: black
stages: [commit]
language: system
entry: black
types: [python]
Install the git hook scripts
$ pre-commit install
pre-commit installed at .git/hooks/pre-commit
$ pre-commit install --hook-type post-merge
pre-commit installed at .git/hooks/post-merge
$ pre-commit install --hook-type pre-merge-commit
pre-commit installed at .git/hooks/pre-merge-commit
You could also run pre-commit install --hook-type pre-push to register pre-push hooks.
(optional) Run against all the files
“it’s usually a good idea to run the hooks against all of the files when adding new hooks (usually pre-commit will only run on the changed files during git hooks)”
pre-commit run --all-files
Git commit
Each time we use git commit to stage some files, these files will be sent to pre-commit to be checked against to the hooks defined in .pre-commit-config.yaml.
Temporarily disabling hooks
The official doc gives the example how to disable explicitly hooks by hooks’ ids: SKIP=flake8 git commit -m "foo", but if you want to disable completely all the hooks, an easy way might be found here by using git commit --no-verify or its shortcut git commit -n. If you use pre-commit during push, you can disable pre-commit during push by git push --no-verify or git push -n.
Automatically enabling pre-commit on repositories
https://pre-commit.com/#automatically-enabling-pre-commit-on-repositories
Leave a comment