Python Lint And Format
Lint
pylint
As pylint has too many options, it’s recommended to use the pylint config file:
# file ~/.pylintrc, can be generated by pylint --generate-rcfile
[MASTER]
[MESSAGES CONTROL]
disable=
C0116, # Missing function or method docstring (missing-function-docstring)
W1203, # Use lazy % formatting in logging functions (logging-fstring-interpolation)
[format]
max-line-length = 88
[MISCELLANEOUS]
# List of note tags to take in consideration, separated by a comma.
notes=FIXME
[VARIABLES]
# List of additional names supposed to be defined in builtins. Remember that
# you should avoid defining new builtins when possible.
additional-builtins=
spark
But we can also ignore some warnings directly in the pylint command:
pylint . -j 0 --disable=C0116,W1203
To show all the inline ignored pylint alerts: pylint --enable=suppressed-message
Ignore Unused Argument given a Function Name Expression
Use dummy variable to ignore the Pylint warning on unused-argument.
flake8
# ignore W503 because of black format. BTW, flake8 also has W504 which is in contrary to W503.
# ignore E501, line too long because we have the same check at Pylint side already.
flake8 . \
--exclude=venv \
--extend-ignore=E203,E501,W503, \
--max-complexity=7 \
--show-source \
--statistics \
--count \
--jobs=auto
flake8 [a_file_path]
To show all the inline ignored flake8 alerts: flake8 --disable-noqa || true
There’s a very nice flake8 plugin called flake8-cognitive-complexity which checks the Cognitive Complexity in addition to the Cyclomatic Complexity provided by flake8 out of the box. We dont need to add extra parameter to use the Cognitive Complexity in flake8, it’s set to --max-cognitive-complexity=7
by default once the plugin is installed. By the way, Sonar sets the Cognitive Complexity threshold to 15 by default.
To fix imported but not used
error in __init__.py
file, could by all attribute (the most elegant) or by –per-file-ignores.
bandit
The bandit config file format is not well documented, I passed a lot of time to test the config.
$ cat .bandit
# https://github.com/PyCQA/bandit/issues/400
exclude_dirs:
- "./venv/*"
# https://github.com/PyCQA/bandit/pull/633
assert_used:
skips:
- "*/*_test.py"
- "*/test_*.py"
# without specifying -c ./bandit, it doesn't work
$ bandit . -r -c ./.bandit
ossaudit
ossaudit uses Sonatype OSS Index to audit Python packages for known vulnerabilities.
It can check installed packages and/or packages specified in dependency files. The following formats are supported with dparse:
- PIP requirement files
- Pipfile
- Pipfile.lock
- tox.ini
- conda.yml
# check installed packages and packages listed in two requirements files
$ ossaudit --installed --file requirements.txt --file requirements-dev.txt
Found 0 vulnerabilities in 214 packages
Github has already provided, free of charge, the vulnerable dependencies alert.
mypy
For projects having sqlalchemy, we often install the sqlalchemy-stubs
plugin as sqlalchemy uses some dynamic classes.
And also django-stubs, pandas-stubs, types-setuptools, types-requests etc.
[mypy]
ignore_missing_imports = True # We recommend using this approach only as a last resort: it's equivalent to adding a # type: ignore to all unresolved imports in your codebase.
plugins = sqlmypy # sqlalchemy-stubs
exclude = (?x)(
^venv
| ^build
)
running mypy:
mypy .
mypy . --exclude [a regular expression that matches file path]
mypy . --exclude venv[//] # exclude venv folder under the root
When using mypy, it would be better to use mypy against to all files in the project, but ont some of them,
ignore lint error in one line
linter | ignore in one line |
---|---|
pylint | (2 spaces)# pylint: disable={errorIdentifier} |
flake8 | (2 spaces)# noqa: {errorIdentifier} |
bandit | (2 spaces)# nosec |
mypy | (2 spaces)# type: ignore |
multiple linters | (2 spaces)# type: ignore # noqa: {errorIdentifier} # pylint: disable={errorIdentifier} |
To ignore Pylint within a code block
# https://stackoverflow.com/a/48836605/5095636
import sys
sys.path.append("xx/xx")
# pylint: disable=wrong-import-position
from x import ( # noqa: E402
a,
b,
)
from y import c # noqa: E402
# pylint: enable=wrong-import-position
Format
isort
isort . --profile=black --virtual-env=venv --recursive --check-only
isort . --profile=black --virtual-env=venv --recursive
isort [a_file_path]
Be very careful with isort, it’s not uncompromising, especially for some codes that dynamically import some modules inside a function instead of from the beginning of a file. People use often this to avoid circular import problem. Always run the tests after the isort.
black
black . --check
black .
black [a_file_path]
Using black with other tools: https://black.readthedocs.io/en/stable/guides/using_black_with_other_tools.html
VSCode
Just my 2 cents, try the errorlens extension in VSCode, it will lint all the warnings/errors on live when coding, it’s really cool.
And don’t forget to install the official SonarLint extension, it will give you extra lint. It eats a lot of memory with its java processes nevertheless.
"isort.args": [
"--profile",
"black"
],
"python.formatting.provider": "none",
"[python]": {
"editor.defaultFormatter": "ms-python.black-formatter",
"editor.formatOnSave": true,
"editor.codeActionsOnSave": {
// "source.organizeImports": true
},
},
"python.linting.banditEnabled": true,
"python.linting.banditArgs": [
"-r",
"-c",
"~/pyproject.toml"
],
"python.linting.ignorePatterns": [
".vscode/*.py",
"**/site-packages/**/*.py",
"venv/"
],
"python.linting.mypyEnabled": true,
"python.linting.mypyArgs": [
"--follow-imports=silent",
"--ignore-missing-imports",
"--show-column-numbers",
"--no-pretty",
"--warn-return-any",
"--warn-unused-configs",
"--show-error-codes"
],
"sonarlint.connectedMode.connections.sonarqube": [
{
"serverUrl": "https://sonar.xxx",
"connectionId": "sonar.xxx"
}
],
"[json]": {
"editor.defaultFormatter": "esbenp.prettier-vscode",
// "editor.defaultFormatter": "esbenp.prettier-vscode",
"editor.formatOnSave": true
},
"[jsonc]": {
"editor.defaultFormatter": "vscode.json-language-features"
},
pyproject.toml
pyproject.toml
is the new standard in Python introduced by PEP 518 (2016) for build system requirements, PEP 621 (2020) for project metadata, and PEP 660 (2021) for wheel based editable installs.
It’s fun to know why Python authority chose this name, and very interesting to understand their POV of different file formats .
All the the major tools (setuptools, pip-tools, poetry) support this new standard, and the repo awesome-pyproject maintains a list of Python tools which are compatible to pyproject.toml
.
We cannot officially declare flake8 config in pyproject.toml.
Hereunder an example of its content for the lint part.
[tool.isort]
profile = "black"
[tool.mypy]
ignore_missing_imports = true
warn_return_any = true
warn_unused_configs = true
show_error_codes = true
# disallow_untyped_defs = true
# strict = true
exclude = [
"^venv/", # we don't need to exclude `.venv` in mypy as hidden folders are excluded by default
"^build/",
"^_local_test/",
]
[tool.bandit]
# we dont need to exclude `.venv` in bandit as it uses wildcast here
exclude_dirs = ["venv", "_local_test"]
[tool.bandit.assert_used]
skips = ["*/*_test.py", "*/test_*.py"]
[tool.pylint.main]
# ! type to use pyspark-stubs
# extension-pkg-allow-list = ["pyspark"]
# ignored-modules = ["pyspark"]
jobs = 0
# [tool.pylint.typecheck]
# # ! type to use pyspark-stubs
# generated-members = ["pyspark.sql.functions"]
[tool.pylint.basic]
good-names = [
"df" # for dataframe
]
[tool.pylint.variables]
# List of additional names supposed to be defined in builtins. Remember that
# you should avoid defining new builtins when possible.
additional-builtins = ["spark"]
[tool.pylint."messages control"]
disable = [
"missing-class-docstring",
"missing-function-docstring",
"logging-fstring-interpolation",
]
[tool.pylint.miscellaneous]
notes = ["FIXME"]
[tool.pylint.format]
max-line-length = 88
expected-line-ending-format = "LF"
# the default doesn't ignore comment line with words between `#` and `http` like:
# the url is https://pylint.pycqa.org/en/latest/user_guide/configuration/all-options.html#ignore-long-lines
ignore-long-lines = "^\\s*(#)+.*<?https?://"
[tool.pytest.ini_options]
addopts="""
-v -s
--cov {source_folder}
--cov-report=html
--cov-report=xml
--junitxml=junit/test-results.xml
--cov-report=term-missing:skip-covered
--cov-fail-under=95
"""
Git pre-commit
“Git hook scripts are useful for identifying simple issues before submission to code review. We run our hooks on every commit to automatically point out issues in code such as missing semicolons, trailing whitespace, and debug statements. By pointing these issues out before code review, this allows a code reviewer to focus on the architecture of a change while not wasting time with trivial style nitpicks.”
pip install pre-commit
pre-commit install
# install the script along with the hook environments in one command
# https://pre-commit.com/index.html#pre-commit-install-hooks
pre-commit install --install-hooks
# Auto-update pre-commit config to the latest repos' versions.
pre-commit autoupdate
# Clean out cached pre-commit files.
pre-commit clean
# Clean unused cached repos.
pre-commit gc
# Run single check
pre-commit run black
# continuous integration
# https://pre-commit.com/index.html#usage-in-continuous-integration
pre-commit run --all-files
# check only files which have changed
pre-commit run --from-ref origin/HEAD --to-ref HEAD
# Azure pipeline example with cache
https://pre-commit.com/index.html#azure-pipelines-example
# automatically enabling pre-commit on repositories
# https://pre-commit.com/index.html#automatically-enabling-pre-commit-on-repositories
git config --global init.templateDir ~/.git-template
pre-commit init-templatedir ~/.git-template
Online examples
pylint github pre-commit-config.yaml
Create a file named .pre-commit-config.yaml
to the root of your project
Although each lint has its own config to exclude some files from checking, pre-commit also has the key exclude with list value or regex to exclude file from sending to linter.
language: system
means using the executables from the same environment of current Python interpreter.
When using mypy in pre-commit, it would be better run pre-commit run --all-files
, mypy doesn’t work well with only diff files sent by pre-commit run --from-ref origin/${pullrequest_target_branch_name} --to-ref HEAD
.
# Installation:
# pip install pre-commit
# pre-commit install
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.3.0
hooks:
- id: check-json
exclude: devcontainer.json
- id: check-yaml
- id: check-toml
- id: end-of-file-fixer
- id: trailing-whitespace
- id: debug-statements
- id: requirements-txt-fixer
- id: detect-private-key
- id: mixed-line-ending
args: ["--fix=lf"]
- id: check-added-large-files
- id: no-commit-to-branch
- repo: https://github.com/Lucas-C/pre-commit-hooks
rev: v1.3.1
hooks:
- id: forbid-crlf
- id: remove-crlf
- id: forbid-tabs
- id: remove-tabs
- repo: https://github.com/pre-commit/mirrors-prettier
rev: v3.0.0-alpha.1
hooks:
- id: prettier
- repo: https://github.com/pre-commit/pygrep-hooks
rev: v1.9.0
hooks:
- id: python-check-blanket-type-ignore
- id: python-check-mock-methods
- id: python-no-log-warn
- id: python-use-type-annotations
- repo: https://github.com/asottile/pyupgrade
rev: v3.1.0
hooks:
- id: pyupgrade
- repo: local
hooks:
- id: isort
name: isort
entry: isort
language: system
types: [python]
- id: black
name: black
entry: black
language: system
types: [python]
- id: bandit
name: bandit
entry: bandit
language: system
types: [python]
args:
- -c
- pyproject.toml
- id: pylint
name: pylint
entry: pylint
language: system
types: [python]
- id: flake8
name: flake8
entry: flake8
language: system
types: [python]
- id: mypy
name: mypy
language: system
entry: mypy
types: [python]
- id: pytest
name: pytest
types: [python]
entry: pytest
language: system
pass_filenames: false
always_run: true
Be aware that especially in a local environment, we often use venv, in such case, it would be better to use above system level lint executables instead of below public ones, the checks will be more accurate.
# example of using online linters
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.3.0
hooks:
- id: check-json
- id: check-yaml
- id: check-toml
- id: end-of-file-fixer
- id: trailing-whitespace
- id: debug-statements
- id: requirements-txt-fixer
- id: detect-private-key
- id: mixed-line-ending
args: ['--fix=lf']
- id: check-added-large-files
- id: no-commit-to-branch
- repo: https://github.com/Lucas-C/pre-commit-hooks
rev: v1.3.1
hooks:
- id: forbid-crlf
- id: remove-crlf
- id: forbid-tabs
- id: remove-tabs
- repo: https://github.com/psf/black
rev: 22.8.0
hooks:
- id: black
name: "Format with Black"
- repo: https://github.com/pycqa/isort
rev: 5.10.1
hooks:
- id: isort
args: ["--profile", "black"]
- repo: https://github.com/pycqa/flake8
rev: 5.0.4
hooks:
- id: flake8
additional_dependencies:
- flake8-bugbear
- flake8-comprehensions
- flake8-simplify
- repo: https://github.com/pre-commit/mirrors-pylint
rev: "v3.0.0a5"
hooks:
- id: pylint
- repo: https://github.com/pre-commit/mirrors-mypy
# it might be better to use local venv installed mypy because it has access to all the modules installed in the venv
rev: v0.981
hooks:
- id: mypy
additional_dependencies:
# just for example
- types-dataclasses >= 0.1.3
- click >= 8.1.0
- repo: https://github.com/Lucas-C/pre-commit-hooks-bandit
rev: v1.0.5
hooks:
- id: python-bandit-vulnerability-check
args:
- --recursive
- .
- -c
- ./.bandit
- repo: https://github.com/pre-commit/mirrors-prettier
rev: v2.7.1
hooks:
- id: prettier
- repo: local
hooks:
- id: bandit
name: local bandit
entry: bandit
language: python
language_version: python3
types: [python]
Install the git hook scripts
$ pre-commit install
pre-commit installed at .git/hooks/pre-commit
$ pre-commit install --hook-type post-merge
pre-commit installed at .git/hooks/post-merge
$ pre-commit install --hook-type pre-merge-commit
pre-commit installed at .git/hooks/pre-merge-commit
You could also run pre-commit install --hook-type pre-push
to register pre-push hooks.
Run against all the files
“it’s usually a good idea to run the hooks against all of the files when adding new hooks (usually pre-commit will only run on the changed files during git hooks)”
pre-commit run --all-files
Run for changed files only in CI
Please check also this official doc.
git fetch origin
pre-commit run --from-ref origin/${pullrequest_target_branch_name} --to-ref HEAD
When using mypy, it would be better to use mypy against to all files in the project, but not the changed one only.
Git commit
Each time we use git commit to stage some files, these files will be sent to pre-commit to be checked against to the hooks defined in .pre-commit-config.yaml
.
Temporarily disabling hooks
The official doc gives the example how to disable explicitly hooks by hooks’ ids: SKIP=flake8 git commit -m "foo"
, but if you want to disable completely all the hooks, an easy way might be found here by using git commit --no-verify
or its shortcut git commit -n
. If you use pre-commit during push, you can disable pre-commit during push by git push --no-verify
or git push -n
.
Automatically enabling pre-commit on repositories
https://pre-commit.com/#automatically-enabling-pre-commit-on-repositories
Leave a comment