Python Lint And Format

January 4, 2021 10 minute read

Lint

pylint

As pylint has too many options, it’s recommended to use the pylint config file:

# file ~/.pylintrc, can be generated by pylint --generate-rcfile

[MASTER]

[MESSAGES CONTROL]
disable=
    C0116, # Missing function or method docstring (missing-function-docstring)
    W1203, # Use lazy % formatting in logging functions (logging-fstring-interpolation)

[format]
max-line-length = 88

[MISCELLANEOUS]
# List of note tags to take in consideration, separated by a comma.
notes=FIXME

[VARIABLES]

# List of additional names supposed to be defined in builtins. Remember that
# you should avoid defining new builtins when possible.
additional-builtins=
    spark

But we can also ignore some warnings directly in the pylint command:

pylint . -j 0 --disable=C0116,W1203

To show all the inline ignored pylint alerts: pylint --enable=suppressed-message

Ignore Unused Argument given a Function Name Expression

Use dummy variable to ignore the Pylint warning on unused-argument.

flake8

# ignore W503 because of black format. BTW, flake8 also has W504 which is in contrary to W503.
# ignore E501, line too long because we have the same check at Pylint side already.
flake8 . \
  --exclude=venv \
  --extend-ignore=E203,E501,W503, \
  --max-complexity=7 \
  --show-source \
  --statistics \
  --count \
  --jobs=auto

flake8 [a_file_path]

To show all the inline ignored flake8 alerts: flake8 --disable-noqa || true

There’s a very nice flake8 plugin called flake8-cognitive-complexity which checks the Cognitive Complexity in addition to the Cyclomatic Complexity provided by flake8 out of the box. We dont need to add extra parameter to use the Cognitive Complexity in flake8, it’s set to --max-cognitive-complexity=7 by default once the plugin is installed. By the way, Sonar sets the Cognitive Complexity threshold to 15 by default.

To fix imported but not used error in __init__.py file, could by all attribute (the most elegant) or by –per-file-ignores.

bandit

The bandit config file format is not well documented, I passed a lot of time to test the config.

$ cat .bandit
# https://github.com/PyCQA/bandit/issues/400
exclude_dirs:
  - "./venv/*"

# https://github.com/PyCQA/bandit/pull/633
assert_used:
  skips:
    - "*/*_test.py"
    - "*/test_*.py"

# without specifying -c ./bandit, it doesn't work
$ bandit . -r -c ./.bandit

ossaudit

ossaudit uses Sonatype OSS Index to audit Python packages for known vulnerabilities.

It can check installed packages and/or packages specified in dependency files. The following formats are supported with dparse:

PIP requirement files
Pipfile
Pipfile.lock
tox.ini
conda.yml

# check installed packages and packages listed in two requirements files
$ ossaudit --installed --file requirements.txt --file requirements-dev.txt
Found 0 vulnerabilities in 214 packages

Github has already provided, free of charge, the vulnerable dependencies alert.

mypy

For projects having sqlalchemy, we often install the sqlalchemy-stubs plugin as sqlalchemy uses some dynamic classes.

And also django-stubs, pandas-stubs, types-setuptools, types-requests etc.

mypy config file:

[mypy]
ignore_missing_imports = True # We recommend using this approach only as a last resort: it's equivalent to adding a # type: ignore to all unresolved imports in your codebase.
plugins = sqlmypy # sqlalchemy-stubs
exclude = (?x)(
    ^venv
    | ^build
  )

running mypy:

mypy .
mypy . --exclude [a regular expression that matches file path]
mypy . --exclude venv[//] # exclude venv folder under the root

When using mypy, it would be better to use mypy against to all files in the project, but ont some of them,

ignore lint error in one line

linter	ignore in one line
pylint	(2 spaces)# pylint: disable={errorIdentifier}
flake8	(2 spaces)# noqa: {errorIdentifier}
bandit	(2 spaces)# nosec
mypy	(2 spaces)# type: ignore
multiple linters	(2 spaces)# type: ignore # noqa: {errorIdentifier} # pylint: disable={errorIdentifier}

To ignore Pylint within a code block

# https://stackoverflow.com/a/48836605/5095636
import sys
sys.path.append("xx/xx")

# pylint: disable=wrong-import-position
from x import (  # noqa: E402
    a,
    b,
)
from y import c  # noqa: E402

# pylint: enable=wrong-import-position

Format

isort

isort . --profile=black --virtual-env=venv --recursive --check-only
isort . --profile=black --virtual-env=venv --recursive
isort [a_file_path]

Be very careful with isort, it’s not uncompromising, especially for some codes that dynamically import some modules inside a function instead of from the beginning of a file. People use often this to avoid circular import problem. Always run the tests after the isort.

black

black . --check
black .
black [a_file_path]

Using black with other tools: https://black.readthedocs.io/en/stable/guides/using_black_with_other_tools.html

VSCode

Just my 2 cents, try the errorlens extension in VSCode, it will lint all the warnings/errors on live when coding, it’s really cool.

And don’t forget to install the official SonarLint extension, it will give you extra lint. It eats a lot of memory with its java processes nevertheless.

  "isort.args": [
    "--profile",
    "black"
  ],
  "python.formatting.provider": "none",
  "[python]": {
    "editor.defaultFormatter": "ms-python.black-formatter",
    "editor.formatOnSave": true,
    "editor.codeActionsOnSave": {
      // "source.organizeImports": true
    },
  },
  "python.linting.banditEnabled": true,
  "python.linting.banditArgs": [
    "-r",
    "-c",
    "~/pyproject.toml"
  ],
  "python.linting.ignorePatterns": [
    ".vscode/*.py",
    "**/site-packages/**/*.py",
    "venv/"
  ],
  "python.linting.mypyEnabled": true,
  "python.linting.mypyArgs": [
    "--follow-imports=silent",
    "--ignore-missing-imports",
    "--show-column-numbers",
    "--no-pretty",
    "--warn-return-any",
    "--warn-unused-configs",
    "--show-error-codes"
  ],
  "sonarlint.connectedMode.connections.sonarqube": [
    {
      "serverUrl": "https://sonar.xxx",
      "connectionId": "sonar.xxx"
    }
  ],
  "[json]": {
    "editor.defaultFormatter": "esbenp.prettier-vscode",
    // "editor.defaultFormatter": "esbenp.prettier-vscode",
    "editor.formatOnSave": true
  },
  "[jsonc]": {
    "editor.defaultFormatter": "vscode.json-language-features"
  },

pyproject.toml

pyproject.toml is the new standard in Python introduced by PEP 518 (2016) for build system requirements, PEP 621 (2020) for project metadata, and PEP 660 (2021) for wheel based editable installs.

It’s fun to know why Python authority chose this name, and very interesting to understand their POV of different file formats .

All the the major tools (setuptools, pip-tools, poetry) support this new standard, and the repo awesome-pyproject maintains a list of Python tools which are compatible to pyproject.toml.

We cannot officially declare flake8 config in pyproject.toml.

Hereunder an example of its content for the lint part.

[tool.isort]
profile = "black"

[tool.mypy]
ignore_missing_imports = true
warn_return_any = true
warn_unused_configs = true
show_error_codes = true
# disallow_untyped_defs = true
# strict = true
exclude = [
    "^venv/", # we don't need to exclude `.venv` in mypy as hidden folders are excluded by default
    "^build/",
    "^_local_test/",
]

[tool.bandit]
# we dont need to exclude `.venv` in bandit as it uses wildcast here
exclude_dirs = ["venv", "_local_test"]
[tool.bandit.assert_used]
skips = ["*/*_test.py", "*/test_*.py"]

[tool.pylint.main]
# ! type to use pyspark-stubs
# extension-pkg-allow-list = ["pyspark"]
# ignored-modules = ["pyspark"]
jobs = 0
# [tool.pylint.typecheck]
# # ! type to use pyspark-stubs
# generated-members = ["pyspark.sql.functions"]
[tool.pylint.basic]
good-names = [
  "df"  # for dataframe
]
[tool.pylint.variables]
# List of additional names supposed to be defined in builtins. Remember that
# you should avoid defining new builtins when possible.
additional-builtins = ["spark"]
[tool.pylint."messages control"]
disable = [
    "missing-class-docstring",
    "missing-function-docstring",
    "logging-fstring-interpolation",
]
[tool.pylint.miscellaneous]
notes = ["FIXME"]
[tool.pylint.format]
max-line-length = 88
expected-line-ending-format = "LF"

# the default doesn't ignore comment line with words between `#` and `http` like:
# the url is https://pylint.pycqa.org/en/latest/user_guide/configuration/all-options.html#ignore-long-lines
ignore-long-lines = "^\\s*(#)+.*<?https?://"

[tool.pytest.ini_options]
addopts="""
    -v -s
    --cov {source_folder}
    --cov-report=html
    --cov-report=xml
    --junitxml=junit/test-results.xml
    --cov-report=term-missing:skip-covered
    --cov-fail-under=95
    """

Git pre-commit

https://pre-commit.com/

“Git hook scripts are useful for identifying simple issues before submission to code review. We run our hooks on every commit to automatically point out issues in code such as missing semicolons, trailing whitespace, and debug statements. By pointing these issues out before code review, this allows a code reviewer to focus on the architecture of a change while not wasting time with trivial style nitpicks.”

pip install pre-commit
pre-commit install

# install the script along with the hook environments in one command
# https://pre-commit.com/index.html#pre-commit-install-hooks
pre-commit install --install-hooks

# Auto-update pre-commit config to the latest repos' versions.
pre-commit autoupdate

# Clean out cached pre-commit files.
pre-commit clean

# Clean unused cached repos.
pre-commit gc

# Run single check
pre-commit run black

# continuous integration
# https://pre-commit.com/index.html#usage-in-continuous-integration
pre-commit run --all-files
# check only files which have changed
pre-commit run --from-ref origin/HEAD --to-ref HEAD

# Azure pipeline example with cache
https://pre-commit.com/index.html#azure-pipelines-example

# automatically enabling pre-commit on repositories
# https://pre-commit.com/index.html#automatically-enabling-pre-commit-on-repositories
git config --global init.templateDir ~/.git-template
pre-commit init-templatedir ~/.git-template

Online examples

pylint github pre-commit-config.yaml

Create a file named `.pre-commit-config.yaml` to the root of your project

Although each lint has its own config to exclude some files from checking, pre-commit also has the key exclude with list value or regex to exclude file from sending to linter.

language: system means using the executables from the same environment of current Python interpreter.

When using mypy in pre-commit, it would be better run pre-commit run --all-files, mypy doesn’t work well with only diff files sent by pre-commit run --from-ref origin/${pullrequest_target_branch_name} --to-ref HEAD.

# Installation:
# pip install pre-commit
# pre-commit install
repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.3.0
    hooks:
      - id: check-json
        exclude: devcontainer.json
      - id: check-yaml
      - id: check-toml
      - id: end-of-file-fixer
      - id: trailing-whitespace
      - id: debug-statements
      - id: requirements-txt-fixer
      - id: detect-private-key
      - id: mixed-line-ending
        args: ["--fix=lf"]
      - id: check-added-large-files
      - id: no-commit-to-branch
  - repo: https://github.com/Lucas-C/pre-commit-hooks
    rev: v1.3.1
    hooks:
      - id: forbid-crlf
      - id: remove-crlf
      - id: forbid-tabs
      - id: remove-tabs
  - repo: https://github.com/pre-commit/mirrors-prettier
    rev: v3.0.0-alpha.1
    hooks:
      - id: prettier
  - repo: https://github.com/pre-commit/pygrep-hooks
    rev: v1.9.0
    hooks:
      - id: python-check-blanket-type-ignore
      - id: python-check-mock-methods
      - id: python-no-log-warn
      - id: python-use-type-annotations
  - repo: https://github.com/asottile/pyupgrade
    rev: v3.1.0
    hooks:
      - id: pyupgrade
  - repo: local
    hooks:
      - id: isort
        name: isort
        entry: isort
        language: system
        types: [python]
      - id: black
        name: black
        entry: black
        language: system
        types: [python]
      - id: bandit
        name: bandit
        entry: bandit
        language: system
        types: [python]
        args:
          - -c
          - pyproject.toml
      - id: pylint
        name: pylint
        entry: pylint
        language: system
        types: [python]
      - id: flake8
        name: flake8
        entry: flake8
        language: system
        types: [python]
      - id: mypy
        name: mypy
        language: system
        entry: mypy
        types: [python]
      - id: pytest
        name: pytest
        types: [python]
        entry: pytest
        language: system
        pass_filenames: false
        always_run: true

Be aware that especially in a local environment, we often use venv, in such case, it would be better to use above system level lint executables instead of below public ones, the checks will be more accurate.

# example of using online linters
repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.3.0
    hooks:
      - id: check-json
      - id: check-yaml
      - id: check-toml
      - id: end-of-file-fixer
      - id: trailing-whitespace
      - id: debug-statements
      - id: requirements-txt-fixer
      - id: detect-private-key
      - id: mixed-line-ending
        args: ['--fix=lf']
      - id: check-added-large-files
      - id: no-commit-to-branch
  - repo: https://github.com/Lucas-C/pre-commit-hooks
    rev: v1.3.1
    hooks:
      - id: forbid-crlf
      - id: remove-crlf
      - id: forbid-tabs
      - id: remove-tabs
  - repo: https://github.com/psf/black
    rev: 22.8.0
    hooks:
      - id: black
        name: "Format with Black"
  - repo: https://github.com/pycqa/isort
    rev: 5.10.1
    hooks:
      - id: isort
        args: ["--profile", "black"]
  - repo: https://github.com/pycqa/flake8
    rev: 5.0.4
    hooks:
      - id: flake8
        additional_dependencies:
          - flake8-bugbear
          - flake8-comprehensions
          - flake8-simplify
  - repo: https://github.com/pre-commit/mirrors-pylint
    rev: "v3.0.0a5"
    hooks:
      - id: pylint
  - repo: https://github.com/pre-commit/mirrors-mypy
    # it might be better to use local venv installed mypy because it has access to all the modules installed in the venv
    rev: v0.981
    hooks:
      - id: mypy
        additional_dependencies:
          # just for example
          - types-dataclasses >= 0.1.3
          - click >= 8.1.0
  - repo: https://github.com/Lucas-C/pre-commit-hooks-bandit
    rev: v1.0.5
    hooks:
      - id: python-bandit-vulnerability-check
        args:
          - --recursive
          - .
          - -c
          - ./.bandit
  - repo: https://github.com/pre-commit/mirrors-prettier
    rev: v2.7.1
    hooks:
      - id: prettier
  - repo: local
    hooks:
    - id: bandit
      name: local bandit
      entry: bandit
      language: python
      language_version: python3
      types: [python]

Install the git hook scripts

$ pre-commit install
pre-commit installed at .git/hooks/pre-commit

$ pre-commit install --hook-type post-merge
pre-commit installed at .git/hooks/post-merge

$ pre-commit install --hook-type pre-merge-commit
pre-commit installed at .git/hooks/pre-merge-commit

You could also run pre-commit install --hook-type pre-push to register pre-push hooks.

Run against all the files

“it’s usually a good idea to run the hooks against all of the files when adding new hooks (usually pre-commit will only run on the changed files during git hooks)”

pre-commit run --all-files

Run for changed files only in CI

Please check also this official doc.

git fetch origin
pre-commit run --from-ref origin/${pullrequest_target_branch_name} --to-ref HEAD

When using mypy, it would be better to use mypy against to all files in the project, but not the changed one only.

Git commit

Each time we use git commit to stage some files, these files will be sent to pre-commit to be checked against to the hooks defined in .pre-commit-config.yaml.

Temporarily disabling hooks

The official doc gives the example how to disable explicitly hooks by hooks’ ids: SKIP=flake8 git commit -m "foo", but if you want to disable completely all the hooks, an easy way might be found here by using git commit --no-verify or its shortcut git commit -n. If you use pre-commit during push, you can disable pre-commit during push by git push --no-verify or git push -n.

Xiang ZHU

Python Lint And Format

Lint

pylint

Ignore Unused Argument given a Function Name Expression

flake8

bandit

ossaudit

mypy

ignore lint error in one line

Format

isort

black

VSCode

pyproject.toml

Git pre-commit

Online examples

Create a file named `.pre-commit-config.yaml` to the root of your project

Install the git hook scripts

Run against all the files

Run for changed files only in CI

Git commit

Temporarily disabling hooks

Automatically enabling pre-commit on repositories

Usage in continuous integration

Share on

Leave a comment

You may also enjoy

Using ast and cst to change Python code

Azure pipeline Windows agent UnicodeEncodeError

Azure pipeline delete blobs from blob storage

Databricks cluster access mode

Xiang ZHU

Lint

pylint

Ignore Unused Argument given a Function Name Expression

flake8

bandit

ossaudit

mypy

ignore lint error in one line

Format

isort

black

VSCode

pyproject.toml

Git pre-commit

Online examples

Create a file named .pre-commit-config.yaml to the root of your project

Install the git hook scripts

Run against all the files

Run for changed files only in CI

Git commit

Temporarily disabling hooks

Automatically enabling pre-commit on repositories

Usage in continuous integration

Share on

Leave a comment

You may also enjoy

Using ast and cst to change Python code

Azure pipeline Windows agent UnicodeEncodeError

Azure pipeline delete blobs from blob storage

Databricks cluster access mode

Create a file named `.pre-commit-config.yaml` to the root of your project