Xiang ZHU

Python DevOps

5 minute read

Lint

pylint

As pylint has too many options, it recommends to use the pylint config file:

# file ~/.pylintrc, can be generated by pylint --generate-rcfile

[MASTER]

[MESSAGES CONTROL]
disable=
    C0116, # Missing function or method docstring (missing-function-docstring)
    W1203, # Use lazy % formatting in logging functions (logging-fstring-interpolation)

[format]
max-line-length = 88

[MISCELLANEOUS]
# List of note tags to take in consideration, separated by a comma.
notes=FIXME

[VARIABLES]

# List of additional names supposed to be defined in builtins. Remember that
# you should avoid defining new builtins when possible.
additional-builtins=
    spark

But we can also ignore some warnings directly in the pylint command:

pylint . -j 0 --disable=C0116,W1203

To show all the inline ignored pylint alerts: pylint --enable=suppressed-message

flake8

# ignore W503 because of black format. BTW, flake8 also has W504 which is in contrary to W503.
flake8 . \
  --exclude=venv \
  --extend-ignore=E203,W503 \
  --max-line-length=88 \
  --max-complexity=7 \
  --show-source \
  --statistics \
  --count \
  --jobs=auto

flake8 [a_file_path]

To show all the inline ignored flake8 alerts: flake8 --disable-noqa

There’s a very nice flake8 plugin called flake8-cognitive-complexity which checks the Cognitive Complexity in addition to the Cyclomatic Complexity provided by flake8 out of the box. We dont need to add extra parameter to use the Cognitive Complexity in flake8, it’s set to --max-cognitive-complexity=7 by default once the plugin is installed. By the way, Sonar sets the Cognitive Complexity threshold to 15 by default.

bandit

The bandit config file format is not well documented, I passed a lot of time to test the config.

$ cat .bandit
# https://github.com/PyCQA/bandit/issues/400
exclude_dirs:
  - "./venv/*"

# https://github.com/PyCQA/bandit/pull/633
assert_used:
  skips:
    - "*/*_test.py"
    - "*/test_*.py"

# without specifying -c ./bandit, it doesn't work
$ bandit . -r -c ./.bandit

ossaudit

ossaudit uses Sonatype OSS Index to audit Python packages for known vulnerabilities.

It can check installed packages and/or packages specified in dependency files. The following formats are supported with dparse:

  • PIP requirement files
  • Pipfile
  • Pipfile.lock
  • tox.ini
  • conda.yml
# check installed packages and packages listed in two requirements files
$ ossaudit --installed --file requirements.txt --file requirements-dev.txt
Found 0 vulnerabilities in 214 packages

Github has already provided, free of charge, the vulnerable dependencies alert.

mypy

For projects having sqlalchemy, we often install the sqlalchemy-stubs plugin as sqlalchemy uses some dynamic classes.

And also django-stubs

mypy config file:

[mypy]
ignore_missing_imports = True # We recommend using this approach only as a last resort: it's equivalent to adding a # type: ignore to all unresolved imports in your codebase.
plugins = sqlmypy # sqlalchemy-stubs
exclude = (?x)(
    ^venv
    | ^build
  )

running mypy:

mypy .
mypy . --exclude [a regular expression that matches file path]
mypy . --exclude venv[//] # exclude venv folder under the root

ignore lint error in one line

linter ingore in one line
pylint (2 spaces)# pylint: disable={errorIdentifier}
flake8 (2 spaces)# noqa: {errorIdentifier}
bandit (2 spaces)# nosec
mypy (2 spaces)# type: ignore
multiple linters (2 spaces)# type: ignore # noqa: {errorIdentifier} # pylint: disable={errorIdentifier}

Format

isort

isort . --profile=black --virtual-env=venv --recursive --check-only
isort . --profile=black --virtual-env=venv --recursive
isort [a_file_path]

Be very careful with isort, it’s not uncompromising, especially for some codes that dynamically import some modules inside a function instead of from the beginning of a file. People use often this to avoid circular import problem. Always run the tests after the isort.

black

black . --check
black .
black [a_file_path]

Using black with other tools: https://black.readthedocs.io/en/stable/guides/using_black_with_other_tools.html

VSCode

Just my 2 cents, try the errorlens extension in VSCode, it will lint all the warnings/errors on live when coding, it’s really cool.

And don’t forget to install the official SonarLint extension, it will give you extra lint. It eats a lot of memory with its java processes nevertheless.

Git pre-commit

https://pre-commit.com/

“Git hook scripts are useful for identifying simple issues before submission to code review. We run our hooks on every commit to automatically point out issues in code such as missing semicolons, trailing whitespace, and debug statements. By pointing these issues out before code review, this allows a code reviewer to focus on the architecture of a change while not wasting time with trivial style nitpicks.”

Online examples

pylint github pre-commit-config.yaml

Create a file named .pre-commit-config.yaml to the root of your project

You could also add pytest (or unittest, or nose, etc.) hook in the pre-commit to make sure all the tests (mostly the unit tests only) are passed before each commit.

repos:
  - repo: local
    hooks:
      - id: pylint
        name: pylint
        stages: [commit]
        language: system
        entry: pylint --disable=C0116 --ignore=venv
        types: [python]

      - id: flake8
        name: flake8
        stages: [commit]
        language: system
        entry: flake8
        types: [python]

      - id: bandit
        name: bandit
        stages: [commit]
        language: system
        entry: bandit
        types: [python]

      - id: mypy
        name: mypy
        stages: [commit]
        language: system
        entry: mypy
        types: [python]

      - id: isort
        name: isort
        stages: [commit]
        language: system
        entry: isort --profile=black --virtual-env=venv
        types: [python]

      - id: black
        name: black
        stages: [commit]
        language: system
        entry: black
        types: [python]

Install the git hook scripts

$ pre-commit install
pre-commit installed at .git/hooks/pre-commit

$ pre-commit install --hook-type post-merge
pre-commit installed at .git/hooks/post-merge

$ pre-commit install --hook-type pre-merge-commit
pre-commit installed at .git/hooks/pre-merge-commit

You could also run pre-commit install --hook-type pre-push to register pre-push hooks.

(optional) Run against all the files

“it’s usually a good idea to run the hooks against all of the files when adding new hooks (usually pre-commit will only run on the changed files during git hooks)”

pre-commit run --all-files

Git commit

Each time we use git commit to stage some files, these files will be sent to pre-commit to be checked against to the hooks defined in .pre-commit-config.yaml.

Temporarily disabling hooks

The official doc gives the example how to disable explicitly hooks by hooks’ ids: SKIP=flake8 git commit -m "foo", but if you want to disable completely all the hooks, an easy way might be found here by using git commit --no-verify or its shortcut git commit -n. If you use pre-commit during push, you can disable pre-commit during push by git push --no-verify or git push -n.

Automatically enabling pre-commit on repositories

https://pre-commit.com/#automatically-enabling-pre-commit-on-repositories

Usage in continuous integration

https://pre-commit.com/#usage-in-continuous-integration

Leave a comment

You may also enjoy