Github Actions - Python

Setting up pip authentication

PIP_INDEX_URL vs PIP_EXTRA_INDEX_URL

In most cases, when setting up private Python package artifacts (like Azure DevOps Artifacts, JFrog Artifactory, etc.) are configured to mirror the public PyPi. In such scenarios, we only need to use PIP_INDEX_URL to point to these private artifacts.

However, some people might use PIP_INDEX_URL point to the public PyPi, and PIP_EXTRA_INDEX_URL to point to the private artifacts. This approach is not recommended, as it results in the public PyPi searched first, followed by the private artifacts. This poses a security risk where a malicious actor can publish a package with the same name as your private one on the public PyPi.

Auth for Azure DevOps Artifacts

Auth by Azure SPN crendentials

In March 2023, there was a great news that Azure Service Principal was been introduced in Azure DevOps, eliminating the use of service account.

1. Create a service principal in Azure Active Directory.
2. Add the service principal to the Azure DevOps Artifacts feed with Contributor role. Package publishing (twine upload) needs Contributor role, but package installation (pip install) only needs Reader role.

3. Add SPN credentials to Github Secrets with name AZURE_CREDENTIALS, and value in JSON format:

    {
      "clientId": "xxxxx",
      "clientSecret": "xxxxx",
      "subscriptionId": "xxxxx",
      "tenantId": "xxxxx"
    }
   

4. Create env var PIP_INDEX_URL in the workflow, and set it to the Azure DevOps Artifacts feed URL.

    - uses: actions/checkout@v4
   
   
    - name: Setup Python
      uses: actions/setup-python@v4
      with:
        python-version: ${{ matrix.python-version }}
        # see below post of a faster Python cache:
        # https://copdips.com/2023/09/github-actions-cache.html#pip-cache-dir-vs-pip-install-dir
        cache: pip
        cache-dependency-path: requirements/*.txt
   
    - name: Azure Login
      uses: azure/login@v1
      with:
        creds: ${{ secrets.AZURE_CREDENTIALS }}
   
    - name: Setup Python package feed
      run: |
        access_token=$(az account get-access-token | jq .accessToken -r)
   
        # setup pip auth
        echo "PIP_INDEX_URL=https://:$access_token@pkgs.dev.azure.com/{azdo_org_name}/_packaging/{azdo_artifacts_feed_name}/pypi/simple/" >> $GITHUB_ENV
   
        # setup twine auth
        cat > ~/.pypirc <<EOF
        [distutils]
        index-servers={azdo_artifacts_feed_name}
        [{azdo_artifacts_feed_name}]
        repository=https://pkgs.dev.azure.com/{azdo_org_name}/_packaging/{azdo_artifacts_feed_name}/pypi/upload
        username=build
        password=$access_token
        EOF
   
        # setup access token for action pypa/gh-action-pypi-publish
        echo "ACCESS_TOKEN=$access_token" >> $GITHUB_ENV
   
    - name: Install dependencies
      run: |
        pip install -U pip
        pip install -r requirements/requirements.txt
   
    - name: Build Python package
      run: |
        # need to install wheel in advance
        python setup.py sdist bdist_wheel
        # modern Python uses `python -m build` instead
   
    # alternative Python package build and check
    - name: Build and Check Package
      uses: hynek/build-and-inspect-python-package@v1.5
   
    - name: Publish Python package by twine
      run: |
        # need to install twine in advance
        twine upload -r {azdo_artifacts_feed_name} dist/*.whl
   
    # alternative Python package publish
    - name: Publish Python package by action
      # does not need to install twine in advance
      uses: pypa/gh-action-pypi-publish@release/v1
      with:
        repository-url: "https://pkgs.dev.azure.com/{azdo_org_name}/_packaging/{azdo_artifacts_feed_name}/pypi/upload"
        password: ${{ env.ACCESS_TOKEN }}
   
    - name: Cleanup secret envs
      run: |
        echo "PIP_INDEX_URL=" >> $GITHUB_ENV
        echo "ACCESS_TOKEN=" >> $GITHUB_ENV
   

Auth by Azure OpenID Connect (OIDC)

We can also setup OpenID Connect (OIDC) between Github Action and Azure. It’s practical because we do not need to worry about Azure SPN secret rotation. However, a drawback is that when setting up OIDC, we must add a filter (subject field in the credential.json). This could be a branch name, tag name, pull request, or environment name, we can not use wildcards in the filter, so we have to set up OIDC for each branch, tag, pull request or environment as needed. This is not very practical. For AWS, there’s no such limitation.

To use Azure OIDC with Github Action, we need to add the following to the workflow:

...
permissions:
  id-token: write
  contents: read

jobs:
  a_job:
    ...
    steps:
      - name: Azure login by OIDC
        uses: azure/login@v1
        with:
          # Official doc puts these 3 fields in secrets, but it's not necessary,
          # as `subject` field in the credential.json prevents other repos from
          # using the same credential. And these are not sensitive info neither.
          tenant-id: ${{ vars.AZURE_TENANT_ID }}
          subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }}
          client-id: ${{ vars.AZURE_CLIENT_ID }}