Troubleshooting Python Twine Cannot Upload Package On Windows

Python has several tools to upload packages to PyPi or some private Artifactory locations. The mostly used one should be twine. Although twine is not a Python originate tool, but it’s officially recommended by Python.org.

Building the package

Just a quick callback on how to build the pacakge. We need to create a file named setup.py at the root of the app. Use another file named MANIFEST.IN to include the non-code files to the package. Don’t forget to set include_package_data=True in setup.py

Wheel

A Built Distribution format introduced by PEP 427, which is intended to replace the Egg format. Wheel is currently supported by pip.

Before the build, ensure that version key in setup.py is well defined.

# to build a python wheel package
# sdist will generate a .tar.gz file in dist/
# bdist_wheel will generate a .whl file in dist/
python setup.py sdist bdist_wheel

Upload built package to PyPi or private Artifactory.

We use twine to upload the Python packages. Before using it, we need to create a file name .pypirc in ~/.

There’s an example from jfrog for .pypirc.

Then, we can upload the package by:

# -r dev, dev is a repo defined in the ~/.pypirc file.
6.2.0> twine upload dist/* -r dev --cert [path_of_artifactory_site_cert_bundle_full_chain_in_pem_format_it_seems_that_no_param_to_ignore_ssl_error_with_twine]

.pypirc path error

Unfortunately, on Windows OS, you might get following error message:

6.2.0> twine upload dist/* --cert [artifactory_site_cert_full_chain_in_pem_format] -r dev

InvalidConfiguration: Missing 'dev' section from the configuration file or not a complete URL in --repository-url.
Maybe you have a out-dated '~/.pypirc' format?
more info: https://docs.python.org/distutils/packageindex.html#pypirc

This error is too generic, one of the reasons is because twine cannot find the file ~/.pypirc, but if you check by get-content ~/.pypirc, it exits.

The reason for this error is that if you’re on Windows, and $env:HOME exists and doesn’t point to the same location as $env:USERPROFILE.

twine uses $env:HOME as ~/ as per os.path.expanduser(), but Windows powershell uses $env:USERPROFILE as ~/. $env:HOME is not set by Windows by default. And Windows administrators often use $env:HOME to redirect the user roaming profile.

.pypirc path error reason

1. Firstly, I set $env:HOME to a temp file, so it is differnet than $env:USERPROFILE

    # Initially $env:HOME doesn't exist
    6.2.0> Get-ChildItem env: | Out-String -st | Select-String 'userpro|home'
   
    ANDROID_SDK_HOME               C:\Android
    HOMEDRIVE                      C:
    HOMEPATH                       \Users\xiang
    USERPROFILE                    C:\Users\xiang
   
    6.2.0> $env:HOME = 'c:/temp'
   
    # now, we have $env:HOME which is different than $env:USERPROFILE
    6.2.0> Get-ChildItem env: | Out-String -st | Select-String 'userpro|home'
   
    ANDROID_SDK_HOME               C:\Android
    HOME                           c:/temp
    HOMEDRIVE                      C:
    HOMEPATH                       \Users\xiang
    USERPROFILE                    C:\Users\xiang
   

2. Check ~/ in Python

    In [1]: import os
   
    In [2]: os.path.expanduser('~/')
    Out[2]: 'c:/temp/'
   

3. Check ~/ in Powershell

    6.2.0> Resolve-Path ~/
   
    Path
    ----
    C:\Users\xiang
   

So if we created the .pypirc file in ~/ in Powershell, twine won’t find it.

Why os.path.expanduser() doesn’t resolve the same ~/ as Powershell

As shown previsouly, Windows Powershell resolves ~/ as $env:USERPROFILE. How about os.path.expanduser()? Let’s check its source code by the inspect module.

In [1]: import os ; print(inspect.getsource(os.path.expanduser))
def expanduser(path):
    """Expand ~ and ~user constructs.

    If user or $HOME is unknown, do nothing."""
    path = os.fspath(path)
    if isinstance(path, bytes):
        tilde = b'~'
    else:
        tilde = '~'
    if not path.startswith(tilde):
        return path
    i, n = 1, len(path)
    while i < n and path[i] not in _get_bothseps(path):
        i += 1

    if 'HOME' in os.environ:
        userhome = os.environ['HOME']
    elif 'USERPROFILE' in os.environ:
        userhome = os.environ['USERPROFILE']
    elif not 'HOMEPATH' in os.environ:
        return path
    else:
        try:
            drive = os.environ['HOMEDRIVE']
        except KeyError:
            drive = ''
        userhome = join(drive, os.environ['HOMEPATH'])

    if isinstance(path, bytes):
        userhome = os.fsencode(userhome)

    if i != 1: #~user
        userhome = join(dirname(userhome), path[1:i])

    return userhome + path[i:]

In [2]:

From the source code, obviously, if $env:HOME exists, expanduser() will return its value. If $env:HOME doesn’t exists, it falls back to $env:USERPROFILE, if not again, it falls back to $env:HOMEDRIVE/$env:HOMEPATH.

Solutions

We have 3 solutions.

1. use twine --config-file to manually specify the .pypirc config file.

2. if $env:HOME exists, copy the .pypirc file to $env:HOME, otherwise to $env:USERPROFILE.

3. declare all the upload params as environment variables.