python_packaging_example

Python Packaging Example

Table of contents

• How-to instructions
• Background
• High level
• Packaging Python Projects
  • Building the package
  • Uploading the package to TestPyPi
  • Uploading the package to PyPi
  • Manually updating the package
• GitHub Actions CI/CD workflows
  • Set up
  • Uploading to TestPyPi via commit to master
  • Uploading to PyPi via tagging
• Running pytest in GitHub CI/CD

---

How-to instructions

• To set up new projects and make releases, check out CI/CD and PyPI releases

Background

This is a working example that uses a GitHub actions CI/CD workflow to test, build and upload a Python package to TestPyPi and PyPi.

I created this example package by working through these guides;

• Packaging Python Projects
• Publishing package distribution releases using GitHub Actions CI/CD workflows
• PyTest With GitHub Actions

The rest of the README describes to set up a new project in the same way.

When set up;

• Test and upload to TestPyPi occurs when the package version number is updated and a commit is made to the master branch
• Test and upload to PyPi occurs when a commit is tagged
• The package can be installed using pip install example-package-grumbit

---

High level

At a high level, the process for setting up GitHub CI/CD project packaging, including pytesting, looks like this;

1. Set up the file structure as per this example package, but leave out ./.github/workflows/publish-to-test-pypi.yml for now.
2. Get local packaging working
3. Get uploading to TestPyPi working
4. Get uploading to PyPi working
5. Add in ./.github/workflows/publish-to-test-pypi.yml and get GitHub CI/CD working

---

Packaging Python Projects

Building the package

• The package metadata is configured in ./pyproject.toml
• Once configured, it can be built with;

cd <pacakges directory>
python3 -m venv .venv # Create the venv if it doesn't exist yet
source .venv/bin/activate
python3 -m pip install --upgrade pip setuptools wheel pip-tools pytest # Install the tools needed for the build tool
python3 -m pip install --upgrade build # Install the build tool itself
python3 -m build # build the package

Uploading the package to TestPyPi

• Upload the package for testing using;

python3 -m pip install --upgrade twine # Install the twine upload tool
python3 -m twine upload --repository testpypi dist/* # Upload to TestPyPi
    # When prompted, the username is __token__ and the password is the TestPyPi global scope API token

• Having uploaded the package, a package specific API token should be set up and saved in TestPyPi

• Check the package can be downloaded and used in a new venv;

cd <some new tmp directory>
python3 -m venv .venv 
source .venv/bin/activate
package_name="example-package-grumBit"
python3 -m pip install --index-url https://test.pypi.org/simple/ --pre ${package_name}  # Check the package can be installed
python3 -c "from example_package_grumbit import example; print(example.add_one(1))" # Check package functions

Uploading the package to PyPi

python3 -m twine upload dist/* # Upload to PyPi
    # When prompted, the username is __token__ and the password is the PyPi global scope API token

• Having uploaded the package, a package specific API token should be set up and saved in PyPi

Manually updating the package

• Each time the package is updated, it's version must be updated in the [project] section of ./pyproject.toml, then it needs to be re-built and uploaded;

vs ./pyproject.toml
python3 -m build # build the package
python3 -m twine check dist/* # check the package can be uploaded
python3 -m twine upload --repository testpypi dist/* # test uploading using TestPyPi
python3 -m twine upload dist/* # Upload to PyPi

---

GitHub Actions CI/CD workflows

Set up

• If the project isn't already sync'd up to GitHub, run;

cd "<the project's directory>"
repo_name="<the new repo's name>"
gh repo create "${repo_name}" --private
git init
git add --all
git commit -m "init"
git branch -M master
git remote add origin [email protected]:grumBit/${repo_name}.git
git push -u origin master

• If the default branch isn't master, either change it on GitHub, or change .github/workflows/publish-to-test-pypi.yml.

• Add the TestPyPi and PyPi API tokens to the repo

• Open the repo on GitHub using gh browse. In the browser, click Settings -> Secrets -> Actions. Then add two new secrets called PYPI_API_TOKEN and TEST_PYPI_API_TOKEN, with the API tokens created after uploading the packages above

• Create and configure .github/workflows/publish-to-test-pypi.yml workflow definition

  • NB: This example package's publish-to-test-pypi.yml already has the parts needed for auto-testing included (see below)

Uploading to TestPyPi via commit to master

• Every time a commit is made to the master branch, the GitHub CI/CD will run.
  • NB: For the packaging to succeed, the version must be updated in ./pyproject.toml.
• All commits to master will be uploaded to TestPyPi

Uploading to PyPi via tagging

• Putting a tag on a commit and pushing it will cause GitHub CI/CD to run and create a PyPi release.

• Use the following to tag the lastest commit (i.e. HEAD) with the version currently configured in ./pyproject.toml;

version_tag=v$(cat ./pyproject.toml | egrep "^version" | cut -d '"' -f2)
version_tag_info="Some release info"
git tag -a "${version_tag}" -m "${version_tag_info}"
git push --tag

• Use the following to tag a prior commit;

version_tag="vX.X.X"
version_tag_info="Some release info"
commit_sha="16fb0fd"
git tag -a "${version_tag}" "${commit_sha}" -m "${version_tag_info}"
git push --tag

---

Running pytest in GitHub CI/CD

• The workflow steps in the GitHub CI/CD guide didn't include running pytests. To get pytest to run the packages dependencies needed to be installed and then pytest run prior to the build step using these additional steps;

    - name: Install requirements
      run: >-
        python -m
        pip install
        --requirement requirements.txt
    - name: Run tests
      run: >-
        python -m
        pytest

• As per the directory structure in the packaging projects guide, I put the tests in a separate hierarchy to the source. This meant __init__.py needed to added to the src/ and tests/ directory like this;
  • NB: I'm not 100% sure about this structuring. It follows the guide and means the test code isn't packaged up, however, I think it's less convenient than embedding test/ folders within the src/ tree. ./pyproject.toml can be configured so that embedded test/ folders are excluded, but I've gone with the "standard" for now.

packaging_tutorial/
├── src/
│   ├── __init__.py
│   └── example_package_grumbit/
│       ├── __init__.py
│       └── example.py
└── tests/
    ├── __init__.py
    └── example_package_grumbit/
        ├── __init__.py
        └── test_example.py