dkdc.dev - Python + Rust: a simple tutorial

This post is intended to provide a simple demonstration of a Python package that relies on core functionality written in Rust. You can view the source code on GitHub.

Tools

In addition to Python and Rust, we’ll use the following tools:

GitHub: repository hosting
GitHub CLI: interact with GitHub from the command line
just: a command runner
mise: a tool for managing dev environments (Python/Rust versions)

For generating and using Rust code in Python, we’ll use PyO3.

Setting up the project

We’ll start with an overview of the project setup.

Picking a name

We need a good name for our project. Ideally, this can match how the package on PyPI. I chose pawz because it was not taken, is short, and leans into what the demo package will do.

Layout

Let’s start by looking at the file tree for pawz:

(venv) cody@dkdc pawz % tree
./
├── LICENSE
├── dev-requirements.txt
├── justfile
├── pyproject.toml
├── readme.md
└── src/
    ├── pawz/
    │   └── __init__.py
    └── pawz-core/
        ├── Cargo.toml
        ├── pyproject.toml
        ├── readme.md
        └── src/
            ├── lib.rs
            └── main.rs

5 directories, 11 files

This setup is copied over from a private project, then modified to update the project name and reduce code to its minimal form.

Justfile

We can look at the justfile – just is a command runner – to get a sense of the things we will be doing:

# Justfile

# load environment variables
set dotenv-load

# aliases
alias fmt:=format

# list justfile recipes
default:
    just --list

# setup
setup:
    @pip install -r dev-requirements.txt

# build
build:
    just clean
    @python -m build src/pawz-core
    @python -m build

# format
format:
    @cargo fmt --manifest-path src/pawz-core/Cargo.toml
    @ruff format .

# install
install:
    @maturin dev -m src/pawz-core/Cargo.toml
    @pip install --upgrade -e '.[all]'

# uninstall
uninstall:
    @pip uninstall pawz pawz-core -y

# release-test
release-test:
    just build
    @twine upload --repository testpypi src/pawz-core/dist/* -u __token__ -p ${PYPI_TEST_TOKEN}
    @twine upload --repository testpypi dist/* -u __token__ -p ${PYPI_TEST_TOKEN}

# release
release:
    just build
    @twine upload src/pawz-core/dist/* -u __token__ -p ${PYPI_TOKEN}
    @twine upload dist/* -u __token__ -p ${PYPI_TOKEN}

# clean
clean:
    @rm -r src/pawz-core/target || True
    @rm -rf src/pawz-core/dist || True
    @rm -rf dist || True

Python and Rust sub-projects

You may have already noticed above we’re building and publishing two Python packages – pawz is a pure Python package that depends on pawz-core, a pure Rust package with Python bindings.

You can see the package definitions in the respective pyproject.tomls:

Important

Due to the website’s exquisite theme, you may not be able to tell that below are two separate tabs.

pawz: pure Python
pawz-core: pure Rust with Python bindings

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "pawz"
version = "0.1.0"
authors = [{ name = "Cody", email = "cody@dkdc.dev" }]
description = "a demo of Python + Rust"
readme = "readme.md"
requires-python = ">=3.11"
classifiers = [
  "Programming Language :: Python :: 3",
  "License :: OSI Approved :: MIT License",
  "Operating System :: OS Independent",
]
dependencies = ["pawz_core"]

[project.urls]
"Homepage" = "https://github.com/lostmygithubaccount/pawz"
"Bug Tracker" = "https://github.com/lostmygithubaccount/pawz/issues"

[build-system]
requires = ["maturin>=1.5,<2.0"]
build-backend = "maturin"

[project]
name = "pawz_core"
dynamic = ["version"]
authors = [{ name = "Cody", email = "cody@dkdc.dev" }]
description = "pawz core functionality (Rust!)"
readme = "readme.md"
requires-python = ">=3.11"
classifiers = [
  "Programming Language :: Python :: 3",
  "License :: OSI Approved :: MIT License",
  "Operating System :: OS Independent",
]
dependencies = []

[tool.maturin]
features = ["pyo3/extension-module"]

[project.urls]
"Homepage" = "https://github.com/lostmygithubaccount/pawz"
"Bug Tracker" = "https://github.com/lostmygithubaccount/pawz/issues"

You’ll notice the primary difference is the build tooling – hatchling for the pure-Python project versus maturin for the Rust project.

With pawz-core, the package version is specified in the Cargo.toml instead of the pyproject.toml – we’ll take a closer look at that later.

`pawz`

Let’s setup our Python package pawz – to start, all you need is an __init__.py file in the correct subdirectory:

# imports
import pawz_core as pc

# exports
__all__ = ["pc"]

All we’re doing at this point is making pawz_core available as a module from pawz. So, we can do something like:

import pawz as pz

pz.pc.open_pawz()

Where the open_pawz() method is defined in pawz_core. We haven’t done that yet, but will later.

`pawz-core`

In addition to the pyproject.toml above, we need a Cargo.toml to define the Rust project:

[package]
name = "pawz_core"
version = "0.1.0"
edition = "2021"

[lib]
name = "pawz_core"
crate-type = ["cdylib"]

[dependencies]
pyo3 = "0.21.1"

Notice the Python package’s version will match the version defined here.

We also need the lib.rs in the src/ directory defining the functionality and exposing it through the Python bindings:

use pyo3::prelude::*;
//use pyo3::exceptions::PyValueError;
//use pyo3::types::PyTuple;

/// A Python module implemented in Rust.
#[pymodule]
#[allow(deprecated)]
fn pawz_core(_py: Python, _m: &PyModule) -> PyResult<()> {
    Ok(())
}

Notice there’s nothing defined yet and a few things are commented out and prepended with an underscore to avoid the compiler complaining. However at this point, everything is fully functional – you can build pawz-core and pawz and import them.

What about the main.rs?

As of writing, the main.rs is a simple hello world:

fn main() {
    println!("Hello, world!");
}

You don’t need the file, but I find it useful to easily try things out in Rust. For instance, you can import things from lib.rs and run them.

Setup the Python environment

Let’s take a look at the dev-requirements.txt:

# python
ruff
build
twine

# rust
maturin

ruff is to format Python code, build is to build the Python packages, twine is to upload them to PyPI, and maturin is to build the Rust package with Python bindings. You can see the use of these tools above in the justfile.

Setup a virtual Python environment

You should generally use one virtual environment per project. I use mise to manage Python (and Rust) versions, but you don’t have to. You can run:

(venv) cody@dkdc pawz % python -m venv venv
(venv) cody@dkdc pawz % source venv/bin/activate

to create your virtual environment. At this point if you’ve installed just, you can run:

(venv) cody@dkdc pawz % just setup

This will install the dev dependencies defined above.

Installing `pawz` and `pawz-core`

You can install pawz and pawz-core in local developer mode with:

(venv) cody@dkdc pawz % just install

Now you’re able to import pawz as pz!

Uploading to GitHub

There are multiple ways to do this, but I prefer something like:

(venv) cody@dkdc pawz % git init
(venv) cody@dkdc pawz % gh repo create pawz --public --source .
(venv) cody@dkdc pawz % git checkout -b main
(venv) cody@dkdc pawz % git add .
(venv) cody@dkdc pawz % git commit -m "initial commit"
(venv) cody@dkdc pawz % git push --set-upstream origin/main

Now, your code should be up on GitHub!

Publishing to PyPI

Let’s publish pawz and pawz-core to PyPI. First, you need to generate and place PyPI tokens in a .env file:

PYPI_TOKEN = "pypi-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
PYPI_TEST_TOKEN = "pypi-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

Then run:

(venv) cody@dkdc pawz % just release-test
(venv) cody@dkdc pawz % just release

Doing something

Now, let’s do something! We’ll write some core functionality in Rust. For this demonstration, we’ll just open up a hardcoded URL pointing to a YouTube video.

Important

In the pyproject.toml and Cargo.toml we should bump the version up so we can ship a new version later.

Rust edits

We need to add the open crate dependency in the Cargo.toml:

[package]
name = "pawz_core"
version = "0.2.0"
edition = "2021"

[lib]
name = "pawz_core"
crate-type = ["cdylib"]

[dependencies]
pyo3 = "0.21.1"
open = "5.1.2"

Then edit lib.rs to create our function and add it to the Python module:

use open;
use pyo3::prelude::*;
//use pyo3::exceptions::PyValueError;
//use pyo3::types::PyTuple;

#[pyfunction]
fn open_pawz() {
    let url = "https://www.youtube.com/watch?v=gJ6slhwPp6E";
    open::that(url).unwrap();
}

/// A Python module implemented in Rust.
#[pymodule]
#[allow(deprecated)]
fn pawz_core(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(open_pawz, m)?)?;
    Ok(())
}

Now if you import pawz_core, you can run pawz_core.open_pawz() from Python!

Python edits

No edits to Python are needed – since pawz_core is available from pawz, we can just call pawz.pawz_core.open_pawz(). You might want something different, i.e. to abstract away the core library code, but this is fine for demonstration purposes.

Git workflow

First, we’ll check what we changed:

Note

I changed a bit more when I actually did this, reflected in some of the output below.

(venv) cody@dkdc pawz % git status
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   pyproject.toml
        modified:   src/pawz-core/Cargo.toml
        modified:   src/pawz-core/src/lib.rs
        modified:   src/pawz/__init__.py

no changes added to commit (use "git add" and/or "git commit -a")

Checkout a branch to commit our changes to:

(venv) cody@dkdc pawz % git checkout -b add-something
Switched to a new branch 'add-something'

Ensure we format our code (Rust and Python):

(venv) cody@dkdc pawz % just fmt
1 file left unchanged

Add our changes and commit them:

(venv) cody@dkdc pawz % git add .
(venv) cody@dkdc pawz % git commit -m "add something"
[add-something 3e8727d] add something
 4 files changed, 17 insertions(+), 5 deletions(-)

Then create a PR on GitHub:

(venv) cody@dkdc pawz % gh pr create
? Where should we push the 'add-something' branch? lostmygithubaccount/pawz

Creating pull request for add-something into main in lostmygithubaccount/pawz

? Title add something
? Body <Received>
? What's next? Submit
remote:
remote:
To https://github.com/lostmygithubaccount/pawz.git
 * [new branch]      HEAD -> add-something
branch 'add-something' set up to track 'origin/add-something'.
https://github.com/lostmygithubaccount/pawz/pull/1

And finally, merge the PR from the CLI:

(venv) cody@dkdc pawz % gh pr merge
Merging pull request lostmygithubaccount/pawz#1 (add something)
? What merge method would you like to use? Squash and merge
? Delete the branch locally and on GitHub? Yes
? What's next? Submit
✓ Squashed and merged pull request lostmygithubaccount/pawz#1 (add something)
remote: Enumerating objects: 19, done.
remote: Counting objects: 100% (19/19), done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 10 (delta 3), reused 9 (delta 3), pack-reused 0
Unpacking objects: 100% (10/10), 1.72 KiB | 586.00 KiB/s, done.
From https://github.com/lostmygithubaccount/pawz
 * branch            main       -> FETCH_HEAD
   a866fd3..318bd89  main       -> origin/main
Updating a866fd3..318bd89
Fast-forward
 pyproject.toml           |  2 +-
 src/pawz-core/Cargo.toml |  3 ++-
 src/pawz-core/src/lib.rs | 10 +++++++++-
 src/pawz/__init__.py     |  7 +++++--
 4 files changed, 17 insertions(+), 5 deletions(-)
✓ Deleted local branch add-something and switched to branch main
✓ Deleted remote branch add-something

Now our source code is updated. You can view this PR here, including the files changed.

Updating on PyPI

With our code updated, we want to update our packages on PyPI. We simply run:

(venv) cody@dkdc pawz % just release

as before, shipping the new version that does something!

Conclusion

I hope this is helpful to someone setting up a Python package with core functionality written in Rust. Of course, there are many ways to do this but I find the above fairly clean.

Tools

Setting up the project

Picking a name

Layout

Justfile

Python and Rust sub-projects

pawz

pawz-core

Setup the Python environment

Setup a virtual Python environment

Installing pawz and pawz-core

Uploading to GitHub

Publishing to PyPI

Doing something

Rust edits

Python edits

Git workflow

Updating on PyPI

Conclusion

`pawz`

`pawz-core`

Installing `pawz` and `pawz-core`