Build release wheels with a checkout-local Podman store

This commit is contained in:
Jan Petykiewicz 2026-11-19 11:06:13 -08:00
commit 6dd54987de
2 changed files with 131 additions and 39 deletions

View file

@ -1,16 +1,18 @@
# Building releases # Building releases
Build packages locally for distribution. For installing and using Pheon, start Build packages locally for distribution. The checkout holds Python build tools
with the [CLI](cli.md) or [Python](python.md) guide. in `.venv/`, release artifacts in `dist/release/`, and Linux wheel caches in
`.cache/wheels/`. Use your installed Rust toolchain and Cargo registry cache.
## Start a release ## Start a release
Use Python 3.11+ and Rust 1.88+. From the repository root, create a build Use Python 3.11+ and Rust 1.88+. Run the commands from the repository root and
environment and put the source archive in an empty `dist/release/` directory. start with an empty `dist/release/` directory.
```sh ```sh
python3 -m venv .venv python3 -m venv .venv
. .venv/bin/activate . .venv/bin/activate
export PIP_CACHE_DIR="$PWD/.cache/wheels/pip"
python -m pip install build twine python -m pip install build twine
python -m build --sdist --outdir dist/release python -m build --sdist --outdir dist/release
``` ```
@ -19,50 +21,28 @@ Use `python -m build --wheel` to build a wheel for your own machine. For a
portable Linux wheel, use the container below. See [development](development.md) portable Linux wheel, use the container below. See [development](development.md)
for compiler checks. for compiler checks.
Python wheels bundle the native library. Install the CLI separately with Cargo.
## Build Linux x86-64 wheels ## Build Linux x86-64 wheels
On x86-64 Linux, use Podman and a Rust toolchain installed through `rustup`. On x86-64 Linux, use rootless Podman and a Rust toolchain installed through
Download the [pinned manylinux image](../contrib/wheels/manylinux-image.txt) `rustup`. Build the portable wheel from the source archive.
once and save it for future releases.
```sh
pheon_wheel_cache="$PWD/.cache/wheels"
pheon_image=$(cat contrib/wheels/manylinux-image.txt)
pheon_cached_image=localhost/pheon-manylinux:2.28-2026.10.26-1
mkdir -p "$pheon_wheel_cache"
podman pull --platform linux/amd64 "$pheon_image"
podman tag "$pheon_image" "$pheon_cached_image"
podman save --format oci-archive \
--output "$pheon_wheel_cache/manylinux_2_28_x86_64.oci.tar" \
"$pheon_cached_image"
```
If the image is missing from Podman, restore it with
`podman load --input .cache/wheels/manylinux_2_28_x86_64.oci.tar`.
Build from the source archive using the local Rust toolchain and Cargo cache.
```sh ```sh
python -m pip install 'cibuildwheel==4.3.5' python -m pip install 'cibuildwheel==4.3.5'
cargo fetch --locked --target x86_64-unknown-linux-gnu python tools/build_manylinux.py dist/release/pheon-*.tar.gz
rust_toolchain=$(rustc --print sysroot)
cargo_registry="${CARGO_HOME:-$HOME/.cargo}/registry"
export CIBW_CONTAINER_ENGINE="podman; disable_host_mount: true; create_args: \
--volume '$PWD/.cache/wheels:/pheon-cache' \
--volume '$rust_toolchain:/rust-toolchain:ro' \
--volume '$cargo_registry:/pheon-cache/cargo/registry:ro'"
export CIBW_MANYLINUX_X86_64_IMAGE=localhost/pheon-manylinux:2.28-2026.10.26-1
export CIBW_ENVIRONMENT_LINUX='PATH=/rust-toolchain/bin:$PATH CARGO_HOME=/pheon-cache/cargo CARGO_NET_OFFLINE=true PIP_CACHE_DIR=/pheon-cache/pip'
export CIBW_BUILD='cp31{1,2,3,4}-manylinux_x86_64'
export CIBW_TEST_COMMAND='python -I {project}/tools/check_installed.py'
unset PHEON_LIBRARY
python -m cibuildwheel --platform linux --archs x86_64 \
--output-dir dist/release dist/release/pheon-*.tar.gz
``` ```
cibuildwheel repairs the wheel and tests the same file on Python 3.11–3.14. The helper pulls the [pinned image](../contrib/wheels/manylinux-image.txt) when
The container downloads any Python dependencies that it needs. needed and reuses it from `.cache/wheels/podman/storage/`. It keeps Podman’s
storage runtime directory, download scratch space, and wheel caches under
`.cache/wheels/`. Podman still uses the normal system user runtime for host
services. To inspect this checkout’s images, use the same helper.
```sh
python tools/build_manylinux.py podman image ls
```
cibuildwheel repairs the wheel and tests it on Python 3.11–3.14. The container
uses the host Rust toolchain and Cargo registry through read-only mounts.
## Build Windows x86-64 packages from Linux ## Build Windows x86-64 packages from Linux
@ -104,4 +84,12 @@ python -I tools/check_installed.py
Build standalone CLI artifacts for each OS and architecture. Include the Build standalone CLI artifacts for each OS and architecture. Include the
executable, license, README, documentation, and Git templates. executable, license, README, documentation, and Git templates.
Publish the validated Python packages to PyPI and the Rust crate to crates.io. ## Publish
Upload the validated Python files you selected, then publish the crate from a
clean release checkout.
```sh
python -m twine upload --repository pypi FILE ...
cargo publish --locked
```

104
tools/build_manylinux.py Normal file
View file

@ -0,0 +1,104 @@
"""Build Linux release wheels using Podman storage in this checkout."""
import argparse
import json
import os
from pathlib import Path
import shlex
import subprocess
import sys
ROOT = Path(__file__).resolve().parents[1]
CACHE = ROOT / ".cache/wheels"
def local_environment() -> dict[str, str]:
podman = CACHE / "podman"
scratch = CACHE / "tmp"
podman.mkdir(parents=True, exist_ok=True)
scratch.mkdir(exist_ok=True)
storage = podman / "storage.conf"
graphroot = json.dumps(str(podman / "storage"), ensure_ascii=False)
runroot = json.dumps(str(podman / "run"), ensure_ascii=False)
config = f"[storage]\ngraphroot = {graphroot}\nrunroot = {runroot}\n"
# Replace the config atomically so simultaneous Podman commands can read it.
if not storage.exists() or storage.read_text(encoding="utf-8") != config:
temporary = podman / f"storage.{os.getpid()}.conf"
temporary.write_text(config, encoding="utf-8")
temporary.replace(storage)
env = dict(os.environ, CONTAINERS_STORAGE_CONF=str(storage), TMPDIR=str(scratch))
for key in ("CONTAINER_HOST", "CONTAINER_CONNECTION"):
env.pop(key, None)
return env
def build(sdist: Path) -> None:
env = local_environment()
image = (ROOT / "contrib/wheels/manylinux-image.txt").read_text(encoding="utf-8").strip()
# Use the same Podman invocation as cibuildwheel to check the active store.
graphroot = subprocess.check_output(
["podman", "info", "--format", "{{.Store.GraphRoot}}"], env=env, text=True,
).strip()
if Path(graphroot).resolve() != (CACHE / "podman/storage").resolve():
raise SystemExit(f"Expected the checkout's Podman store, found {graphroot}")
subprocess.run(
["cargo", "fetch", "--locked", "--target", "x86_64-unknown-linux-gnu"],
cwd=ROOT, env=env, check=True,
)
toolchain = subprocess.check_output(
["rustc", "--print", "sysroot"], cwd=ROOT, env=env, text=True,
).strip()
registry = Path(env.get("CARGO_HOME", Path.home() / ".cargo")).resolve() / "registry"
for directory in ("cargo/registry", "pip", "target", "cibuildwheel"):
(CACHE / directory).mkdir(parents=True, exist_ok=True)
mounts = [
f"{toolchain}:/rust-toolchain:ro",
f"{CACHE / 'cargo'}:/pheon-cache/cargo",
f"{registry}:/pheon-cache/cargo/registry:ro",
f"{CACHE / 'pip'}:/pheon-cache/pip",
f"{CACHE / 'target'}:/pheon-cache/target",
]
create_args = shlex.join([arg for mount in mounts for arg in ("--volume", mount)])
env.update(
CIBW_CONTAINER_ENGINE=f"podman; disable_host_mount: true; create_args: {create_args}",
CIBW_MANYLINUX_X86_64_IMAGE=image,
CIBW_ENVIRONMENT_LINUX=(
"PATH=/rust-toolchain/bin:$PATH CARGO_HOME=/pheon-cache/cargo "
"CARGO_TARGET_DIR=/pheon-cache/target CARGO_NET_OFFLINE=true "
"PIP_CACHE_DIR=/pheon-cache/pip"
),
CIBW_CACHE_PATH=str(CACHE / "cibuildwheel"),
CIBW_BUILD="cp31{1,2,3,4}-manylinux_x86_64",
CIBW_TEST_COMMAND="python -I {project}/tools/check_installed.py",
)
env.pop("PHEON_LIBRARY", None)
subprocess.run(
[sys.executable, "-m", "cibuildwheel", "--platform", "linux", "--archs", "x86_64",
"--output-dir", str(sdist.parent), str(sdist)],
cwd=ROOT, env=env, check=True,
)
def main() -> int:
if sys.argv[1:2] == ["podman"]:
return subprocess.run(
["podman", "--remote=false", *sys.argv[2:]], env=local_environment(),
).returncode
parser = argparse.ArgumentParser(
description=__doc__, epilog="Use 'podman ARGS' to inspect the local store.",
)
parser.add_argument("sdist", type=Path, help="Pheon source archive")
args = parser.parse_args()
if not args.sdist.is_file():
parser.error(f"source archive does not exist: {args.sdist}")
build(args.sdist.resolve())
return 0
if __name__ == "__main__":
sys.exit(main())