feat(toolchains): support runtime registration from manifest by rickeylev · Pull Request #3802 · bazel-contrib/rules_python

rickeylev · 2026-05-31T21:13:32Z

Currently, all supported Python runtime versions and their
platform-specific metadata (URLs, SHA256s, strip_prefix) must be
hardcoded in python/versions.bzl. This makes it slow and difficult
to adopt new Python versions or custom builds without updating
rules_python itself.

This PR introduces the ability to dynamically fetch and register
Python runtimes from a remote python-build-standalone (PBS) manifest
file (e.g., SHA256SUMS).

This is supported via two new attributes in python.override:

add_runtime_manifest_urls: A list of URLs pointing to manifest
files to parse and register.
runtime_manifest_sha: The SHA256 hash of the manifest file.

The manifest file format is the python-build-standalone SHA256SUMS
format (SHA FILENAME), extended to allow arbitrary URLs for the
filename (to allow more arbitrary locations).

Currently, all supported Python runtime versions and their platform-specific metadata (URLs, SHA256s, strip_prefix) must be hardcoded in `python/versions.bzl`. This makes it slow and difficult to adopt new Python versions or custom builds without updating `rules_python` itself. This PR introduces the ability to dynamically fetch and register Python runtimes from a remote python-build-standalone (PBS) manifest file (e.g., `SHA256SUMS`). This is supported via two new attributes in `python.override`: - `add_runtime_manifest_urls`: A list of URLs pointing to manifest files to parse and register. - `runtime_manifest_sha`: The SHA256 hash of the manifest file.

gemini-code-assist

Code Review

This pull request introduces support for dynamically fetching and registering Python runtimes from a python-build-standalone manifest file using python.override(add_runtime_manifest_urls = ..., runtime_manifest_sha = ...). It includes a new Starlark parser for the manifest files, integration into the toolchain configuration logic, and comprehensive unit and integration tests. Critical feedback has been provided regarding the use of unsupported string methods (removesuffix/removeprefix) in Starlark, a runtime error in handling mctx.download results without allow_fail = True, and a locale-dependent date parsing issue in the integration tests.

This mock module was extracted from bazel-contrib#3802. It provides a helper for defining a mock module for the python bzlmod extension, which is standalone and generally useful for testing rules_python's bzlmod extension.

…ce mode Workspace builds running under older Bazel versions do not support Bzlmod module extensions. This commit introduces Starlark helper macros to conditionally gate and register Bzlmod-specific unit tests, resolving test suite loading crashes in workspace CI jobs.

Removes the external test_helpers.bzl under tests/python/ and inlines the register_python_tests macro directly inside python_tests.bzl. This simplifies the test suite layout while keeping legacy workspace gating completely intact.

aignas

Thank you. This was on my mind for some time, so thanks for taking a stab at it.

What do you think about doing the following:

Registering the manifests we use in our MODULE.bazel file
Changing the rule that prints the contents that we should paste to python/versions.bzl to instead update the file with what is defined in the MODULE.bazel file. This way we know the manifest files are interpreted in exactly the same way as we would normally do it.

aignas · 2026-06-01T01:28:19Z

+    add_runtime_manifest_urls = [
+        "https://github.com/astral-sh/python-build-standalone/releases/download/20260414/SHA256SUMS",
+    ],
+    runtime_manifest_sha = "ce18fdfd47c66830a40ea9b9e314a14b1636bbfd684501bc5ca1fc6d55a7933f",


This is really good. Maybe we could dog-food the API and update the python/versions.bzl.

Are you ok doing this in a separate PR? It doesn't look too bad, but its enough of a change in how the default runtimes are registered i think it deserves its own change. proof of concept

aignas · 2026-06-01T01:37:36Z

+    if plus:
+        python_version = left
+        build_version, sep, rest = tail.partition("-")
+        if not sep:
+            return None
+    else:
+        python_version, sep, rest = left.partition("-")
+        if not sep:
+            return None
+        build_version = ""


What about using our own PEP440 compliant parser?

The format 3.10.20+20260414 is compliant.

We could the use the version comparison function to do the ordering.

Use the PEP440 parser on the whole file name, or just the X.Y.Z+build_version component?

Unless there's a python-build-standalone reference that says their naming format is PEP440 compliant, I'm -1 on using it for the whole filename; that seems just incidental.

Using it just for the "X.Y.Z+build_version" -- I guess so? So you mean return the full object/struct the PEP440 parser returns?

This mock module was extracted from bazel-contrib#3802. It provides a helper for defining a mock module for the python bzlmod extension, which is standalone and generally useful for testing rules_python's bzlmod extension.

rickeylev

what do you think of (1) register our manifests in module.bazel, (2) changing rule to update the manifest specified by (2)

SGTM. I wasn't sure how we should use this ourselves. I like your idea of allowing label() to point to a file.

rickeylev · 2026-06-06T03:57:22Z

+    if plus:
+        python_version = left
+        build_version, sep, rest = tail.partition("-")
+        if not sep:
+            return None
+    else:
+        python_version, sep, rest = left.partition("-")
+        if not sep:
+            return None
+        build_version = ""


Use the PEP440 parser on the whole file name, or just the X.Y.Z+build_version component?

Unless there's a python-build-standalone reference that says their naming format is PEP440 compliant, I'm -1 on using it for the whole filename; that seems just incidental.

Using it just for the "X.Y.Z+build_version" -- I guess so? So you mean return the full object/struct the PEP440 parser returns?

Enables local PBS manifest file resolution in Bzlmod mode. Implemented attr.label_list attribute on python.override and updated _populate_from_pbs_manifest to read local files via module_ctx.read.

Separates free-threading and archive type from the build flavor string in parse_filename. Also includes canonical documentation citation in the docstring and updates manifest parsing assertions.

Documents local manifest file loading alongside remote manifest URLs. Includes unified Starlark example using @// label prefix.

Flattens parsed manifest entries into a single list and sorts by archive flavor (install_only > install_only_stripped > full) so smaller standalone archives take precedence.

Restores explicit whitelist validation on entry.archive_flavor to prevent unsupported standalone release asset formats from polluting available toolchain mappings.

# Conflicts: # tests/support/mocks/python_ext.bzl

rickeylev · 2026-06-06T17:51:32Z

PTAL

rickeylev requested a review from aignas as a code owner May 31, 2026 21:13

gemini-code-assist Bot reviewed May 31, 2026

View reviewed changes

Comment thread python/private/python.bzl Outdated

Comment thread python/private/pbs_manifest.bzl

Comment thread tests/integration/runtime_manifests/basic_test.py

rickeylev mentioned this pull request May 31, 2026

test(mocks): extract mock python extension helper #3803

Merged

rickeylev force-pushed the feat.toolchain.manifest branch 3 times, most recently from 57479a5 to 85e7778 Compare May 31, 2026 21:58

rickeylev added 2 commits May 31, 2026 21:58

aignas reviewed Jun 1, 2026

View reviewed changes

rickeylev commented Jun 6, 2026

View reviewed changes

rickeylev added 6 commits June 6, 2026 15:50

Add add_runtime_manifest_files to python.override

2f37ffc

Enables local PBS manifest file resolution in Bzlmod mode. Implemented attr.label_list attribute on python.override and updated _populate_from_pbs_manifest to read local files via module_ctx.read.

Expose freethreaded and archive_flavor in PBS manifest parsing

1cad325

Separates free-threading and archive type from the build flavor string in parse_filename. Also includes canonical documentation citation in the docstring and updates manifest parsing assertions.

Update docs/toolchains.md for add_runtime_manifest_files

738d995

Documents local manifest file loading alongside remote manifest URLs. Includes unified Starlark example using @// label prefix.

Prefer install_only archive flavors in PBS manifest resolution

fd22fc7

Flattens parsed manifest entries into a single list and sorts by archive flavor (install_only > install_only_stripped > full) so smaller standalone archives take precedence.

Ignore unrecognized archive flavors in PBS manifest resolution

8eb3ff2

Restores explicit whitelist validation on entry.archive_flavor to prevent unsupported standalone release asset formats from polluting available toolchain mappings.

Merge upstream main into feat.toolchain.manifest

e3e8a78

# Conflicts: # tests/support/mocks/python_ext.bzl

rickeylev mentioned this pull request Jun 6, 2026

refactor(toolchains): register runtimes using manifest #3812

Open

Uh oh!

Conversation

rickeylev commented May 31, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aignas left a comment

Choose a reason for hiding this comment

Uh oh!

aignas Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

rickeylev Jun 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aignas Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

rickeylev Jun 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rickeylev left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rickeylev Jun 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rickeylev commented Jun 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants