CI: allow specifying custom driver versions in test matrix by leofang · Pull Request #2176 · NVIDIA/cuda-python

leofang · 2026-06-07T01:49:28Z

Description

closes #293
closes #1265

Extends the DRIVER field in ci/test-matrix.yml to accept an explicit driver version string (e.g. 580.65.06) in addition to the existing latest / earliest.

Linux — new ci/tools/install_gpu_driver.sh swaps the driver in-job. It is adapted nearly verbatim from the runner team's nvgha-driver CLI (copied so we don't depend on its rollout schedule). The script nsenters onto the host for the install and refreshes the toolkit bind mounts back inside the test container. Because our script lives in the GH workspace mount (container-only), the host-side re-exec reads the script from stdin via bash -s < "$0" rather than running "$0" directly (the relative path doesn't resolve after the mount-namespace flip).
Windows — ci/tools/install_gpu_driver.ps1 is split into two scripts: install_gpu_driver.ps1 (now install-only, reads DRIVER from env, errors on latest/earliest) and a new ci/tools/configure_driver_mode.ps1 (driver-mode + pnputil device cycle, runs on every job). This also fixes a long-standing wart: the previous script unconditionally installed a hardcoded 581.15 even when the matrix row used a latest/earliest runner that already carried the right driver.

Matrix wiring (in both test-wheel-linux.yml and test-wheel-windows.yml):

compute-matrix adds a new RUNNER_DRIVER field per row — equal to DRIVER for latest / earliest, otherwise latest. runs-on: is keyed on RUNNER_DRIVER so custom-DRIVER rows land on the most recent pre-installed runner image (the install scripts perform the actual swap).
On Linux, container.options only adds --privileged --pid=host for custom-DRIVER rows (required by the nsenter dance).
On Linux, custom DRIVER combined with FLAVOR=wsl is rejected eagerly in compute-matrix — the in-container swap doesn't work under WSL.
The "Ensure GPU is working" step (nvidia-smi) now runs after the install / configure step in every workflow, so it validates the post-install driver state on custom-DRIVER rows.
coverage.yml (Windows path, hardcoded DRIVER: latest) was updated alongside since it was the other caller of the old combined script.

Matrix rows flipped to exercise the new code path:

PR matrix (Linux, every PR):
- amd64 / 3.13 / 13.3.0 / local-CTK / rtxpro6000 → DRIVER: '610.43.02'
- amd64 / 3.14 / 13.3.0 / local-CTK / l4 → DRIVER: '610.43.02'
Nightly numba-cuda:
- Linux amd64 / 3.12 / 13.3.0 / l4 → DRIVER: '580.65.06'
- Windows amd64 / 3.12 / 13.3.0 / l4 → DRIVER: '596.36'

Also enables workflow_dispatch: on ci.yml so the main CI pipeline can be re-run manually from the Actions UI (no inputs — the workflow already builds every wheel it tests, and the existing should-skip / detect-changes gates handle non-PR events correctly).

All other matrix rows continue to use DRIVER: latest and are unaffected (same runners, no install step, no privileged container).

Checklist

New or existing tests cover these changes.
The documentation is up to date with these changes.

Extends the DRIVER field in ci/test-matrix.yml beyond 'latest'/'earliest' to accept an explicit version string (e.g. '580.65.06'). For Linux, ci/tools/install_gpu_driver.sh (adapted from nv-gha-runners/vm-images PR NVIDIA#256) swaps the driver in-job via nsenter when the row uses a custom version; for Windows, ci/tools/install_gpu_driver.ps1 is split into install + configure_driver_mode, with the install step gated on the DRIVER value and the mode step always running. The matrix row is routed to a 'latest' runner image when the DRIVER is a custom version (the install scripts perform the swap themselves). Container privileges on Linux (--privileged --pid=host) are added only on rows with a custom DRIVER. Custom DRIVER + FLAVOR=wsl is rejected eagerly in the compute-matrix step. Two existing nightly-numba-cuda rows exercise the new path: - Linux amd64 / 13.3.0 / l4 -> 580.65.06 - Windows amd64 / 13.3.0 / l4 -> 610.47 Closes NVIDIA#293 Closes NVIDIA#1265

copy-pr-bot · 2026-06-07T01:49:33Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

leofang · 2026-06-07T01:56:12Z

/ok to test b1b6070

github-actions · 2026-06-07T02:18:00Z

Doc Preview CI
🚀 View preview at https://nvidia.github.io/cuda-python/pr-preview/pr-2176/
https://nvidia.github.io/cuda-python/pr-preview/pr-2176/cuda-core/
https://nvidia.github.io/cuda-python/pr-preview/pr-2176/cuda-bindings/
https://nvidia.github.io/cuda-python/pr-preview/pr-2176/cuda-pathfinder/
Preview will be ready when the GitHub Pages deployment is complete.

….yml dispatch - install_gpu_driver.sh: pipe the script body to the host-side bash via stdin (bash -s < "$0") instead of re-execing "$0". The script lives in the GH workspace mount (container-only), so the relative path doesn't resolve after nsenter switches the mount namespace. The < "$0" fd is opened before nsenter and survives the flip. - test-matrix.yml: Windows nightly-numba-cuda row 610.47 -> 596.36 (610.47 isn't published on the CDN; install hit 404). - ci.yml: add workflow_dispatch: trigger so the pipeline can be re-run manually. The existing should-skip / detect-changes gates already handle non-PR events.

leofang · 2026-06-07T03:13:14Z

/ok to test 3e016b5

leofang · 2026-06-07T03:24:03Z

    # nightly-numba-cuda
    - { MODE: 'nightly-numba-cuda', ARCH: 'amd64', PY_VER: '3.12', CUDA_VER: '12.9.1', LOCAL_CTK: '0', GPU: 'l4', GPU_COUNT: '1', DRIVER: 'latest', DRIVER_MODE: 'TCC' }
-    - { MODE: 'nightly-numba-cuda', ARCH: 'amd64', PY_VER: '3.12', CUDA_VER: '13.3.0', LOCAL_CTK: '0', GPU: 'l4', GPU_COUNT: '1', DRIVER: 'latest', DRIVER_MODE: 'TCC' }
+    - { MODE: 'nightly-numba-cuda', ARCH: 'amd64', PY_VER: '3.12', CUDA_VER: '13.3.0', LOCAL_CTK: '0', GPU: 'l4', GPU_COUNT: '1', DRIVER: '596.36',  DRIVER_MODE: 'TCC' }


xref: https://github.com/NVIDIA/cuda-python/actions/runs/27081260952/job/79927472454

leofang · 2026-06-07T03:24:58Z

    # nightly-numba-cuda
    - { MODE: 'nightly-numba-cuda', ARCH: 'amd64', PY_VER: '3.12', CUDA_VER: '12.9.1', LOCAL_CTK: '0', GPU: 'l4', GPU_COUNT: '1', DRIVER: 'latest' }
-    - { MODE: 'nightly-numba-cuda', ARCH: 'amd64', PY_VER: '3.12', CUDA_VER: '13.3.0', LOCAL_CTK: '0', GPU: 'l4', GPU_COUNT: '1', DRIVER: 'latest' }
+    - { MODE: 'nightly-numba-cuda', ARCH: 'amd64', PY_VER: '3.12', CUDA_VER: '13.3.0', LOCAL_CTK: '0', GPU: 'l4', GPU_COUNT: '1', DRIVER: '580.65.06' }


xref https://github.com/NVIDIA/cuda-python/actions/runs/27081260952/job/79927472925

So nvidia-smi validates the post-install driver state on custom-DRIVER rows. Windows test-wheel + coverage already use Install -> Configure -> Ensure; this brings the Linux test-wheel job into line.

Exercises the custom-driver install path on every PR (not just nightly). Both rows are amd64 / 13.3.0 / local-CTK, on l4 and rtxpro6000 -- both in the 'open' kernel-module flavor (only Volta needs 'legacy').

leofang · 2026-06-07T03:31:32Z

/ok to test 4a23b23

rwgk · 2026-06-07T06:17:53Z

+    $nvidia_devices = Get-PnpDevice -Class Display -FriendlyName "NVIDIA*"
+    foreach ($device in $nvidia_devices) {
+        Write-Output "Restarting device: $($device.FriendlyName) ($($device.InstanceId))"
+        pnputil /disable-device "$($device.InstanceId)"
+        pnputil /enable-device "$($device.InstanceId)"
+    }


It seems this function now cycles display devices on every Windows job, even when no driver was installed and nvidia-smi -fdm reports the mode was already correct (e.g. Driver model is already set to MCDM for GPU 00000000:0A:00.0. in the log). Is that intentional? In the h100/MCDM CI failure, NVML becomes unavailable right after the pnputil cycle here, so I’m wondering if the device restart should only happen after an install or an actual mode change.

Linux: After install_gpu_driver.sh stops nvidia-persistenced and the apt purge removes the package, the .run installer reinstalls the systemd service but leaves it stopped. cuda.core's test_persistence_mode_enabled fails with NVML_ERROR_UNKNOWN on driver 610.43.02 when the daemon is not running; explicitly start it again at the end of host_install(). Windows: configure_driver_mode.ps1's trailing 'Start-Sleep -Seconds 5' is not enough on slower-coming-back-up multi-GPU rows (observed: 2x H100 MCDM). Replace it with a poll-until-success loop on nvidia-smi with a 60s deadline, matching the runner-team nvgha-driver.ps1 pattern. Previously masked because every Windows row used to run the full install pipeline; with custom-DRIVER plumbing, latest/earliest rows skip the install and the cycle is no longer preceded by warm-up time.

leofang · 2026-06-07T14:18:30Z

/ok to test d33a928

Runner-latest L4 images come up with Persistence-M=On (set somewhere in the runner team's image setup, not in cuda-python). Our .run install leaves it Off, which breaks cuda.core's test_persistence_mode_enabled on driver 610.43.02 -- the test calls device.is_persistence_mode_enabled = False on a device that already reports False, and 610.43.02 returns NVML_ERROR_UNKNOWN for that no-op set. Restore the runner baseline by calling `nvidia-smi -pm 1` at the end of host_install() (sets the kernel persistence flag directly via NVML). Also daemon-reload + start nvidia-persistenced.service best-effort so tools that look for the daemon find it; `set -x` around this trailing block so the next run's log confirms which lines fired.

leofang · 2026-06-07T15:19:05Z

/ok to test 00896dc

refresh_container_libs() used 'cp -f --remove-destination' (verbatim from the runner team's nvgha-driver), which without -p/--preserve strips the SUID/SGID bits on the destination. /usr/bin/nvidia-modprobe ships 4755 and NVML's state-changing calls (e.g. nvmlDeviceSetPersistenceMode) route through it; once SUID is gone the container-side call returns NVML_ERROR_UNKNOWN, which is what cuda.core's test_persistence_mode_enabled was hitting. Add a stat diagnostic line at the end of refresh_container_libs() so the next CI log records nvidia-modprobe's post-refresh mode.

github-actions Bot added the CI/CD CI/CD infrastructure label Jun 7, 2026

leofang self-assigned this Jun 7, 2026

leofang added the P0 High priority - Must do! label Jun 7, 2026

leofang added this to the cuda.core v1.1.0 milestone Jun 7, 2026

leofang commented Jun 7, 2026

View reviewed changes

leofang added 2 commits June 7, 2026 03:27

CI: move 'Ensure GPU is working' after 'Install GPU driver' on Linux

c0ca869

So nvidia-smi validates the post-install driver state on custom-DRIVER rows. Windows test-wheel + coverage already use Install -> Configure -> Ensure; this brings the Linux test-wheel job into line.

CI: flip two PR-matrix Linux rows to DRIVER=610.43.02

4a23b23

Exercises the custom-driver install path on every PR (not just nightly). Both rows are amd64 / 13.3.0 / local-CTK, on l4 and rtxpro6000 -- both in the 'open' kernel-module flavor (only Volta needs 'legacy').

rwgk reviewed Jun 7, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: allow specifying custom driver versions in test matrix#2176

CI: allow specifying custom driver versions in test matrix#2176
leofang wants to merge 7 commits into
NVIDIA:mainfrom
leofang:leof/ci-custom-driver-versions

leofang commented Jun 7, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jun 7, 2026

Uh oh!

leofang commented Jun 7, 2026

Uh oh!

github-actions Bot commented Jun 7, 2026

Preview will be ready when the GitHub Pages deployment is complete.

Uh oh!

leofang commented Jun 7, 2026

Uh oh!

leofang Jun 7, 2026

Uh oh!

leofang Jun 7, 2026

Uh oh!

leofang commented Jun 7, 2026

Uh oh!

rwgk Jun 7, 2026

Uh oh!

leofang commented Jun 7, 2026

Uh oh!

leofang commented Jun 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

leofang commented Jun 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

copy-pr-bot Bot commented Jun 7, 2026

Uh oh!

leofang commented Jun 7, 2026

Uh oh!

github-actions Bot commented Jun 7, 2026

Preview will be ready when the GitHub Pages deployment is complete.

Uh oh!

leofang commented Jun 7, 2026

Uh oh!

leofang Jun 7, 2026

Choose a reason for hiding this comment

Uh oh!

leofang Jun 7, 2026

Choose a reason for hiding this comment

Uh oh!

leofang commented Jun 7, 2026

Uh oh!

rwgk Jun 7, 2026

Choose a reason for hiding this comment

Uh oh!

leofang commented Jun 7, 2026

Uh oh!

leofang commented Jun 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

leofang commented Jun 7, 2026 •

edited

Loading