[DRAFT] experiment: add test sharding

[daniel-sanche]

Experimenting with test sharding

Unit Tests

• if less than 10 packages are changed, execute one test per python version (existing behaviour)
• if more than 10 packages would be in a test, add an additional shard for each python runtime, and spread out packages between them
• use an upper limit of 10 shards (per runtime), with packages evenly distributed
• adds an end unit test complete step, which is only green if all shards pass. This can be our new required check for unit tests

System Tests

• Previously system tests run sequentially as a single script
• With this change, they run in parallel, reported as separate Targets in TestGrid
• test logs are still all combined into the primary build.log for viewing in one place

Fail Fast

• if any single unit test shard or system test job fails, the remaining end early to provide quick feedback

[gemini-code-assist]

Code Review

This pull request introduces parallel execution for system tests and implements a sharding mechanism for CI jobs, including a new Python script to group packages and updates to the test runner script. Feedback on these changes focuses on improving reliability and safety: first, by using an EXIT trap in .kokoro/system.sh to guarantee cleanup of isolated gcloud configuration directories in case of test failures; and second, by avoiding global toggles of set -e in ci/run_conditional_tests.sh and instead capturing test exit codes using the || operator.

[gemini-code-assist]

Since run_package_test runs with set -e enabled, any test failure or error will cause the subshell to exit immediately. This prevents the manual cleanup at the end of the function from running, leading to leaked temporary directories in /tmp.

Using an EXIT trap ensures that the temporary directory is reliably cleaned up when the subshell exits, regardless of whether the tests succeeded or failed.

Suggested change

	# Isolate gcloud config for parallel execution
	export CLOUDSDK_CONFIG=$(mktemp -d)
	# Isolate gcloud config for parallel execution
	export CLOUDSDK_CONFIG=$(mktemp -d)
	trap 'rm -rf "${CLOUDSDK_CONFIG}"' EXIT

[gemini-code-assist]

With the EXIT trap handling the cleanup of CLOUDSDK_CONFIG automatically, this manual cleanup is redundant and can be safely removed.

[gemini-code-assist]

Toggling set -e globally can lead to unexpected side effects or mask errors in other parts of the script if not carefully managed. A more idiomatic and safer way to capture the exit status of a command under set -e is to use the || operator, which prevents the shell from exiting on failure without needing to disable set -e globally.

Suggested change

	set +e
	${test_script}
	ret=$?
	set -e
	ret=0
	${test_script} || ret=$?