[Python] Fix Python Linux distribtests copy conflicts (grpc#39558)

sreenithi · copybara-github · commit 78cfd39afe6d · 2025-05-22T03:34:38.000-07:00
Recently, the Python Linux distribtests have been having flake failures with an error like: ``` + '[' artifacts '!=' '' ']' + cp -r /tmp/tmp.nb7ZJvoaIS/artifacts /tmpfs/altsrc/github/grpc cp: cannot create regular file '/tmpfs/altsrc/github/grpc/artifacts/grpcio-1.73.0.dev0-cp311-cp311-linux_armv7l.whl': File exists ``` The root cause was found to be one of our recent changes while adding support for musllinux_1_1_aarch64 wheels. Some context on the Linux distribtests build - As part of building the Linux artifacts (`.whl` files), we have [3 types of build targets](https://github.com/grpc/grpc/blob/master/tools/run_tests/task_runner.py#L29-L32) - `artifact_targets`, `distrib_targets` and `package_targets`. Recently in Q1, we added support for `musllinux_1_1_aarch64` wheels and separated the build for these wheels into a separate job by using exclude parameters in the original `distribtests_python` config: [Reference](https://github.com/grpc/grpc/blob/master/tools/internal_ci/linux/grpc_distribtests_python.cfg#L30). While both '[artifact_targets](https://github.com/grpc/grpc/blob/master/tools/internal_ci/linux/grpc_distribtests_python.sh#L44)' and '[distrib_targets](https://github.com/grpc/grpc/blob/master/tools/internal_ci/linux/grpc_distribtests_python.sh#L73)' use these exclude filters during invocation, these exclude filters were missing for the '[package_targets](https://github.com/grpc/grpc/blob/master/tools/internal_ci/linux/grpc_distribtests_python.sh#L56)' causing both package builds to run as below: ``` tools/run_tests/task_runner.py -f package linux python -x build_packages/sponge_log.xml 2025-05-13 16:08:21,278 START: Building targets. Will build 2 targets: python_package, labels ['package', 'python', 'linux'] python_package_musllinux_1_1_aarch64, labels ['package', 'python', 'linux', 'musllinux_1_1', 'aarch64'] ``` But looking at [build_package_python.sh](https://github.com/grpc/grpc/blob/master/tools/run_tests/artifacts/build_package_python.sh#L24) shows that the job only copies all the built artifacts starting with `python_` with no further filters on the type of architecture, meaning that both jobs were trying to copy all the python built artifacts to the same mounted directory parallely, causing these copy conflicts/race conditions. To resolve this, the following fixes are done in this PR: * add exclude flags for musllinux aarch64 while building package_targets * improve package_targets job to include the exclude flag and selectively copy files, such that `python_package_musllinux_1_1_aarch64` job only copies those artifacts and the `python_package` job doesn't copy the musllinux aarch64 artifacts * fixed exclude flags logic as it wasn't working as expected Closes grpc#39558 COPYBARA_INTEGRATE_REVIEW=grpc#39558 from sreenithi:fix_linux_distribtest_cp c51d74e PiperOrigin-RevId: 761890365
diff --git a/tools/internal_ci/linux/grpc_distribtests_python.sh b/tools/internal_ci/linux/grpc_distribtests_python.sh
@@ -49,12 +49,14 @@ mkdir -p input_artifacts
 cp -r artifacts/* input_artifacts/ || true
 
 # This step simply collects python artifacts from subdirectories of input_artifacts/ and copies them to artifacts/
-if [[ "${IS_AARCH64_MUSL}" == "True" ]]; then
-  # Not using TASK_RUNNER_EXTRA_FILTERS since we don't have a target with presubmit tag.
-  tools/run_tests/task_runner.py -f package linux python musllinux_1_1 aarch64 -x build_packages/sponge_log.xml || FAILED="true"
-else
-  tools/run_tests/task_runner.py -f package linux python -x build_packages/sponge_log.xml || FAILED="true"
-fi
+
+# PythonPackage targets do not support the `presubmit` label.
+# For this reason we remove `presubmit` label selector from TASK_RUNNER_EXTRA_FILTERS,
+# which looks like TASK_RUNNER_EXTRA_FILTERS="presubmit -e aarch64 musllinux_1_1"
+# for a presubmit with an exclude filter.
+PACKAGE_TASK_RUNNER_EXTRA_FILTERS="${TASK_RUNNER_EXTRA_FILTERS//presubmit /}"
+
+tools/run_tests/task_runner.py -f package linux python ${PACKAGE_TASK_RUNNER_EXTRA_FILTERS} -x build_packages/sponge_log.xml || FAILED="true"
 
 # the next step expects to find the artifacts from the previous step in the "input_artifacts" folder.
 # in addition to that, preserve the contents of "artifacts" directory since we want kokoro
diff --git a/tools/run_tests/artifacts/build_package_python.sh b/tools/run_tests/artifacts/build_package_python.sh
@@ -13,15 +13,21 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-set -ex
+set -eux
 
 cd "$(dirname "$0")/../../.."
 
 mkdir -p artifacts/
 
 # All the python packages have been built in the artifact phase already
 # and we only collect them here to deliver them to the distribtest phase.
-cp -r "${EXTERNAL_GIT_ROOT}"/input_artifacts/python_*/* artifacts/ || true
+find "${EXTERNAL_GIT_ROOT}"/input_artifacts/ \
+    -maxdepth 1 \
+    -type d \
+    -name "${ARTIFACT_PREFIX}*" \
+    -not -name "${EXCLUDE_PATTERN}" \
+    -print0 \
+        | xargs -0 -I% find % -type f -maxdepth 1 -exec cp -v {} ./artifacts \;
 
 # TODO: all the artifact builder configurations generate a grpcio-VERSION.tar.gz
 # source distribution package, and only one of them will end up
diff --git a/tools/run_tests/artifacts/package_targets.py b/tools/run_tests/artifacts/package_targets.py
@@ -165,15 +165,23 @@ def build_jobspec(self, inner_jobs=None):
         dockerfile_dir = (
             "tools/dockerfile/grpc_artifact_python_manylinux2014_x64"
         )
+        environ = {
+            "PYTHON": "/opt/python/cp39-cp39/bin/python",
+            "ARTIFACT_PREFIX": "python_",
+            "EXCLUDE_PATTERN": "python_musllinux_1_1_aarch64_*",
+        }
         if "musllinux_1_1" in self.platform and "aarch64" in self.arch:
             dockerfile_dir = (
                 "tools/dockerfile/grpc_artifact_python_musllinux_1_1_aarch64"
             )
+            environ["ARTIFACT_PREFIX"] = "python_musllinux_1_1_aarch64_"
+            environ["EXCLUDE_PATTERN"] = ""
+
         return create_docker_jobspec(
             self.name,
             dockerfile_dir,
             "tools/run_tests/artifacts/build_package_python.sh",
-            environ={"PYTHON": "/opt/python/cp39-cp39/bin/python"},
+            environ=environ,
         )
 
 
diff --git a/tools/run_tests/task_runner.py b/tools/run_tests/task_runner.py
@@ -115,7 +115,10 @@ def _create_build_map():
 targets = [t for t in targets if all(f in t.labels for f in args.filter)]
 
 # Exclude target if it has ALL of the specified exclude labels.
-targets = [t for t in targets if not all(l in args.exclude for l in t.labels)]
+if args.exclude:
+    targets = [
+        t for t in targets if not all(l in t.labels for l in args.exclude)
+    ]
 
 print("Will build %d targets:" % len(targets))
 for target in targets:
