ci(test): experiment with -n 8 xdist workers#1500
Closed
ramakrishnap-nv wants to merge 7 commits into
Closed
Conversation
Add pytest-xdist to test_python_common dependencies and pass -n auto to the non-nightly cuopt pytest invocation to parallelize tests across available CPUs. Server tests are excluded due to fixed-port binding (localhost:18900). The run_cuopt_pytests.sh script already cd-s into python/cuopt/cuopt where xdist+coverage is known to work. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Ramakrishna Prabhu <ramakrishnap@nvidia.com>
-n auto caused two failures: - cudaErrorInvalidDevice: too many workers competing for the same GPU - warmstart timeout: two workers starting gRPC servers on the same port Switch to -n 2 to limit GPU contention. Fix the gRPC port conflict in _start_grpc_server_fixture by adding the xdist worker ID (from PYTEST_XDIST_WORKER) to the port, giving each worker a unique port within the 100-unit gap between fixture classes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Ramakrishna Prabhu <ramakrishnap@nvidia.com>
-n 2 passed cleanly; try -n 4 for more parallelism. The CI GPUs (L4/24GB, H100/80GB, RTX Pro 6000/48GB) have enough VRAM for 4 concurrent CUDA contexts without contention. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Ramakrishna Prabhu <ramakrishnap@nvidia.com>
-n 4 is untested across all GPU types; stay conservative at -n 2 which is known clean. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Ramakrishna Prabhu <ramakrishnap@nvidia.com>
Add -n 2 to run_cuopt_server_pytests.sh. Fix the port conflict by introducing _worker_port() in utils.py, which offsets the base port (5555) by the xdist worker ID (from PYTEST_XDIST_WORKER). Both the cuoptproc fixture and RequestClient default port now use _worker_port() so each worker gets its own server instance on a unique port. Falls back to 5555 when not running under xdist. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Ramakrishna Prabhu <ramakrishnap@nvidia.com>
bdice notes RAPIDS generally uses -n 6 or -n 8; safe max depends on GPU memory (target: 16GB). Testing -n 8 for both cuopt and server tests to measure speedup vs stability tradeoff. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Ramakrishna Prabhu <ramakrishnap@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
ramakrishnap-nv
changed the base branch from
feat/pytest-xdist-python-tests
to
main
July 1, 2026 21:05
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Testing
-n 8for bothrun_cuopt_pytests.shandrun_cuopt_server_pytests.sh, per bdice's comment on #1486:Base PR #1486 uses
-n 2(conservative). This PR tests whether-n 8is stable across all GPU types in the matrix, including the V100 (16GB) and L4 (24GB).What to watch
cudaErrorInvalidDeviceorcudaErrorMemoryAllocationin routing/LP tests → GPU pressure too hightest_cpu_only_execution.py→ port gap insufficient (gap is 100 units, safe for up to 99 workers)-n 2baseline (~3m 48s cuopt, ~5m 34s server)🤖 Generated with Claude Code