[FEA] Multi-node Out of Core Streaming KMeans API#2066
Conversation
…nto combine-batch
…nto combine-batch
…nto combine-batch
|
/ok to test 7cb1397 |
|
/ok to test a743386 |
|
/ok to test 2f15953 |
|
/ok to test 87e8819 |
|
/ok to test c856300 |
|
/ok to test 61808e4 |
|
/ok to test 20fee2e |
|
/ok to test defb785 |
| - nccl ${{ nccl_version }} | ||
| # ucx/ucxx are linked by the multi-GPU (MG) tests | ||
| - libucxx ${{ ucxx_version }} | ||
| - ucx |
There was a problem hiding this comment.
These are needed at build time, right? We also need to list these under the cached host env, under line 74.
There was a problem hiding this comment.
We also need to list these under the cached host env, under line 74.
@divyegala this is the place from where I removed it, right? Should I add it again?
|
/ok to test 8dd17f3 |
bdice
left a comment
bdice
left a comment
There was a problem hiding this comment.
Ah, this is an entirely new dependency on ucxx. We have several other tasks to handle.
https://github.com/NVIDIA/cuvs/blob/main/ci/release/update-version.sh needs changes like raft has.
Also reference how RAFT uses UCXX_VERSION and UCXX_BRANCH throughout the repo, we'll need to replicate all that infrastructure.
7 participants
Merge after #2015 and #2017
Allows a stream of input matrices per worker, that are further batched using the
streaming_batch_sizeparameter. Reasoning: We should be able to supply dask partitions (on host) directly without having to concatenate them into one consolidated matrix.As a part of this PR, we also unify the multi-GPU implementations into one (earlier the out of core implementation was separate).
Tests: We get rid of the separate out of core test file. The single MG testing unit is taking care of both out of core and on device matrices.