[Task] Investigate slow integrate-test stalls in message sender barrier

[imbajin]

Background

The Integrate test job for HugeGraph Computer can take a long time even though the integration suite is small. The slow part is usually not the number of tests, but a long wait in the message/input synchronization path.

An observed CI log repeatedly prints:

EtcdClient - Wait for keys with prefix 'BSP_WORKER_INPUT_DONE' and timeout 86400000ms, expect 1 keys but actual got 0 keys

The same log shows the worker entering input step and starting vertex message sending before the wait:

WorkerService inputstep started
MessageSendManager - Start sending message(type=VERTEX)

So the master is waiting for the worker's BSP_WORKER_INPUT_DONE signal, but the worker has not reached Bsp4Worker.workerInputDone() yet.

Initial code pointers

• CI runs mvn test -P integrate-test -ntp in .github/workflows/computer-ci.yml.
• The integrate-test profile includes IntegrateTestSuite, which currently contains SenderIntegrateTest.
• SenderIntegrateTest has only a few cases, but testOneWorkerWithBusyClient() intentionally slows the send path by wrapping the client's send function with Thread.sleep(100).
• WorkerInputManager.loadGraph() sends vertices and edges first. Only after it returns does WorkerService.inputstep() call bsp4Worker.workerInputDone().
• ComputerOptions.BSP_WAIT_WORKERS_TIMEOUT and BSP_WAIT_MASTER_TIMEOUT default to 24 hours, so a hidden sender/session/input problem can become a very slow CI wait instead of a fast, actionable failure.

Relevant files:

.github/workflows/computer-ci.yml
computer/computer-test/src/main/java/org/apache/hugegraph/computer/suite/integrate/IntegrateTestSuite.java
computer/computer-test/src/main/java/org/apache/hugegraph/computer/suite/integrate/SenderIntegrateTest.java
computer/computer-core/src/main/java/org/apache/hugegraph/computer/core/input/WorkerInputManager.java
computer/computer-core/src/main/java/org/apache/hugegraph/computer/core/worker/WorkerService.java
computer/computer-core/src/main/java/org/apache/hugegraph/computer/core/sender/QueuedMessageSender.java
computer/computer-core/src/main/java/org/apache/hugegraph/computer/core/sender/MessageSendManager.java
computer/computer-api/src/main/java/org/apache/hugegraph/computer/core/config/ComputerOptions.java

Related prior symptom: #203 reported The origin future must be null in SenderIntegrateTest. That may be in the same control-message/future/session area, but this task is specifically about the slow CI wait and fail-fast/debuggability of the integration test.

Suggested investigation

1. Reproduce the integration suite with etcd available:

   cd computer
   mvn test -P integrate-test -Dtest=IntegrateTestSuite -ntp

2. Confirm which test case spends time before BSP_WORKER_INPUT_DONE. Start with SenderIntegrateTest#testOneWorkerWithBusyClient.

3. Trace the input path:

   SenderIntegrateTest
     -> WorkerService.execute()
     -> WorkerService.inputstep()
     -> WorkerInputManager.loadGraph()
     -> MessageSendManager.startSend()/finishSend()
     -> QueuedMessageSender.send()
     -> Bsp4Worker.workerInputDone()
   

4. Check whether START/FINISH control futures in QueuedMessageSender can be left stale, completed late, or hidden behind the sender thread. The old [Bug] The origin future must be null #203 stack around futureRef is a useful clue.

5. Make the test fail fast and print useful diagnostics. Possible directions:

   • set much smaller bsp.wait_workers_timeout / bsp.wait_master_timeout for integration tests;
   • add a JUnit/test-level timeout around each integration case;
   • dump worker/master thread states when the input barrier is not reached;
   • ensure sender exceptions propagate to both the worker future and the master-side wait;
   • replace the sleep-based busy-client simulation with a more deterministic back-pressure or blocked-client fixture.

Expected result

• Integration tests should not spend many minutes printing only BSP_WORKER_INPUT_DONE wait logs.
• If the sender/input path is broken, the test should fail quickly with an actionable error and enough thread/session state to locate the failing component.
• The slow/busy-client path should have regression coverage so future changes do not reintroduce the long wait.

Newcomer scope

This is a good newcomer task because the suspected area is narrow: one integration suite, the input-step barrier, and the message sender control future path. A complete fix does not need a large algorithm or distributed-runtime redesign; first improving timeout/diagnostics and then isolating the sender/session condition would already be valuable.

[wipheg]

Hi @imbajin,

I'd like to take a look at this issue if no one is working on it yet.

I'll start by reproducing the integration test locally and checking where the worker gets stuck before reaching BSP_WORKER_INPUT_DONE, especially around the sender/input path mentioned in the description.

Once I have some findings, I'll post an update here.

Thanks!

[PhoenixRushi]

Hi @imbajin,

I'd like to investigate this issue.

From my initial review, it seems that the worker enters the input/message-sending flow but may not reach workerInputDone(), which causes the master to keep waiting for `BSP_WORKER_INPUT_DONE.

I'll try to reproduce the integration test locally and trace the execution path around WorkerInputManager, MessageSendManager, and QueuedMessageSender to identify where the flow is getting blocked.

I'll share my findings here as I dig deeper.

Thanks !

[imbajin]

@wipheg @PhoenixRushi Thanks both for your interest.

This issue is mainly an investigation task, so it would be helpful if both of you
can take a look and share findings here.

To avoid duplicated work, please post short updates when you have:

• a reproducible command/environment;
• the exact test case that stalls;
• thread/session logs around WorkerInputManager, MessageSendManager, or
  QueuedMessageSender;
• a suspected root cause.

Once one of you has a clear root cause and a fix direction, feel free to open a
PR and link it here. The other person is very welcome to help verify the PR or
review the diagnosis.

Please coordinate in this thread before opening overlapping PRs.

[PhoenixRushi]

Thanks for the clarification.

I'll start by reproducing the issue locally and gathering the requested details, including the exact test behavior and logs around WorkerInputManager, MessageSendManager, and QueuedMessageSender.

I'll post my findings here as I progress so we can avoid overlapping work.