Skip to content

Group Worker Processes#1778

Open
BryonLewis wants to merge 1 commit into
mainfrom
group-worker-process
Open

Group Worker Processes#1778
BryonLewis wants to merge 1 commit into
mainfrom
group-worker-process

Conversation

@BryonLewis

Copy link
Copy Markdown
Collaborator

Hopefully this suplants #1768

Summary

Cancelling a running pipeline/training job could leave the Girder job stuck in Cancelling indefinitely. The cancel monitor only signaled the shell PID; VIAME/KWIVER children kept running and held stdout open, so stream_subprocess never finished acknowledging the cancel.
This fixes that at the worker: launch the subprocess in a new session and terminate the whole process group on cancel, so the job actually stops and moves to Canceled.
Supersedes / is an alternative to #1768 for the common “stuck Cancelling while a worker is running” failure mode.

Problem

When a user cancels a job:

  1. Girder marks the job CANCELING and Celery revokes the task.
  2. The worker’s cancel monitor detects that and tries to stop the subprocess.
  3. Most DIVE tasks launch pipelines with shell=True, so Popen’s PID is a shell that then starts KWIVER/training children.
    Previously the monitor only did process.send_signal(...) on that shell. The shell could die while children kept writing to the pipe. The main thread blocked forever in stdout.readline, never reached updateStatus(CANCELED), and the UI sat on Cancelling for as long as the orphaned work kept running (often hours/days).

Fix

In dive_tasks/utils.py stream_subprocess:

  • Default start_new_session=True so the shell and its descendants share a process group.
  • On cancel, _kill_process_group sends SIGTERM then SIGKILL via os.killpg.
  • Close stdout after the kill so a stuck readline unblocks if a stray writer remains.
  • On cancel, wait with a timeout, escalate the group kill if needed, then set CANCELED and raise CanceledError.

#1768 treats the sympton and not the cause of the problem, this PR is closer to fixing the cause (subprocesses of VIAME not being properlly killed)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant