Group Worker Processes#1778
Open
BryonLewis wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hopefully this suplants #1768
Summary
Cancelling a running pipeline/training job could leave the Girder job stuck in Cancelling indefinitely. The cancel monitor only signaled the shell PID; VIAME/KWIVER children kept running and held stdout open, so
stream_subprocessnever finished acknowledging the cancel.This fixes that at the worker: launch the subprocess in a new session and terminate the whole process group on cancel, so the job actually stops and moves to Canceled.
Supersedes / is an alternative to #1768 for the common “stuck Cancelling while a worker is running” failure mode.
Problem
When a user cancels a job:
CANCELINGand Celery revokes the task.shell=True, soPopen’s PID is a shell that then starts KWIVER/training children.Previously the monitor only did
process.send_signal(...)on that shell. The shell could die while children kept writing to the pipe. The main thread blocked forever instdout.readline, never reachedupdateStatus(CANCELED), and the UI sat on Cancelling for as long as the orphaned work kept running (often hours/days).Fix
In
dive_tasks/utils.pystream_subprocess:start_new_session=Trueso the shell and its descendants share a process group._kill_process_groupsendsSIGTERMthenSIGKILLviaos.killpg.stdoutafter the kill so a stuckreadlineunblocks if a stray writer remains.CANCELEDand raiseCanceledError.#1768 treats the sympton and not the cause of the problem, this PR is closer to fixing the cause (subprocesses of VIAME not being properlly killed)