fix(ci3): don't crash CI when redis is unavailable in redis_setexz#24646
Closed
AztecBot wants to merge 1 commit into
Closed
fix(ci3): don't crash CI when redis is unavailable in redis_setexz#24646AztecBot wants to merge 1 commit into
AztecBot wants to merge 1 commit into
Conversation
redis_cli is a no-op when CI_REDIS_AVAILABLE=0 and never reads stdin, so the
gzip piped into it dies with SIGPIPE ('gzip: stdout: Broken pipe') and, under
'set -o pipefail', aborts the whole run at the first redis_setexz call in
bootstrap_ec2. Drain stdin instead when redis is unavailable.
Surfaced by the aztec-claude nightly barretenberg debug build, which runs on a
GitHub-hosted runner with no redis reachable.
Collaborator
Author
|
Automatically closing this stale claudebox draft PR (no updates for 5+ days). Re-open if still needed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
redis_setexzunconditionally pipesgzipintoredis_cli:When redis is unavailable (
CI_REDIS_AVAILABLE=0),redis_cliis a no-op that returns immediately without reading stdin. Thegzipon the left then getsSIGPIPE, printsgzip: stdout: Broken pipe, and exits non-zero. Becauseci3/source→source_optionssetsset -euo pipefail, that non-zero pipeline status aborts the entire run.This fires at the very first
redis_setexzcall inbootstrap_ec2:so CI dies at bootstrap, before any build work. All four callers (
bootstrap_ec2,cache_log,run_test_cmd,denoise) pipe data intoredis_setexz, so every one of them is exposed to the same crash in a redis-less environment.The fix guards on
CI_REDIS_AVAILABLE(mirroringredis_cli) and drains stdin when there is nowhere to store it, so producers never see a broken pipe. When redis is available, behaviour is unchanged.How this surfaced
The aztec-claude nightly barretenberg debug build (run 29141841707) fails instantly with exactly this error:
That job runs on a GitHub-hosted
ubuntu-latestrunner where redis is not reachable (no bastion SSH key), soCI_REDIS_AVAILABLE=0and the firstredis_setexzcall aborts the run. The identical nightly in aztec-packages is green because its runner has the CI secrets that open the redis tunnel, so this path is never exercised there — this change hardens ci3 for any redis-less environment.Test
Reproduced the SIGPIPE abort with
CI_REDIS_AVAILABLE=0underset -euo pipefail(run aborts before the next statement), then confirmed the patched function drains stdin and the pipeline exits 0.bash -n ci3/source_redispasses.Created by claudebox · group:
slackbot