Skip to content

fix(ci3): don't crash CI when redis is unavailable in redis_setexz#24646

Closed
AztecBot wants to merge 1 commit into
nextfrom
cb/ci3-redis-setexz-sigpipe
Closed

fix(ci3): don't crash CI when redis is unavailable in redis_setexz#24646
AztecBot wants to merge 1 commit into
nextfrom
cb/ci3-redis-setexz-sigpipe

Conversation

@AztecBot

Copy link
Copy Markdown
Collaborator

What

redis_setexz unconditionally pipes gzip into redis_cli:

function redis_setexz {
  gzip | redis_cli -x SETEX $1 $2 &>/dev/null
}

When redis is unavailable (CI_REDIS_AVAILABLE=0), redis_cli is a no-op that returns immediately without reading stdin. The gzip on the left then gets SIGPIPE, prints gzip: stdout: Broken pipe, and exits non-zero. Because ci3/sourcesource_options sets set -euo pipefail, that non-zero pipeline status aborts the entire run.

This fires at the very first redis_setexz call in bootstrap_ec2:

echo "CI booting..." | redis_setexz "$CI_LOG_ID" 300

so CI dies at bootstrap, before any build work. All four callers (bootstrap_ec2, cache_log, run_test_cmd, denoise) pipe data into redis_setexz, so every one of them is exposed to the same crash in a redis-less environment.

The fix guards on CI_REDIS_AVAILABLE (mirroring redis_cli) and drains stdin when there is nowhere to store it, so producers never see a broken pipe. When redis is available, behaviour is unchanged.

How this surfaced

The aztec-claude nightly barretenberg debug build (run 29141841707) fails instantly with exactly this error:

--- Run barretenberg-debug CI ---

gzip: stdout: Broken pipe
##[error]Process completed with exit code 1.

That job runs on a GitHub-hosted ubuntu-latest runner where redis is not reachable (no bastion SSH key), so CI_REDIS_AVAILABLE=0 and the first redis_setexz call aborts the run. The identical nightly in aztec-packages is green because its runner has the CI secrets that open the redis tunnel, so this path is never exercised there — this change hardens ci3 for any redis-less environment.

Note: this hardens the ci3 framework and removes the proximate crash. The aztec-claude mirror's nightly has a separate, non-code blocker — that repo lacks the AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / BUILD_INSTANCE_SSH_KEY secrets the EC2 build needs — which is an operator/infra action (provision the secrets in aztec-claude, or drop the mirrored workflow) and cannot be fixed by this PR.

Test

Reproduced the SIGPIPE abort with CI_REDIS_AVAILABLE=0 under set -euo pipefail (run aborts before the next statement), then confirmed the patched function drains stdin and the pipeline exits 0. bash -n ci3/source_redis passes.


Created by claudebox · group: slackbot

redis_cli is a no-op when CI_REDIS_AVAILABLE=0 and never reads stdin, so the
gzip piped into it dies with SIGPIPE ('gzip: stdout: Broken pipe') and, under
'set -o pipefail', aborts the whole run at the first redis_setexz call in
bootstrap_ec2. Drain stdin instead when redis is unavailable.

Surfaced by the aztec-claude nightly barretenberg debug build, which runs on a
GitHub-hosted runner with no redis reachable.
@AztecBot AztecBot added ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR. labels Jul 11, 2026
@AztecBot

Copy link
Copy Markdown
Collaborator Author

Automatically closing this stale claudebox draft PR (no updates for 5+ days). Re-open if still needed.

@AztecBot AztecBot closed this Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant