Skip to content

Recover fully from a partial reindex failure - #327

Open
damilolaedwards wants to merge 1 commit into
ethpandaops:masterfrom
damilolaedwards:fix/reindex-partial-failure-recovery
Open

Recover fully from a partial reindex failure#327
damilolaedwards wants to merge 1 commit into
ethpandaops:masterfrom
damilolaedwards:fix/reindex-partial-failure-recovery

Conversation

@damilolaedwards

Copy link
Copy Markdown

Summary

When one search index rebuilt successfully but another failed in the same
reindex pass, only the failed index was left parked. The one that
succeeded stayed live under the new embedding model even though
builtModel was never advanced to record that, so the runtime's own
bookkeeping no longer matched what was actually live. Recovery also
depended on the proxy's served model changing again on a later tick; if it
reverted to the model builtModel already recorded, the model-change guard
saw no difference and never retried, leaving the failed index parked
indefinitely with no way back short of a restart.

On a partial failure, every index the reindex pass touched is now parked,
not just the one that failed, so nothing is ever left live in a space the
runtime isn't tracking. The runtime also tracks that a reindex is
incomplete and the background refresher retries on every tick while it
is, regardless of whether the served model still looks different from
what's recorded, so a retry against the same model can still succeed and
fully restore service.

Test plan

  • go test -race ./pkg/searchruntime/...
  • New test drives a real reindex through a successful baseline, a
    partial failure where one index succeeds and another fails, and a
    recovery once the same model is retried
  • go build ./..., go vet ./...

When one search index rebuilt successfully but another failed in the
same reindex pass, only the failed index was left parked. The one
that succeeded stayed live under the new embedding model even though
builtModel was never advanced to record that, so the runtime's own
bookkeeping no longer matched what was actually live. Recovery also
depended on the proxy's served model changing again on a later tick;
if it reverted to the model builtModel already recorded, the
model-change guard saw no difference and never retried, leaving the
failed index parked indefinitely with no way back short of a restart.

On a partial failure, park every index the reindex pass touched, not
just the one that failed, so nothing is ever left live in a space the
runtime isn't tracking. Track that a reindex is incomplete and have
the background refresher retry on every tick while it is, regardless
of whether the served model still looks different from what's
recorded, so a retry against the same model can still succeed and
fully restore service.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant