Reschedule control loop after metastore overload rejection

[fmassot]

Summary

• reschedule the next ControlPlanLoop when shard rebalancing returns a metastore error that does not require an actor restart
• preserve the existing supervisor restart behavior for errors with an uncertain transaction outcome
• add an actor-level regression test for a pre-dispatch TooManyRequests rejection

This is the first focused extraction from #6592.

Root cause

MetastoreError::TooManyRequests is classified as certainly aborted, so the control-plane actor correctly stays alive. However, the handler returned before scheduling the next periodic control tick. The actor therefore remained healthy while its control loop silently stopped.

The fix schedules one successor tick only when error conversion returns an outer Ok. Errors that return ActorExitStatus still restart the actor and do not leave an extra timer behind.

Scope

This PR intentionally excludes the generation fencing, watcher lifecycle, readiness, indexing scheduler, ingest router, and metastore admission changes from #6592.

Validation

• cargo test -p quickwit-control-plane --lib control_plane::tests::test_control_plan_loop_continues_after_too_many_requests -- --exact --nocapture (repeated three times)
• cargo test -p quickwit-control-plane --lib (120 passed)
• cargo +nightly fmt --all -- --check
• cargo clippy -p quickwit-control-plane --lib --tests -- -D warnings
• git diff --check

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2e64ace857

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Clean up initialized shards before retrying rebalances

When rebalance_shards fails in try_open_shards after init_shards succeeds but before open_shards_on_metastore_and_model records the shard (for example a metastore TooManyRequests), the initialized shard is neither inserted in the model nor closed. Rescheduling the control loop here makes every subsequent 5s tick allocate and initialize another shard with a new ULID while the metastore remains overloaded, so those unregistered shards/queues accumulate on the ingester until some separate retain-shards sync or restart cleans them up.

Useful? React with 👍 / 👎.