Skip to content

Cosmos: run query response status checks inside execution strategy retries#38591

Merged
AndriySvyryd merged 15 commits into
mainfrom
copilot/fix-partition-key-range-gone
Jul 17, 2026
Merged

Cosmos: run query response status checks inside execution strategy retries#38591
AndriySvyryd merged 15 commits into
mainfrom
copilot/fix-partition-key-range-gone

Conversation

Copilot AI commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Fixes #38032

Cosmos queries could surface transient 410 Gone (PartitionKeyRangeGone / substatus 1002) during partition splits because EnsureSuccessStatusCode() was executed outside EF’s retry boundary for stream-based query paths. As a result, transient response-status failures were not retried even though the execution strategy treats Gone as retryable.

  • Retry boundary correction (stream query path)

    • Updated CosmosClientWrapper.DocumentAsyncEnumerable.AsyncEnumerator.MoveNextAsync so ReadNextAsync and EnsureSuccessStatusCode() execute within _executionStrategy.ExecuteAsync(...).
    • On failure, the ResponseMessage is disposed before rethrow to avoid leaking failed-attempt responses across retries.
  • Retry boundary + iterator recreation correction (paging query path)

    • Updated PagingQueryingEnumerable.AsyncEnumerator.MoveNextAsync to run ReadNextAsync + EnsureSuccessStatusCode() under _cosmosQueryContext.ExecutionStrategy.ExecuteAsync(...).
    • Moved CreateQuery(...) inside the retry delegate so each retry attempt uses a new FeedIterator.
    • Retries now execute with the current continuation token as strategy state, ensuring retries resume from the last successful page and avoid replaying already-returned results.
    • Added the same failure-path disposal behavior for ResponseMessage.
  • Regression coverage

    • Existing unit coverage in CosmosClientWrapperTest verifies transient Gone status is retried for stream query execution.
    • Added a paging-focused regression test (ToPageAsync_retries_with_new_feed_iterator_from_last_successful_continuation_token) that simulates:
      1. first page success with continuation token,
      2. transient Gone on next-page read,
      3. retry with a recreated iterator using the same continuation token.
    • Verifies no duplicate results are returned across the retry boundary.
_responseMessage = await _executionStrategy.ExecuteAsync(
    _query,
    static async (_, query, ct) =>
    {
        var response = await query.ReadNextAsync(ct).ConfigureAwait(false);
        try
        {
            response.EnsureSuccessStatusCode();
        }
        catch
        {
            response.Dispose();
            throw;
        }

        return response;
    },
    null,
    cancellationToken).ConfigureAwait(false);

Copilot AI and others added 9 commits July 9, 2026 01:34
Co-authored-by: AndriySvyryd <6539701+AndriySvyryd@users.noreply.github.com>
Co-authored-by: AndriySvyryd <6539701+AndriySvyryd@users.noreply.github.com>
Co-authored-by: AndriySvyryd <6539701+AndriySvyryd@users.noreply.github.com>
Co-authored-by: AndriySvyryd <6539701+AndriySvyryd@users.noreply.github.com>
Co-authored-by: AndriySvyryd <6539701+AndriySvyryd@users.noreply.github.com>
Co-authored-by: AndriySvyryd <6539701+AndriySvyryd@users.noreply.github.com>
Co-authored-by: AndriySvyryd <6539701+AndriySvyryd@users.noreply.github.com>
Co-authored-by: AndriySvyryd <6539701+AndriySvyryd@users.noreply.github.com>
Co-authored-by: AndriySvyryd <6539701+AndriySvyryd@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix Cosmos DB 410 gone error with hierarchical partitioning Cosmos: run query response status checks inside execution strategy retries Jul 9, 2026
Copilot AI requested a review from AndriySvyryd July 9, 2026 02:03
…tion token

Co-authored-by: AndriySvyryd <6539701+AndriySvyryd@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adjusts Cosmos query execution so that ResponseMessage.EnsureSuccessStatusCode() runs inside EF Core’s execution-strategy retry boundary for stream-based and paging query paths, allowing transient 410 Gone responses (e.g., during partition splits) to be retried correctly.

Changes:

  • Wrapped ReadNextAsync + EnsureSuccessStatusCode() in _executionStrategy.ExecuteAsync(...) for the stream query enumerator (CosmosClientWrapper).
  • Wrapped paging ReadNextAsync + EnsureSuccessStatusCode() in the paging enumerator’s execution strategy, and moved iterator creation into the retry delegate to ensure a fresh FeedIterator per attempt.
  • Added/extended Cosmos unit tests to validate retries and that paging retries resume from the last successful continuation token without duplicating results.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
test/EFCore.Cosmos.Tests/Storage/Internal/CosmosClientWrapperTest.cs Adds regression tests and helper iterators/messages to validate retry behavior for stream and paging query paths.
src/EFCore.Cosmos/Storage/Internal/CosmosClientWrapper.cs Moves response status validation inside the execution strategy for the stream query enumerator and disposes failed responses before retry.
src/EFCore.Cosmos/Query/Internal/CosmosShapedQueryCompilingExpressionVisitor.PagingQueryingEnumerable.cs Ensures paging reads + status validation happen inside execution strategy retries and recreates FeedIterator per retry attempt using the current continuation token.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Comment thread src/EFCore.Cosmos/Storage/Internal/CosmosClientWrapper.cs
…classes

Co-authored-by: AndriySvyryd <6539701+AndriySvyryd@users.noreply.github.com>
Copilot AI requested a review from AndriySvyryd July 11, 2026 03:25
@AndriySvyryd
AndriySvyryd marked this pull request as ready for review July 11, 2026 03:43
@AndriySvyryd
AndriySvyryd requested a review from a team as a code owner July 11, 2026 03:43
Copilot AI review requested due to automatic review settings July 11, 2026 03:43

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment thread test/EFCore.Cosmos.Tests/Storage/Internal/CosmosClientWrapperTest.cs Outdated
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 13, 2026 20:04
@AndriySvyryd
AndriySvyryd requested a review from cincuranet July 13, 2026 20:05

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 16, 2026 17:38

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

@AndriySvyryd
AndriySvyryd merged commit 346d9e0 into main Jul 17, 2026
16 checks passed
@AndriySvyryd
AndriySvyryd deleted the copilot/fix-partition-key-range-gone branch July 17, 2026 09:45
@dotnet-milestone-bot dotnet-milestone-bot Bot added this to the 11.0-preview7 milestone Jul 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cosmos DB: 410 Gone (Substatus 1002) - PartitionKeyRangeGone with Hierarchical Partitioning

4 participants