Skip to content

HDDS-15766. Make Recon OM DB large tarball transfer reliable#10696

Open
devmadhuu wants to merge 5 commits into
apache:masterfrom
devmadhuu:HDDS-15766
Open

HDDS-15766. Make Recon OM DB large tarball transfer reliable#10696
devmadhuu wants to merge 5 commits into
apache:masterfrom
devmadhuu:HDDS-15766

Conversation

@devmadhuu

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Problem:
Recon keeps its own copy of the OM DB and syncs it two ways: incremental delta updates (getDBUpdates) for the normal case, and a full OM DB snapshot download as the fallback when delta sync cannot be used (Recon fell too far behind, OM no longer has the requested updates, etc.).

The full-snapshot fallback was the weak part:

  • It used a single one-shot GET /dbCheckpoint call that streamed the entire OM DB as one tar.
  • Any mid-stream failure restarted the download from byte 0.
  • It always transferred the complete DB, even the SST files Recon already had.

On large clusters the OM DB is big, so this approach was slow and fragile.
Approach: Instead of maintaining Recon's own download logic, Recon now reuses the same incremental bootstrap mechanism an OM follower uses to catch up from the OM leader:

  • A chunked POST /v2/dbCheckpoint request that carries a toExcludeList[] of SST files the caller already has.
  • The leader only sends the missing files and hard-links the rest (dedup), ending the stream with a completion marker.
  • Resume/retry behavior comes from the shared, proven OM code path rather than a Recon-specific re-implementation.

On top of the follower behavior, Recon seeds the exclude list from its own currently-installed (live) OM DB before the download starts. RocksDB SST file numbers are content-stable within a DB lineage, so a 000123.sst Recon already has is byte-identical to the leader's. Those SSTs are hard-linked into the candidate dir and advertised in the exclude list, so even a "full" snapshot only transfers the SST files that actually changed. The assembled DB is always complete and correct — a full snapshot still triggers a full task reprocess in Recon, exactly as before; only the bytes on the wire go down.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15766

How was this patch tested?

Testing:

  • New unit tests TestReconRDBSnapshotProvider
  • Updated TestOzoneManagerServiceProviderImpl— drives the new provider path for success and failure cases.
  • Existing TestReconWithOzoneManager.testOmDBSyncWithSeqNumberMismatch provides end-to-end coverage of the full-snapshot fallback against a mini-cluster
  • TestReconWithOzoneManagerHA updated to drop the stale v1-URL assertion, keeping its end-to-end leader-sync verification.

@devmadhuu devmadhuu changed the title HDDS-15766 - Recon - Make Recon OM DB large tar ball transfer reliable. HDDS-15766. Recon - Make Recon OM DB large tar ball transfer reliable. Jul 9, 2026
- TestReconRDBSnapshotProvider: guard File.list() results with requireNonNull.
- OzoneManagerServiceProviderImpl: make omSnapshotDBParentDir a local variable.
@devmadhuu devmadhuu marked this pull request as ready for review July 9, 2026 11:53
@adoroszlai adoroszlai changed the title HDDS-15766. Recon - Make Recon OM DB large tar ball transfer reliable. HDDS-15766. Make Recon OM DB large tar ball transfer reliable. Jul 9, 2026
@adoroszlai adoroszlai changed the title HDDS-15766. Make Recon OM DB large tar ball transfer reliable. HDDS-15766. Make Recon OM DB large tarball transfer reliable Jul 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant