HDDS-15765. Fail fast when --node-id is omitted for OM compact/defrag on an HA cluster by chihsuan · Pull Request #10699 · apache/ozone

chihsuan · 2026-07-09T15:04:13Z

What changes were proposed in this pull request?

On an HA OM cluster, ozone repair om compact and ozone admin om snapshot defrag without --node-id do not fail cleanly; they hang for a long time before eventually timing out.

Root cause: with no --node-id/--service-id, OMNodeDetails.getOMNodeDetailsFromConf(conf, null, null) falls back to the base config key ozone.om.address, which in an HA deployment defaults to 0.0.0.0:9862. That default is the OM server bind address (bind to all interfaces) and is meaningless as a client target. Because it resolves to a non-null (but useless) address, the tool builds a valid-looking OMNodeDetails for 0.0.0.0 and proceeds to the RPC, which then retries in the Hadoop IPC retry loop and blocks.

This PR adds a fail-fast check in both CLI commands: when --node-id is omitted but the config defines an HA service (OmUtils.isServiceIdsDefined(conf)), the command exits immediately and asks the operator to specify --node-id, instead of resolving to 0.0.0.0. Each command reports the error with its own existing convention.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15765

How was this patch tested?

New unit tests TestCompactOMDB#testCompactHAWithoutNodeIdFailsFast and TestDefragSubCommand#testDefragHAWithoutNodeIdFailsFast
Existing TestCompactOMDB, TestDefragSubCommand, and TestOMNodeDetails still pass, and checkstyle is clean.
Manual end-to-end run on the compose/ozone-ha cluster (om1/om2/om3, service.ids=omservice):

Command	Before	After
`ozone repair om compact --cf keyTable` (no `--node-id`)	hangs until timeout	returns immediately: `This is an HA OM cluster; specify --node-id ...`
`ozone admin om snapshot defrag` (no `--node-id`)	hangs until timeout	returns immediately: `Error: This is an HA OM cluster; specify --node-id ...`
`ozone repair om compact --cf keyTable --service-id omservice --node-id om3`	issues request	unchanged: `Compaction request issued for om.db of om node: om3`
`ozone admin om snapshot defrag --service-id omservice --node-id om3`	reaches `om3:9862`	unchanged: reaches real `om3:9862` (not `0.0.0.0`)

Generated-by: Claude Code (Opus 4.8)

… on an HA cluster

…ests

rich7420

LGTM! thanks for the patch

sreejasahithi

Thanks @chihsuan for fixing this

CommandLine descriptions or the --help text for both commands does not mention the HA --node-id requirement.
It would be better to add there as well.
Should HA nodeId validation failure exit non-zero, or is exit 0 ok here?

adoroszlai · 2026-07-13T12:24:48Z

Should HA nodeId validation failure exit non-zero, or is exit 0 ok here?

Good point.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

chihsuan · 2026-07-13T12:50:39Z

Thanks for raising this! @sreejasahithi

CommandLine descriptions or the --help text for both commands does not mention the HA --node-id requirement.
It would be better to add there as well.

Sounds good! I updated the --node-id descriptions for both commands in e6758ff.

Should HA nodeId validation failure exit non-zero, or is exit 0 ok here?

Yeah, I agree. I considered returning a non-zero value, but the existing OMNodeDetails validation returns exit code 0, so I kept the new validation consistent. I can change it in this PR if you prefer; otherwise, we can address exit codes consistently in a follow-up. What do you think?

sreejasahithi · 2026-07-14T16:43:27Z

Yeah, I agree. I considered returning a non-zero value, but the existing OMNodeDetails validation returns exit code 0, so I kept the new validation consistent. I can change it in this PR if you prefer; otherwise, we can address exit codes consistently in a follow-up. What do you think?

I am fine tracking consistent non zero exit codes in a follow up Jira if you want to keep this PR scoped for fail fast fix for missing --node-id.

sreejasahithi

Please also update the --help outputs of the commands in the respective docs as they still show the old --node-id description.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

chihsuan · 2026-07-15T09:32:06Z

I am fine tracking consistent non zero exit codes in a follow up Jira if you want to keep this PR scoped for fail fast fix for missing --node-id.

Thanks @sreejasahithi I created HDDS-15865 to track consistent non-zero exit codes for OM compact and snapshot defrag validation failures.

HDDS-15765. Update command help output in docs

I’ve also updated the checked-in --help output for both commands. 4747b6a

sreejasahithi

LGTM

sarvekshayr · 2026-07-15T12:22:23Z

Thanks @chihsuan for the patch, @rich7420 and @sreejasahithi for the review.

* master: (53 commits) HDDS-15876. Remove ozone-csi (apache#10780) HDDS-15508. Add 2.2.0 to compatibility/upgrade test (apache#10782) HDDS-15867. [JDK25] Bump byte-buddy to 1.18.11 (apache#10776) HDDS-15809. Validate x-amz-copy-source-range on UploadPartCopy (apache#10711). HDDS-15678. OFS isDirectory/isFile should not trigger pipeline refresh or return block locations (apache#10741) HDDS-15171. Add available space check on follower during bootstrap. (apache#10185) HDDS-14043. Fix ls -e UnsupportedOperationException on ofs/o3fs (apache#10209) HDDS-15709. Replace random name suffix with uniqueObjectName (apache#10773) HDDS-15849. Force channel shutdown in XceiverClientGrpc.close (apache#10747) HDDS-15765. Fail fast when --node-id is omitted for OM compact/defrag on an HA cluster (apache#10699) HDDS-14989. Delay follower SCM DN server start until Ratis log catch-up. (apache#10617). HDDS-15456. Add SCM DB lookup and identify orphan(wrt SCM) and deleted-but-present containers (apache#10547) HDDS-15812. Rename test utils for naming convention (apache#10766) HDDS-15600. Fix ListObjects response for encoding-type and empty delimiter (apache#10586). HDDS-15656. Fix non-atomic data size accumulation in ContainerBalancer move callback (apache#10708) HDDS-15667. `containerbalancer status`: show stop reason and iteration details after stop (apache#10681) HDDS-15823. Fix flaky TestFailureHandlingByClient#testContainerExclusionWithClosedContainerException (apache#10721) HDDS-15837. Add health-aware container ID listing in SCM (apache#10736) HDDS-15851. Return 403 for expired or out-of-range pre-signed S3 URLs (apache#10749) HDDS-15836. Add Archiver.appendFile for incremental TAR writes (apache#10735) ... Conflicts: hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestSCMNodeManager.java hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestSCMNodeMetrics.java hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyInfo.java hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/RepeatedOmKeyInfo.java hadoop-ozone/dist/src/main/compose/upgrade/test.sh hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/upgrade/HddsUpgradeTestUtils.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/upgrade/TestDNDataDistributionFinalization.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/upgrade/TestHDDSUpgrade.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/upgrade/TestScmDataDistributionFinalization.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/upgrade/TestScmHAFinalization.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/service/TestBlockDeletionService.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java

* HDDS-14496-zdu: (56 commits) HDDS-15848. Add new upgrade action annotations for component versions (apache#10744) HDDS-15876. Remove ozone-csi (apache#10780) HDDS-14825. Add Grafana Dashboard and Metrics for ZDU (apache#10602) HDDS-15508. Add 2.2.0 to compatibility/upgrade test (apache#10782) HDDS-15867. [JDK25] Bump byte-buddy to 1.18.11 (apache#10776) HDDS-15809. Validate x-amz-copy-source-range on UploadPartCopy (apache#10711). HDDS-15678. OFS isDirectory/isFile should not trigger pipeline refresh or return block locations (apache#10741) HDDS-15171. Add available space check on follower during bootstrap. (apache#10185) HDDS-14043. Fix ls -e UnsupportedOperationException on ofs/o3fs (apache#10209) HDDS-15709. Replace random name suffix with uniqueObjectName (apache#10773) HDDS-15677. Update client side version checks to use `isSupportedBy`. (apache#10611) HDDS-15849. Force channel shutdown in XceiverClientGrpc.close (apache#10747) HDDS-15765. Fail fast when --node-id is omitted for OM compact/defrag on an HA cluster (apache#10699) HDDS-14989. Delay follower SCM DN server start until Ratis log catch-up. (apache#10617). HDDS-15456. Add SCM DB lookup and identify orphan(wrt SCM) and deleted-but-present containers (apache#10547) HDDS-15812. Rename test utils for naming convention (apache#10766) HDDS-15600. Fix ListObjects response for encoding-type and empty delimiter (apache#10586). HDDS-15656. Fix non-atomic data size accumulation in ContainerBalancer move callback (apache#10708) HDDS-15667. `containerbalancer status`: show stop reason and iteration details after stop (apache#10681) HDDS-15823. Fix flaky TestFailureHandlingByClient#testContainerExclusionWithClosedContainerException (apache#10721) ... Conflicts: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerPrepare.java

chihsuan added 2 commits July 9, 2026 23:03

HDDS-15765. Fail fast when --node-id is omitted for OM compact/defrag…

5f60e19

… on an HA cluster

HDDS-15765. Use OZONE_OM_SERVICE_IDS_KEY constant in compact/defrag t…

ccf5f81

…ests

adoroszlai requested a review from sarvekshayr July 10, 2026 11:06

adoroszlai added the command-line label Jul 10, 2026

rich7420 approved these changes Jul 11, 2026

View reviewed changes

sreejasahithi reviewed Jul 13, 2026

View reviewed changes

HDDS-15765. Document HA node ID requirement in command help

e6758ff

Copilot AI review requested due to automatic review settings July 13, 2026 12:43

Copilot AI reviewed Jul 13, 2026

jojochuang added the AI-gen label Jul 13, 2026

jojochuang self-requested a review July 13, 2026 16:34

adoroszlai requested a review from sreejasahithi July 13, 2026 21:05

sreejasahithi reviewed Jul 14, 2026

View reviewed changes

HDDS-15765. Update command help output in docs

4747b6a

Copilot AI review requested due to automatic review settings July 15, 2026 09:21

Copilot AI reviewed Jul 15, 2026

chihsuan requested a review from sreejasahithi July 15, 2026 09:32

sreejasahithi approved these changes Jul 15, 2026

View reviewed changes

sarvekshayr approved these changes Jul 15, 2026

View reviewed changes

sarvekshayr merged commit e0bdbc2 into apache:master Jul 15, 2026
46 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDDS-15765. Fail fast when --node-id is omitted for OM compact/defrag on an HA cluster#10699

HDDS-15765. Fail fast when --node-id is omitted for OM compact/defrag on an HA cluster#10699
sarvekshayr merged 4 commits into
apache:masterfrom
chihsuan:HDDS-15765

chihsuan commented Jul 9, 2026 •

edited

Loading

Uh oh!

rich7420 left a comment

Uh oh!

sreejasahithi left a comment •

edited

Loading

Uh oh!

adoroszlai commented Jul 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

chihsuan commented Jul 13, 2026

Uh oh!

sreejasahithi commented Jul 14, 2026

Uh oh!

sreejasahithi left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

chihsuan commented Jul 15, 2026

Uh oh!

sreejasahithi left a comment

Uh oh!

Uh oh!

sarvekshayr commented Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Conversation

chihsuan commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

rich7420 left a comment

Choose a reason for hiding this comment

Uh oh!

sreejasahithi left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adoroszlai commented Jul 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

chihsuan commented Jul 13, 2026

Uh oh!

sreejasahithi commented Jul 14, 2026

Uh oh!

sreejasahithi left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

chihsuan commented Jul 15, 2026

Uh oh!

sreejasahithi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sarvekshayr commented Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

chihsuan commented Jul 9, 2026 •

edited

Loading

sreejasahithi left a comment •

edited

Loading