HDDS-15667. `containerbalancer status`: show stop reason and iteration details after stop by sreejasahithi · Pull Request #10681 · apache/ozone

sreejasahithi · 2026-07-07T04:55:20Z

What changes were proposed in this pull request?

Currently, when Container Balancer is stopped, ozone admin containerbalancer status only reports: "ContainerBalancer is Not Running."

It would be helpful if after stop, status shows the last known run state:

Stop reason (also in non-verbose mode)
Start/stop timestamps, balancing duration, and configuration used ( with --verbose flag)
Last iteration details ( with --verbose flag)
Iteration history details (with --verbose -H flag)

What is the link to the Apache JIRA

HDDS-15667

How was this patch tested?

Tested manually in docker cluster and added test cases.
Sample runs:

bash-5.1$ ozone admin containerbalancer status --verbose --history
ContainerBalancer is Not Running.
Stop reason: INITIALIZATION_FAILED
Message: Failed to initialize a container balancer iteration. Details:  iteration number 1, Did not find any unbalanced Datanodes.
Started at: 2026-07-14 05:09:39
Stopped at: 2026-07-14 05:09:39
Balancing duration: 0s

Container Balancer Configuration values:
Key                                                Value
Threshold                                          10
Max Datanodes to Involve per Iteration(percent)    20
Max Size to Move per Iteration                     500GB
Max Size Entering Target per Iteration             26GB
Max Size Leaving Source per Iteration              26GB
Number of Iterations                               10
Time Limit for Single Container's Movement         65min
Time Limit for Single Container's Replication      50min
Interval between each Iteration                    0min
Whether to Enable Network Topology                 false
Whether to Trigger Refresh Datanode Usage Info     false
Container IDs to Include in Balancing              None
Container IDs to Exclude from Balancing            None
Datanodes Specified to be Balanced                 None
Datanodes Excluded from Balancing                  None

Last iteration info:
-

Completed iteration history:
-

bash-5.1$ ozone admin containerbalancer status --verbose -H
ContainerBalancer is Not Running.
Stop reason: USER_REQUESTED
Message: Stopped by user request.
Started at: 2026-07-13 07:24:20
Stopped at: 2026-07-13 07:26:14
Balancing duration: 1m 54s

Container Balancer Configuration values:
Key                                                Value
Threshold                                          0.1
Max Datanodes to Involve per Iteration(percent)    100
Max Size to Move per Iteration                     500GB
Max Size Entering Target per Iteration             26GB
Max Size Leaving Source per Iteration              26GB
Number of Iterations                               3
Time Limit for Single Container's Movement         65min
Time Limit for Single Container's Replication      50min
Interval between each Iteration                    0min
Whether to Enable Network Topology                 false
Whether to Trigger Refresh Datanode Usage Info     false
Container IDs to Include in Balancing              None
Container IDs to Exclude from Balancing            None
Datanodes Specified to be Balanced                 None
Datanodes Excluded from Balancing                  None

Last iteration info:
Key                                                Value
Iteration number                                   1
Iteration duration                                 1m 54s
Iteration result                                   ITERATION_COMPLETED
Size scheduled to move                             150 MB
Moved data size                                    50 MB
Scheduled to move containers                       2
Already moved containers                           1
Failed to move containers                          0
Failed to move containers by timeout               0
Entered data to nodes                              
79c3cce6-4218-4966-90db-e964c2426473 <- 100 MB
5f5d448c-cfd6-4b6f-9c5e-152d66bbce48 <- 50 MB
Exited data from nodes                             
790eafdd-7d7f-4c5f-948d-684f43c67437 -> 100 MB
248e72f4-afe2-4070-9e6c-4555c39f86e2 -> 50 MB

Completed iteration history:
-

bash-5.1$ ozone admin containerbalancer start -t 0.1 -d 100 -i 3
Container Balancer started successfully.
bash-5.1$ ozone admin containerbalancer status --verbose -H
ContainerBalancer is Not Running.
Stop reason: CAN_NOT_BALANCE_ANY_MORE
Message: No more eligible container moves were found.
Started at: 2026-07-08 12:45:25
Stopped at: 2026-07-08 12:46:20
Balancing duration: 55s

Container Balancer Configuration values:
Key                                                Value
Threshold                                          0.1
Max Datanodes to Involve per Iteration(percent)    100
Max Size to Move per Iteration                     500GB
Max Size Entering Target per Iteration             26GB
Max Size Leaving Source per Iteration              26GB
Number of Iterations                               3
Time Limit for Single Container's Movement         65min
Time Limit for Single Container's Replication      50min
Interval between each Iteration                    0min
Whether to Enable Network Topology                 false
Whether to Trigger Refresh Datanode Usage Info     false
Container IDs to Include in Balancing              None
Container IDs to Exclude from Balancing            None
Datanodes Specified to be Balanced                 None
Datanodes Excluded from Balancing                  None

Last iteration info:
Key                                                Value
Iteration number                                   2
Iteration duration                                 0s
Iteration result                                   CAN_NOT_BALANCE_ANY_MORE
Size scheduled to move                             0 B
Moved data size                                    0 B
Scheduled to move containers                       0
Already moved containers                           0
Failed to move containers                          0
Failed to move containers by timeout               0
Entered data to nodes
 -
Exited data from nodes
 -

Completed iteration history:
Key                                                Value
Iteration number                                   1
Iteration duration                                 30s
Iteration result                                   ITERATION_COMPLETED
Size scheduled to move                             300 MB
Moved data size                                    300 MB
Scheduled to move containers                       3
Already moved containers                           3
Failed to move containers                          0
Failed to move containers by timeout               0
Entered data to nodes
0c916231-0bc0-4303-910e-a5e3eedbf0c0 <- 100 MB
39b69c76-67aa-4a04-8675-5ad143a63ad7 <- 100 MB
30b0de7b-a6e5-40c3-ab6b-40b35adf85f7 <- 100 MB
Exited data from nodes
1eccc612-654a-4aa2-919f-2b916aa7f43b -> 100 MB
850f87cb-19ab-4188-bfe3-ff83672f8341 -> 100 MB
a6f2b660-ddfb-4738-817e-9a477428365e -> 100 MB

bash-5.1$ ozone admin containerbalancer status
ContainerBalancer is Not Running.
Stop reason: CAN_NOT_BALANCE_ANY_MORE
Message: No more eligible container moves were found.

bash-5.1$ ozone admin containerbalancer start -t 0.1 -d 100 -i 2
Container Balancer started successfully.
bash-5.1$ ozone admin containerbalancer status --verbose -H
ContainerBalancer is Not Running.
Stop reason: COMPLETED_ALL_ITERATIONS
Message: Completed all configured number of iterations.
Started at: 2026-07-08 12:55:17
Stopped at: 2026-07-08 12:56:59
Balancing duration: 1m 42s

Container Balancer Configuration values:
Key                                                Value
Threshold                                          0.1
Max Datanodes to Involve per Iteration(percent)    100
Max Size to Move per Iteration                     500GB
Max Size Entering Target per Iteration             26GB
Max Size Leaving Source per Iteration              26GB
Number of Iterations                               2
Time Limit for Single Container's Movement         65min
Time Limit for Single Container's Replication      50min
Interval between each Iteration                    0min
Whether to Enable Network Topology                 false
Whether to Trigger Refresh Datanode Usage Info     false
Container IDs to Include in Balancing              None
Container IDs to Exclude from Balancing            None
Datanodes Specified to be Balanced                 None
Datanodes Excluded from Balancing                  None

Last iteration info:
Key                                                Value
Iteration number                                   2
Iteration duration                                 36s
Iteration result                                   ITERATION_COMPLETED
Size scheduled to move                             100 MB
Moved data size                                    0 B
Scheduled to move containers                       3
Already moved containers                           3
Failed to move containers                          0
Failed to move containers by timeout               0
Entered data to nodes
1eccc612-654a-4aa2-919f-2b916aa7f43b <- 40 MB
850f87cb-19ab-4188-bfe3-ff83672f8341 <- 40 MB
a6f2b660-ddfb-4738-817e-9a477428365e <- 20 MB
Exited data from nodes
0c916231-0bc0-4303-910e-a5e3eedbf0c0 -> 40 MB
39b69c76-67aa-4a04-8675-5ad143a63ad7 -> 40 MB
30b0de7b-a6e5-40c3-ab6b-40b35adf85f7 -> 20 MB

Completed iteration history:
Key                                                Value
Iteration number                                   1
Iteration duration                                 41s
Iteration result                                   ITERATION_COMPLETED
Size scheduled to move                             160 MB
Moved data size                                    160 MB
Scheduled to move containers                       3
Already moved containers                           3
Failed to move containers                          0
Failed to move containers by timeout               0
Entered data to nodes
0c916231-0bc0-4303-910e-a5e3eedbf0c0 <- 80 MB
39b69c76-67aa-4a04-8675-5ad143a63ad7 <- 20 MB
30b0de7b-a6e5-40c3-ab6b-40b35adf85f7 <- 60 MB
Exited data from nodes
1eccc612-654a-4aa2-919f-2b916aa7f43b -> 20 MB
850f87cb-19ab-4188-bfe3-ff83672f8341 -> 80 MB
a6f2b660-ddfb-4738-817e-9a477428365e -> 60 MB

Green CI : https://github.com/sreejasahithi/ozone/actions/runs/28775666320

…and iteration details (of last and history) after stop

ArafatKhan2198

Thanks for the patch @sreejasahithi
Some comments for you

rakeshadr

Thanks @sreejasahithi for working on this. Added a few comments, please go thru it.

…ason and made minor updates

…mber

ashishkumar50

@sreejasahithi Thanks for working on this, Please find the comments inline.

…D as the stop reason in finally block

sreejasahithi · 2026-07-13T13:46:44Z

@ashishkumar50 could you please help re run the checks as the latest commit got green ci in my fork
https://github.com/sreejasahithi/ozone/actions/runs/29241610279

ashishkumar50

LGTM

rakeshadr

+1 LGTM.

sarvekshayr · 2026-07-15T04:09:12Z

Thanks @sreejasahithi for the patch, @ArafatKhan2198, @ashishkumar50 and @rakeshadr for the review.

* master: (53 commits) HDDS-15876. Remove ozone-csi (apache#10780) HDDS-15508. Add 2.2.0 to compatibility/upgrade test (apache#10782) HDDS-15867. [JDK25] Bump byte-buddy to 1.18.11 (apache#10776) HDDS-15809. Validate x-amz-copy-source-range on UploadPartCopy (apache#10711). HDDS-15678. OFS isDirectory/isFile should not trigger pipeline refresh or return block locations (apache#10741) HDDS-15171. Add available space check on follower during bootstrap. (apache#10185) HDDS-14043. Fix ls -e UnsupportedOperationException on ofs/o3fs (apache#10209) HDDS-15709. Replace random name suffix with uniqueObjectName (apache#10773) HDDS-15849. Force channel shutdown in XceiverClientGrpc.close (apache#10747) HDDS-15765. Fail fast when --node-id is omitted for OM compact/defrag on an HA cluster (apache#10699) HDDS-14989. Delay follower SCM DN server start until Ratis log catch-up. (apache#10617). HDDS-15456. Add SCM DB lookup and identify orphan(wrt SCM) and deleted-but-present containers (apache#10547) HDDS-15812. Rename test utils for naming convention (apache#10766) HDDS-15600. Fix ListObjects response for encoding-type and empty delimiter (apache#10586). HDDS-15656. Fix non-atomic data size accumulation in ContainerBalancer move callback (apache#10708) HDDS-15667. `containerbalancer status`: show stop reason and iteration details after stop (apache#10681) HDDS-15823. Fix flaky TestFailureHandlingByClient#testContainerExclusionWithClosedContainerException (apache#10721) HDDS-15837. Add health-aware container ID listing in SCM (apache#10736) HDDS-15851. Return 403 for expired or out-of-range pre-signed S3 URLs (apache#10749) HDDS-15836. Add Archiver.appendFile for incremental TAR writes (apache#10735) ... Conflicts: hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestSCMNodeManager.java hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestSCMNodeMetrics.java hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyInfo.java hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/RepeatedOmKeyInfo.java hadoop-ozone/dist/src/main/compose/upgrade/test.sh hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/upgrade/HddsUpgradeTestUtils.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/upgrade/TestDNDataDistributionFinalization.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/upgrade/TestHDDSUpgrade.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/upgrade/TestScmDataDistributionFinalization.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/upgrade/TestScmHAFinalization.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/service/TestBlockDeletionService.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java

* HDDS-14496-zdu: (56 commits) HDDS-15848. Add new upgrade action annotations for component versions (apache#10744) HDDS-15876. Remove ozone-csi (apache#10780) HDDS-14825. Add Grafana Dashboard and Metrics for ZDU (apache#10602) HDDS-15508. Add 2.2.0 to compatibility/upgrade test (apache#10782) HDDS-15867. [JDK25] Bump byte-buddy to 1.18.11 (apache#10776) HDDS-15809. Validate x-amz-copy-source-range on UploadPartCopy (apache#10711). HDDS-15678. OFS isDirectory/isFile should not trigger pipeline refresh or return block locations (apache#10741) HDDS-15171. Add available space check on follower during bootstrap. (apache#10185) HDDS-14043. Fix ls -e UnsupportedOperationException on ofs/o3fs (apache#10209) HDDS-15709. Replace random name suffix with uniqueObjectName (apache#10773) HDDS-15677. Update client side version checks to use `isSupportedBy`. (apache#10611) HDDS-15849. Force channel shutdown in XceiverClientGrpc.close (apache#10747) HDDS-15765. Fail fast when --node-id is omitted for OM compact/defrag on an HA cluster (apache#10699) HDDS-14989. Delay follower SCM DN server start until Ratis log catch-up. (apache#10617). HDDS-15456. Add SCM DB lookup and identify orphan(wrt SCM) and deleted-but-present containers (apache#10547) HDDS-15812. Rename test utils for naming convention (apache#10766) HDDS-15600. Fix ListObjects response for encoding-type and empty delimiter (apache#10586). HDDS-15656. Fix non-atomic data size accumulation in ContainerBalancer move callback (apache#10708) HDDS-15667. `containerbalancer status`: show stop reason and iteration details after stop (apache#10681) HDDS-15823. Fix flaky TestFailureHandlingByClient#testContainerExclusionWithClosedContainerException (apache#10721) ... Conflicts: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerPrepare.java

HDDS-15667. Containerbalancer status command should show stop reason …

067136c

…and iteration details (of last and history) after stop

adoroszlai changed the title ~~HDDS-15667. Containerbalancer status command should show stop reason and iteration details after stop~~ HDDS-15667. containerbalancer status: show stop reason and iteration details after stop Jul 7, 2026

ArafatKhan2198 reviewed Jul 8, 2026

View reviewed changes

rakeshadr reviewed Jul 8, 2026

View reviewed changes

Added enum for balancer stop reason and split stopMessage from stopRe…

c778e3a

…ason and made minor updates

sreejasahithi requested review from ArafatKhan2198 and rakeshadr July 8, 2026 15:01

Fixed initialization failure stop message to use 1 based iteration nu…

9fae4b4

…mber

ashishkumar50 reviewed Jul 13, 2026

View reviewed changes

Made recordStopReason synchronized and used UNKNOWN instead of STOPPE…

8d3700a

…D as the stop reason in finally block

sreejasahithi marked this pull request as ready for review July 13, 2026 10:08

sreejasahithi requested a review from ashishkumar50 July 13, 2026 10:55

sreejasahithi added 2 commits July 13, 2026 19:23

Merge branch 'master' of github.com:apache/ozone into HDDS-15667

f03fb7e

TestableCluster renamed to MockCluster

504afd1

rakeshadr reviewed Jul 14, 2026

View reviewed changes

Addressed review comments

b572d28

sreejasahithi requested a review from rakeshadr July 14, 2026 05:30

ArafatKhan2198 approved these changes Jul 14, 2026

View reviewed changes

ashishkumar50 approved these changes Jul 14, 2026

View reviewed changes

rakeshadr approved these changes Jul 14, 2026

View reviewed changes

sarvekshayr merged commit e31f36d into apache:master Jul 15, 2026
88 of 90 checks passed

Uh oh!

Conversation

sreejasahithi commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

ArafatKhan2198 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rakeshadr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ashishkumar50 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sreejasahithi commented Jul 13, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ashishkumar50 left a comment

Choose a reason for hiding this comment

Uh oh!

rakeshadr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sarvekshayr commented Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sreejasahithi commented Jul 7, 2026 •

edited

Loading