Skip to content

HDDS-15648. Improve TestSnapshotDiffManager#testThreadPoolIsFull drain scenario#10632

Merged
smengcl merged 2 commits into
apache:masterfrom
cchung100m:hdds-15648
Jul 10, 2026
Merged

HDDS-15648. Improve TestSnapshotDiffManager#testThreadPoolIsFull drain scenario#10632
smengcl merged 2 commits into
apache:masterfrom
cchung100m:hdds-15648

Conversation

@cchung100m

@cchung100m cchung100m commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Improve TestSnapshotDiffManager#testThreadPoolIsFull drai n scenario by filling the pool before releasing workers

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15648

How was this patch tested?

Existing unit and integration tests

@cchung100m
HDDS-15648. Improve TestSnapshotDiffManager#testThreadPoolIsFull drai…
3b123bb 
…n scenario by filling the pool before releasing workers
@cchung100m
cchung100m marked this pull request as ready for review June 30, 2026 01:16
chungen0126
chungen0126 reviewed Jun 30, 2026
View reviewed changes
Comment thread ...ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestSnapshotDiffManager.java
Comment on lines +1215 to +1219
if (drainBetweenBatches && latchOpened) {
while (totalSubmitted - completedJobs.get() >= fullThreadPoolSize) {
Thread.sleep(10);
}
}

@chungen0126 chungen0126 Jun 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @cchung100m for working on this. We should avoid using a while loop with Thread.sleep for busy-waiting here. Let's refactor this using JavaUtils#attempt to make the waiting logic cleaner and safer.

@cchung100m cchung100m Jul 1, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @chungen0126
Thanks for review comments. I have updated the part you mentioned.

@cchung100m
cchung100m marked this pull request as draft June 30, 2026 05:53
@cchung100m
cchung100m force-pushed the hdds-15648 branch 7 times, most recently from c1ac22a to c210dc6 Compare June 30, 2026 16:12
@cchung100m
cchung100m marked this pull request as ready for review July 1, 2026 02:39
@adoroszlai adoroszlai changed the title HDDS-15648. Improve TestSnapshotDiffManager#testThreadPoolIsFull drai n scenario by filling the pool before releasing workers HDDS-15648. Improve TestSnapshotDiffManager#testThreadPoolIsFull drain scenario Jul 1, 2026
@adoroszlai adoroszlai added the test label Jul 4, 2026
@cchung100m
cchung100m requested a review from chungen0126 July 6, 2026 07:42
SaketaChalamchala
SaketaChalamchala approved these changes Jul 7, 2026
View reviewed changes

@SaketaChalamchala SaketaChalamchala left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the patch @cchung100m. LGTM.

smengcl
smengcl approved these changes Jul 10, 2026
View reviewed changes

@smengcl smengcl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. Pending CI

@smengcl
smengcl merged commit 756f714 into apache:master Jul 10, 2026
72 of 76 checks passed
@smengcl

smengcl commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Thanks @cchung100m for the patch. Thanks @chungen0126 @SaketaChalamchala for reviewing this.

@cchung100m

cchung100m commented Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

Thanks to @chungen0126 @SaketaChalamchala @smengcl @adoroszlai @jojochuang

@cchung100m
cchung100m deleted the hdds-15648 branch July 10, 2026 03:12
errose28 added a commit that referenced this pull request Jul 14, 2026
@errose28
Merge branch 'master' into HDDS-14496-zdu
b556be1 
* master: (416 commits)
  HDDS-15808. Replace json-simple with Jackson in httpfsgateway (#10710)
  HDDS-15753. Rename TestClock to MockClock (#10712)
  HDDS-14759. Local Empty file is written and retained even when key write fails (#10707)
  HDDS-15647. Bump Node.js to 24.17.0 (#10590)
  HDDS-15783. Handle empty lists in ScmListCodec deserialization (#10690)
  HDDS-15768. Bracket IPv6 literals in Ratis peer addresses (#10686)
  HDDS-15679. [Recon] Fix DN dropdown mismatch with pending deletion API (#10615).
  HDDS-15692. Enable GitHub Copilot code review when a pull request is opened (#10705)
  HDDS-15739. Fix ListObjectsV2 continuation-token: wrong XML element name and empty-token handling (#10668)
  HDDS-15648. Improve TestSnapshotDiffManager#testThreadPoolIsFull drain scenario (#10632)
  HDDS-15803. Fix parameter number warning in OMKeyRequest.allocateBlock. (#10701)
  HDDS-15480. Potential for NPE / infinite loop in StreamBlockReader (#10431)
  HDDS-15523. Speed up TestOMDbCheckpointServletInodeBasedXfer (#10680)
  HDDS-15788. Fix SST filtering BOOTSTRAP_LOCK contention introduced by RocksDB 10 upgrade (#10692)
  HDDS-15752. Rename base tests and helpers for naming convention in OM (#10672)
  HDDS-15754. Rename TestableCluster to MockCluster (#10691)
  HDDS-15755. Bump develocity-maven-extension to 2.5.0 (#10674)
  HDDS-14956. Add S3 HAProxy to ozonesecure-ha/test-haproxy-s3g.sh (#10685)
  HDDS-15762. Remove mini-chaos-tests (#10682)
  HDDS-14577. Handle missing metadata dir when updating container state (#10565).
  ...

Conflicts:
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneManagerVersion.java
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeStateMachine.java
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/states/endpoint/TestHeartbeatEndpointTask.java
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeManager.java
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeStateManager.java
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/SafeModeRuleFactory.java
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/upgrade/FinalizationStateManagerImpl.java
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestNodeStateManager.java
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestPendingContainerTracker.java
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/MockPipelineManager.java
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/server/TestSCMClientProtocolServer.java
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyInfo.java
hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/protocolPB/TestS3GrpcOmTransport.java
hadoop-ozone/dist/src/main/compose/testlib.sh
hadoop-ozone/dist/src/main/compose/upgrade/compose/ha/docker-compose.yaml
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMRatisSnapshots.java
hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerRequestHandler.java
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java
errose28 added a commit to errose28/ozone that referenced this pull request Jul 14, 2026
@errose28
Merge branch 'HDDS-14496-zdu' into om-version-client-cleanup
3c70a64 
* HDDS-14496-zdu: (417 commits)
  HDDS-15808. Replace json-simple with Jackson in httpfsgateway (apache#10710)
  HDDS-15753. Rename TestClock to MockClock (apache#10712)
  HDDS-14759. Local Empty file is written and retained even when key write fails (apache#10707)
  HDDS-15647. Bump Node.js to 24.17.0 (apache#10590)
  HDDS-15783. Handle empty lists in ScmListCodec deserialization (apache#10690)
  HDDS-15768. Bracket IPv6 literals in Ratis peer addresses (apache#10686)
  HDDS-15679. [Recon] Fix DN dropdown mismatch with pending deletion API (apache#10615).
  HDDS-15692. Enable GitHub Copilot code review when a pull request is opened (apache#10705)
  HDDS-15739. Fix ListObjectsV2 continuation-token: wrong XML element name and empty-token handling (apache#10668)
  HDDS-15648. Improve TestSnapshotDiffManager#testThreadPoolIsFull drain scenario (apache#10632)
  HDDS-15803. Fix parameter number warning in OMKeyRequest.allocateBlock. (apache#10701)
  HDDS-15480. Potential for NPE / infinite loop in StreamBlockReader (apache#10431)
  HDDS-15523. Speed up TestOMDbCheckpointServletInodeBasedXfer (apache#10680)
  HDDS-15788. Fix SST filtering BOOTSTRAP_LOCK contention introduced by RocksDB 10 upgrade (apache#10692)
  HDDS-15752. Rename base tests and helpers for naming convention in OM (apache#10672)
  HDDS-15754. Rename TestableCluster to MockCluster (apache#10691)
  HDDS-15755. Bump develocity-maven-extension to 2.5.0 (apache#10674)
  HDDS-14956. Add S3 HAProxy to ozonesecure-ha/test-haproxy-s3g.sh (apache#10685)
  HDDS-15762. Remove mini-chaos-tests (apache#10682)
  HDDS-14577. Handle missing metadata dir when updating container state (apache#10565).
  ...

Conflicts:
hadoop-ozone/client/src/test/java/org/apache/hadoop/ozone/client/rpc/TestRpcClient.java
errose28 added a commit to errose28/ozone that referenced this pull request Jul 14, 2026
@errose28
Merge branch 'HDDS-14496-zdu' into worktree-version-metrics
8cdf925 
* HDDS-14496-zdu: (417 commits)
  HDDS-15808. Replace json-simple with Jackson in httpfsgateway (apache#10710)
  HDDS-15753. Rename TestClock to MockClock (apache#10712)
  HDDS-14759. Local Empty file is written and retained even when key write fails (apache#10707)
  HDDS-15647. Bump Node.js to 24.17.0 (apache#10590)
  HDDS-15783. Handle empty lists in ScmListCodec deserialization (apache#10690)
  HDDS-15768. Bracket IPv6 literals in Ratis peer addresses (apache#10686)
  HDDS-15679. [Recon] Fix DN dropdown mismatch with pending deletion API (apache#10615).
  HDDS-15692. Enable GitHub Copilot code review when a pull request is opened (apache#10705)
  HDDS-15739. Fix ListObjectsV2 continuation-token: wrong XML element name and empty-token handling (apache#10668)
  HDDS-15648. Improve TestSnapshotDiffManager#testThreadPoolIsFull drain scenario (apache#10632)
  HDDS-15803. Fix parameter number warning in OMKeyRequest.allocateBlock. (apache#10701)
  HDDS-15480. Potential for NPE / infinite loop in StreamBlockReader (apache#10431)
  HDDS-15523. Speed up TestOMDbCheckpointServletInodeBasedXfer (apache#10680)
  HDDS-15788. Fix SST filtering BOOTSTRAP_LOCK contention introduced by RocksDB 10 upgrade (apache#10692)
  HDDS-15752. Rename base tests and helpers for naming convention in OM (apache#10672)
  HDDS-15754. Rename TestableCluster to MockCluster (apache#10691)
  HDDS-15755. Bump develocity-maven-extension to 2.5.0 (apache#10674)
  HDDS-14956. Add S3 HAProxy to ozonesecure-ha/test-haproxy-s3g.sh (apache#10685)
  HDDS-15762. Remove mini-chaos-tests (apache#10682)
  HDDS-14577. Handle missing metadata dir when updating container state (apache#10565).
  ...

Conflicts:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/metrics/SCMMetrics.java
errose28 added a commit to errose28/ozone that referenced this pull request Jul 14, 2026
@errose28
Merge branch 'HDDS-14496-zdu' into HDDS-14580-remove-prepare
583d9ac 
* HDDS-14496-zdu: (416 commits)
  HDDS-15808. Replace json-simple with Jackson in httpfsgateway (apache#10710)
  HDDS-15753. Rename TestClock to MockClock (apache#10712)
  HDDS-14759. Local Empty file is written and retained even when key write fails (apache#10707)
  HDDS-15647. Bump Node.js to 24.17.0 (apache#10590)
  HDDS-15783. Handle empty lists in ScmListCodec deserialization (apache#10690)
  HDDS-15768. Bracket IPv6 literals in Ratis peer addresses (apache#10686)
  HDDS-15679. [Recon] Fix DN dropdown mismatch with pending deletion API (apache#10615).
  HDDS-15692. Enable GitHub Copilot code review when a pull request is opened (apache#10705)
  HDDS-15739. Fix ListObjectsV2 continuation-token: wrong XML element name and empty-token handling (apache#10668)
  HDDS-15648. Improve TestSnapshotDiffManager#testThreadPoolIsFull drain scenario (apache#10632)
  HDDS-15803. Fix parameter number warning in OMKeyRequest.allocateBlock. (apache#10701)
  HDDS-15480. Potential for NPE / infinite loop in StreamBlockReader (apache#10431)
  HDDS-15523. Speed up TestOMDbCheckpointServletInodeBasedXfer (apache#10680)
  HDDS-15788. Fix SST filtering BOOTSTRAP_LOCK contention introduced by RocksDB 10 upgrade (apache#10692)
  HDDS-15752. Rename base tests and helpers for naming convention in OM (apache#10672)
  HDDS-15754. Rename TestableCluster to MockCluster (apache#10691)
  HDDS-15755. Bump develocity-maven-extension to 2.5.0 (apache#10674)
  HDDS-14956. Add S3 HAProxy to ozonesecure-ha/test-haproxy-s3g.sh (apache#10685)
  HDDS-15762. Remove mini-chaos-tests (apache#10682)
  HDDS-14577. Handle missing metadata dir when updating container state (apache#10565).
  ...

Conflicts:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerRequestHandler.java
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/upgrade/TestOMCancelPrepareRequest.java
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@SaketaChalamchala SaketaChalamchala SaketaChalamchala approved these changes
@smengcl smengcl smengcl approved these changes
@chungen0126 chungen0126 Awaiting requested review from chungen0126

Assignees

No one assigned

Labels

test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@cchung100m @smengcl @SaketaChalamchala @chungen0126 @adoroszlai