ci/maintenance-unit-tests: stop+rm long-up named test-service containers

Tests that exercise module_postgres / module_redis / module_immich
(and friends) spawn long-running service containers — postgres,
postgres-immich, mysql, redis — and only call `remove` if the test
reaches that stage. An interrupted test (or one that only does
install) leaves them wedged on the runner; the diagnostic step on
this branch's first run showed postgres-immich and mysql with
'Up 7 days', and a `postgres` container in 'Created' state for 8h
because its 5432 publish was held by the older one.

Add a second cleanup block to both diagnostic steps, ahead of the
orphaned-Created sweep, that stop+rm's exactly these four names if
they have been *running for more than 2 hours*. The threshold is
the safety knob: native job timeout-minutes is 90, emulated 210
(3.5h), so 2h sits comfortably below "test still in flight" and
well above "leftover from a prior run". Anything younger is left
alone — the next slot's start will pick it up if needed.

Implementation:

  * Names are exact-match anchored ('^name$') so a future sibling
    like 'mysql-readonly' or 'postgres-staging' isn't caught.
  * Stopped / Created containers are deliberately skipped here —
    the orphaned-Created sweep below uses the bare-hash filter
    and handles them safely.
  * Uptime via 'docker inspect ... .State.StartedAt' → 'date -d'
    → epoch math. GNU date is standard on Linux runners.
  * 'docker stop --time 30' gives postgres / mysql a chance to
    flush the WAL before SIGKILL; '|| true' on each docker call
    so a transient hiccup doesn't fail the test slot.
  * Output is one line per name ('absent', 'recent <2h (skip)',
    'stopped (skip — orphan sweep handles below)', or
    'stop+rm: <name> (up Xh)') so the GHA log shows exactly which
    containers were touched and why.

Runs *before* the orphaned-Created sweep so a freshly-stopped
service isn't briefly seen as orphan in the same pass.

Same block in both gradle-native and gradle-emulated diagnostic
steps so the cleanup fires on both runner hosts (the arm64 host
is only seen by native; the amd64 host is seen by both).

Author: igorpecovnik

.github/workflows/maintenance-unit-tests.yml

@@ -242,6 +242,57 @@ jobs:
           docker system df || true
           echo "::endgroup::"
 
+          echo "::group::Cleanup: stop and remove named test-service containers (uptime > 2h)"
+          # Tests that exercise module_postgres / module_redis /
+          # module_immich (and friends) spawn long-running service
+          # containers and only tear them down if the test reaches
+          # its `remove` stage. A test that's interrupted, or one
+          # that only calls `install`, leaves these wedged on the
+          # runner; the diagnostic above showed postgres-immich and
+          # mysql 'Up 7 days' for that reason.
+          #
+          # Skip containers that have been up less than 2h — they
+          # could be a sibling matrix slot's test currently in
+          # flight (native timeout-minutes is 90, emulated 210, so
+          # 2h is comfortably below "real test still running" and
+          # well above "leftover from a prior run").
+          #
+          # Stopped / Created containers are left to the
+          # orphaned-Created sweep below (which uses the bare-hash
+          # filter to avoid the same false-positive concern).
+          #
+          # Exact-match anchored names (^name$) so a future sibling
+          # like 'mysql-readonly' or 'postgres-staging' isn't caught.
+          # Runs *before* the orphaned-Created sweep so a freshly
+          # stopped service isn't briefly seen as orphan in the same
+          # pass. 'docker stop --time 30' gives postgres / mysql a
+          # chance to flush before SIGKILL; '|| true' on each step
+          # so a transient hiccup doesn't fail the test slot.
+          now_epoch=$(date +%s)
+          threshold=$(( 2 * 3600 ))
+          for c in postgres-immich mysql redis postgres; do
+            if ! docker ps -a --filter "name=^${c}$" --format '{{.Names}}' | grep -q "^${c}$"; then
+              echo "  absent : $c"
+              continue
+            fi
+            running=$(docker inspect "$c" --format '{{.State.Running}}' 2>/dev/null || echo false)
+            if [[ "$running" != "true" ]]; then
+              echo "  stopped (skip — orphan sweep handles below): $c"
+              continue
+            fi
+            started=$(docker inspect "$c" --format '{{.State.StartedAt}}' 2>/dev/null)
+            started_epoch=$(date -d "$started" +%s 2>/dev/null || echo "$now_epoch")
+            age=$(( now_epoch - started_epoch ))
+            if (( age < threshold )); then
+              printf "  recent <2h (skip): %s (up %dm)\n" "$c" $(( age / 60 ))
+              continue
+            fi
+            docker stop --time 30 "$c" >/dev/null 2>&1 || true
+            docker rm           "$c" >/dev/null 2>&1 || true
+            printf "  stop+rm: %s (up %dh)\n" "$c" $(( age / 3600 ))
+          done
+          echo "::endgroup::"
+
           echo "::group::Cleanup: orphaned 'Created' containers (image ref is bare hash)"
           # Pattern: stopped-but-never-started containers whose Image
           # column is a 12-char hex ID (no repo:tag). These are
@@ -412,6 +463,57 @@ jobs:
           docker system df || true
           echo "::endgroup::"
 
+          echo "::group::Cleanup: stop and remove named test-service containers (uptime > 2h)"
+          # Tests that exercise module_postgres / module_redis /
+          # module_immich (and friends) spawn long-running service
+          # containers and only tear them down if the test reaches
+          # its `remove` stage. A test that's interrupted, or one
+          # that only calls `install`, leaves these wedged on the
+          # runner; the diagnostic above showed postgres-immich and
+          # mysql 'Up 7 days' for that reason.
+          #
+          # Skip containers that have been up less than 2h — they
+          # could be a sibling matrix slot's test currently in
+          # flight (native timeout-minutes is 90, emulated 210, so
+          # 2h is comfortably below "real test still running" and
+          # well above "leftover from a prior run").
+          #
+          # Stopped / Created containers are left to the
+          # orphaned-Created sweep below (which uses the bare-hash
+          # filter to avoid the same false-positive concern).
+          #
+          # Exact-match anchored names (^name$) so a future sibling
+          # like 'mysql-readonly' or 'postgres-staging' isn't caught.
+          # Runs *before* the orphaned-Created sweep so a freshly
+          # stopped service isn't briefly seen as orphan in the same
+          # pass. 'docker stop --time 30' gives postgres / mysql a
+          # chance to flush before SIGKILL; '|| true' on each step
+          # so a transient hiccup doesn't fail the test slot.
+          now_epoch=$(date +%s)
+          threshold=$(( 2 * 3600 ))
+          for c in postgres-immich mysql redis postgres; do
+            if ! docker ps -a --filter "name=^${c}$" --format '{{.Names}}' | grep -q "^${c}$"; then
+              echo "  absent : $c"
+              continue
+            fi
+            running=$(docker inspect "$c" --format '{{.State.Running}}' 2>/dev/null || echo false)
+            if [[ "$running" != "true" ]]; then
+              echo "  stopped (skip — orphan sweep handles below): $c"
+              continue
+            fi
+            started=$(docker inspect "$c" --format '{{.State.StartedAt}}' 2>/dev/null)
+            started_epoch=$(date -d "$started" +%s 2>/dev/null || echo "$now_epoch")
+            age=$(( now_epoch - started_epoch ))
+            if (( age < threshold )); then
+              printf "  recent <2h (skip): %s (up %dm)\n" "$c" $(( age / 60 ))
+              continue
+            fi
+            docker stop --time 30 "$c" >/dev/null 2>&1 || true
+            docker rm           "$c" >/dev/null 2>&1 || true
+            printf "  stop+rm: %s (up %dh)\n" "$c" $(( age / 3600 ))
+          done
+          echo "::endgroup::"
+
           echo "::group::Cleanup: orphaned 'Created' containers (image ref is bare hash)"
           # Pattern: stopped-but-never-started containers whose Image
           # column is a 12-char hex ID (no repo:tag). These are