pcp2openmetrics: add live timestamp mode#2637
Conversation
📝 WalkthroughSummary by CodeRabbit
Walkthrough
ChangesLive timestamp mode
Sequence Diagram(s)sequenceDiagram
participant User
participant Parser as pcp2openmetrics.py option parser
participant Writer as PCP2OPENMETRICS.write_openmetrics()
participant Output as OpenMetrics output
User->>Parser: run with -l / --times
Parser->>Writer: set self.time_stamp = 1
Writer->>Output: emit sample lines with timestamps
Poem
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/pcp2openmetrics/pcp2openmetrics.1`:
- Around line 421-425: The SYNOPSIS short-option cluster is missing the newly
documented lowercase l option, so update the synopsis in pcp2openmetrics.1 to
match the documented command-line options. Keep the short-option list consistent
with the `-l, --time-stamp` entry by adding `l` in the compact options section
so users can discover it from the top-level usage summary.
In `@src/pcp2openmetrics/pcp2openmetrics.py`:
- Around line 142-146: The counter timestamp cache in pcp2openmetrics.py can
grow without bound because counter_timestamps only ever adds new (metric_name,
instance_id) keys. Update the live-mode counter handling paths around the
related metric processing code (including the sections near the referenced
counter-timestamp logic) to evict stale entries when instances disappear or are
no longer observed. Use the existing counter_timestamps structure as the cleanup
point so long-running exporter stability is preserved.
- Around line 58-59: The self.keys list in pcp2openmetrics.py has a missing
separator between the time_stamp and count_scale entries, causing those config
directives to merge into one invalid key. Update the list in the initialization
block where self.keys is defined so time_stamp and count_scale are separate
elements, preserving recognition of both directives.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Repository UI (inherited), Organization UI (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: c37c1883-be10-4564-8af4-8c0dd5fd28c9
⛔ Files ignored due to path filters (1)
qa/1131.outis excluded by!**/*.out
📒 Files selected for processing (3)
qa/1131src/pcp2openmetrics/pcp2openmetrics.1src/pcp2openmetrics/pcp2openmetrics.py
| # Counter metric timestamp tracking | ||
| # key - (metric_name, instance_id) | ||
| # value - timestamp when first seen | ||
| self.counter_timestamps = {} | ||
|
|
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟠 Major | 🏗️ Heavy lift
Prevent unbounded growth of counter_timestamps in long-running live mode.
self.counter_timestamps only adds keys and never removes stale (metric, inst) entries. With high instance churn, this can grow indefinitely and degrade long-lived exporter stability.
Also applies to: 509-513, 520-523
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/pcp2openmetrics/pcp2openmetrics.py` around lines 142 - 146, The counter
timestamp cache in pcp2openmetrics.py can grow without bound because
counter_timestamps only ever adds new (metric_name, instance_id) keys. Update
the live-mode counter handling paths around the related metric processing code
(including the sections near the referenced counter-timestamp logic) to evict
stale entries when instances disappear or are no longer observed. Use the
existing counter_timestamps structure as the cleanup point so long-running
exporter stability is preserved.
lmchilton
force-pushed
the
pcp2openmetrics-timestamps
branch
from
371f9c2 to
bcc4cc6
Compare
| if self.time_stamp and openmetrics_type(desc) == "counter": | ||
| counter_key = (metric, inst) | ||
| creation_ts = self.counter_timestamps.get(counter_key, ts) | ||
| body += '%s%s %s\n' % (openmetrics_name_ts, openmetrics_labels(inst, name, desc, labels), creation_ts) |
There was a problem hiding this comment.
I think its incorrect to expose timestamps for counter metrics only here - we should treat live mode the same as archive (a few lines above) in this regard, and export the timestamp for the sample (as per the v1 OM spec, and as we always have).
The "_created" lines are indeed intended for counters only, re-reading the spec(s). They could be applicable to both archive and live PCP values, however, looking at our code - and in particular the fetchgroup code behind the python API - these are not going to be handled correctly (there is no knowledge of "mark" records in src/libpcp/src/fetchgroup.c which is the code being used by pmconfig.py and in turn by pcp2openmetrics.py). We should ignore the _created timestamp, its not ever going to be correctly handled here.
Finally, the PMWEBAPI(3) uses the parameter name "times" to enable these timestamps - could we mirror that here? i.e. instead of --time_stamp lets use --times? (and this should remain the default in archive mode).
There was a problem hiding this comment.
@natoscott thank you for the clarification! I updated everything
update pcp2openmetrics tool to allow for live timestamp mode option. This update follows openmetric / prometheus exposing timestamp guidelines. Updated man page & QA performancecopilot#1131 Resolves upstream RFE issue [performancecopilot#2538]
lmchilton
force-pushed
the
pcp2openmetrics-timestamps
branch
from
bcc4cc6 to
3e0d882
Compare
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@qa/1131`:
- Around line 107-108: The new QA case uses pcp2openmetrics with only -s2, which
still falls back to the tool’s default 10-second interval and slows the test.
Update the command in this test case to pass an explicit short time interval
with pcp2openmetrics so the assertion remains fast while preserving coverage;
locate the change in the qa/1131 test around the pcp2openmetrics invocation.
In `@src/pcp2openmetrics/pcp2openmetrics.py`:
- Line 204: The `--times` option help text in `pmSetLongOption` is misleading
because the implementation in `pcp2openmetrics.py` only appends a sample
timestamp to emitted series rather than adding a new metric. Update the
`--times` description to reflect the actual behavior, and keep the wording
aligned with the logic used later when timestamps are emitted so users
understand what the flag does.
- Line 58: The config option handling in the pcp2openmetrics parser is wired to
the wrong runtime field, so `times=yes` never affects output. Update the option
mapping in the config parsing logic around the `times` entry so it sets
`self.time_stamp` (or rename the runtime flag consistently), and verify the
output gating that reads `self.time_stamp` now reflects the
`pcp2openmetrics.conf` setting.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Repository UI (inherited), Organization UI (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: b8a86648-056d-4fd0-9d75-45cbdadeebdb
⛔ Files ignored due to path filters (1)
qa/1131.outis excluded by!**/*.out
📒 Files selected for processing (3)
qa/1131src/pcp2openmetrics/pcp2openmetrics.1src/pcp2openmetrics/pcp2openmetrics.py
| echo "--- pcp2openmetrics -s2 --times hinv.ncpu" | ||
| pcp2openmetrics -s2 --times hinv.ncpu | _filter_pcp2openmetrics |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟡 Minor | ⚡ Quick win
Avoid the default 10-second sleep in this new QA case.
-s2 here inherits the tool's default interval, so this single assertion adds about 10 seconds to the test run. Passing an explicit short -t keeps the regression fast without weakening coverage.
Minimal fix
-echo "--- pcp2openmetrics -s2 --times hinv.ncpu"
-pcp2openmetrics -s2 --times hinv.ncpu | _filter_pcp2openmetrics
+echo "--- pcp2openmetrics -s2 -t1 --times hinv.ncpu"
+pcp2openmetrics -s2 -t1 --times hinv.ncpu | _filter_pcp2openmetrics📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| echo "--- pcp2openmetrics -s2 --times hinv.ncpu" | |
| pcp2openmetrics -s2 --times hinv.ncpu | _filter_pcp2openmetrics | |
| echo "--- pcp2openmetrics -s2 -t1 --times hinv.ncpu" | |
| pcp2openmetrics -s2 -t1 --times hinv.ncpu | _filter_pcp2openmetrics |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@qa/1131` around lines 107 - 108, The new QA case uses pcp2openmetrics with
only -s2, which still falls back to the tool’s default 10-second interval and
slows the test. Update the command in this test case to pass an explicit short
time interval with pcp2openmetrics so the assertion remains fast while
preserving coverage; locate the change in the qa/1131 test around the
pcp2openmetrics invocation.
| self.keys = ('source', 'output', 'derived', 'globals', | ||
| 'samples', 'interval', 'precision', 'daemonize', | ||
| 'timefmt', 'everything', | ||
| 'timefmt', 'everything', 'times', |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
pmconfig_file="$(fd -i '^pmconfig\.py$' . src | head -n1)"
echo "Inspecting: ${pmconfig_file}"
rg -n -C3 'def read_options|setattr|times|time_stamp' "$pmconfig_file" \
src/pcp2openmetrics/pcp2openmetrics.pyRepository: performancecopilot/pcp
Length of output: 8199
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '150,240p' src/python/pcp/pmconfig.py
printf '\n---\n'
sed -n '1,140p' src/pcp2openmetrics/pcp2openmetrics.py
printf '\n---\n'
sed -n '160,245p' src/pcp2openmetrics/pcp2openmetrics.pyRepository: performancecopilot/pcp
Length of output: 14180
Map the config option to time_stamp
times=yes in pcp2openmetrics.conf is parsed onto self.times, but output is gated by self.time_stamp, so the config key never changes behavior. Map this option to the runtime flag (or rename the flag to match).
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/pcp2openmetrics/pcp2openmetrics.py` at line 58, The config option
handling in the pcp2openmetrics parser is wired to the wrong runtime field, so
`times=yes` never affects output. Update the option mapping in the config
parsing logic around the `times` entry so it sets `self.time_stamp` (or rename
the runtime flag consistently), and verify the output gating that reads
`self.time_stamp` now reflects the `pcp2openmetrics.conf` setting.
| opts.pmSetLongOption("space-scale-force", 1, "B", "SCALE", "forced space unit") | ||
| opts.pmSetLongOption("time-scale", 1, "y", "SCALE", "default time unit") | ||
| opts.pmSetLongOption("time-scale-force", 1, "Y", "SCALE", "forced time unit") | ||
| opts.pmSetLongOption("times", 0, "l", "", "addition of timestamp gauge metric") |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Make the --times help text match the implementation.
This option does not add a new “timestamp gauge metric”; Line 505 appends a sample timestamp to emitted series. The current --help text will send users looking for a metric that never appears.
Suggested wording
- opts.pmSetLongOption("times", 0, "l", "", "addition of timestamp gauge metric")
+ opts.pmSetLongOption("times", 0, "l", "", "append sample timestamps to live output")📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| opts.pmSetLongOption("times", 0, "l", "", "addition of timestamp gauge metric") | |
| opts.pmSetLongOption("times", 0, "l", "", "append sample timestamps to live output") |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/pcp2openmetrics/pcp2openmetrics.py` at line 204, The `--times` option
help text in `pmSetLongOption` is misleading because the implementation in
`pcp2openmetrics.py` only appends a sample timestamp to emitted series rather
than adding a new metric. Update the `--times` description to reflect the actual
behavior, and keep the wording aligned with the logic used later when timestamps
are emitted so users understand what the flag does.
2 participants
update pcp2openmetrics tool to allow for live timestamp mode option. This update follows openmetric / prometheus exposing timestamp guidelines. Updated man page & QA #1131
Resolves issue #2538