Skip to content

[Metrics] Add preemption metrics and accumulation logic - #5396

Open
PauloVLB wants to merge 1 commit into
preemption-pollingfrom
preemption-metrics
Open

[Metrics] Add preemption metrics and accumulation logic#5396
PauloVLB wants to merge 1 commit into
preemption-pollingfrom
preemption-metrics

Conversation

@PauloVLB

@PauloVLB PauloVLB commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

b/538558303

Note: this PR is chained with #5394

Problem

When a preemption occurs, we were only recording the duration of the active fuzzing round as wasted time. If a task is preempted mid-way, all previous successful rounds in that same task are also completely wasted since the task never reaches postprocess. This leads to a significant underestimation of wasted time.

Proposed Solution

Enhanced time tracking to capture the waste across the entire fuzzing session while maintaining metric accuracy.

  • Added FUZZER_PREEMPTED_TOTAL_FUZZ_TIME and JOB_PREEMPTED_TOTAL_FUZZ_TIME metrics to monitoring_metrics.py.
  • Enhanced _TrackFuzzTime in fuzz_task.py to track accumulated wasted time across all rounds of a session (_accumulated_wasted_time).
  • Upon preemption, the callback now emits the Total Accumulated Session Time (Current Round + Previous Rounds) to preempted_total_time.
  • To fix subtraction math in dashboards, the callback also emits the duration of the interrupted round only to total_time, ensuring it reflects the attempted time without gaps.
  • Added tracker.preempted flag to skip metric emission in __exit__ for interrupted rounds and avoid double counting.

Validation

Validated in dev environment.

image image

@PauloVLB
PauloVLB force-pushed the preemption-metrics branch from 1064ba3 to f369557 Compare July 28, 2026 21:58
@PauloVLB
PauloVLB marked this pull request as ready for review July 28, 2026 22:13
@PauloVLB
PauloVLB requested a review from a team as a code owner July 28, 2026 22:13
@PauloVLB
PauloVLB force-pushed the preemption-metrics branch from f369557 to 0c5f861 Compare July 29, 2026 13:27
],
)

FUZZER_PREEMPTED_TOTAL_FUZZ_TIME = monitor.CounterMetric(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't include build set up and corpus sync time, which is also effectively wasted time from a fuzzing perspective. But our total fuzz time metrics also don't include that, so I understand that we may want it to match.

I don't know how long that set up takes, but it might be worth tracking that as well.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, not blocking at all because any metrics here will help but definitely worth discussing/thinking about

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it was done that way to match the existing metric, so the subtraction (total_time - preempted_time) makes sense.

Measuring the duration of the full main stage (including build set up and corpus sync) is the next step I'm working on. Thanks for pointing that out!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants