Skip to content

informer resetting resourceversion to "" leads to watch event missing #15106

@JackTroy

Description

@JackTroy

Pre-requisites

  • I have double-checked my configuration
  • I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
  • I have searched existing issues and could not find a match for this bug
  • I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

First we found that there is always the error: withParam value could not be parsed as a JSON list. This is the original problem.
Then we found out that argo would receive duplicate watch and miss event in between. Also there is periodical spike for WorkflowTaskResult watch event in apiserver metrics.
After digging in, we realize that informer would reset ResourceVersion = "" no matter how. This would cause watch can not continue from last rv and get all current event from cache.

We will provide a fix for it.

Version(s)

v3.6.10

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

x-task-843048499:
  id: x-task-843048499
  name: x-task.g
  displayName: g
  type: Skipped
  templateName: g
  templateScope: local/
  phase: Error
  boundaryID: x-task
  message: >-
    withParam value could not be parsed as a JSON list:
    {{tasks.parse.outputs.parameters.output-list}}: invalid character '{'
    looking for beginning of object key string
  startedAt: '2025-08-18T22:33:42Z'
  finishedAt: '2025-08-18T22:33:42Z'
  estimatedDuration: 541
  nodeFlag: {}
  children:
    - x-task-711488156

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions