After upgrading to Apache Beam Python SDK to 2.75.0, runtime parameters passed to a Dataflow Flex Template are not being resolved correctly.
A parameter defined using add_value_provider_argument() is being interpreted as a string representation of RuntimeValueProvider instead of the actual runtime value. There are few args passed which seems to have no issues, only difference is that there is no default value set in all the args but a value is passed expect for the one which has the issue.
Expected behaviour
When a runtime parameter is defined:
class MyOptions(PipelineOptions): @classmethod def _add_argparse_args(cls, parser): parser.add_value_provider_argument( "--vpt_vp_arg", dest="vpt_vp_arg" required=False )
and accessed inside a DoFn:
vpt_vp_arg = options.vpt_vp_arg.get()
I expect:
None
or the configured runtime value.
Actual behaviour
get() returns:
'RuntimeValueProvider(option_name=vpt_vp_arg, value_type=str, default_value=None)'
Environment
Apache Beam Python SDK: 2.75.0
Runner: Google Cloud Dataflow
Template type: Flex Template
After upgrading to Apache Beam Python SDK to 2.75.0, runtime parameters passed to a Dataflow Flex Template are not being resolved correctly.
A parameter defined using add_value_provider_argument() is being interpreted as a string representation of RuntimeValueProvider instead of the actual runtime value. There are few args passed which seems to have no issues, only difference is that there is no default value set in all the args but a value is passed expect for the one which has the issue.
Expected behaviour
When a runtime parameter is defined:
class MyOptions(PipelineOptions): @classmethod def _add_argparse_args(cls, parser): parser.add_value_provider_argument( "--vpt_vp_arg", dest="vpt_vp_arg" required=False )and accessed inside a DoFn:
vpt_vp_arg = options.vpt_vp_arg.get()I expect:
None
or the configured runtime value.
Actual behaviour
get() returns:
'RuntimeValueProvider(option_name=vpt_vp_arg, value_type=str, default_value=None)'Environment
Apache Beam Python SDK: 2.75.0
Runner: Google Cloud Dataflow
Template type: Flex Template