使用函数将作业转换为数据流

Question

参数谷歌函数

eu quero passar参数参数参数查询

模板文件：

"steps":[
      {
         "kind":"ParallelRead",
         "name":"s1",
         "properties":{
            "bigquery_export_format":"FORMAT_AVRO",
            "bigquery_flatten_results":true,
            "bigquery_query":"select * from `myproject2497.teste.teste`",
            "bigquery_use_legacy_sql":false,
            "display_data":[
               {
                  "key":"source",
                  "label":"Read Source",
                  "namespace":"apache_beam.io.iobase.Read",
                  "shortValue":"BigQuerySource",
                  "type":"STRING",
                  "value":"apache_beam.io.gcp.bigquery.BigQuerySource"
               },
               {
                  "key":"query",
                  "label":"Query",
                  "namespace":"apache_beam.io.gcp.bigquery.BigQuerySource",
                  "type":"STRING",
                  "value":"select * from `myproject2497.teste.teste limit 1`"
               },

谷歌云的功能： blob.upload\u from\u字符串（时间戳）

parameters = {"bql": bql}
jobname = "poc"    
gcsPath="gs://exemplebucket1321/teste/templates/Bee"
body = {
    "jobName": "{jobname}".format(jobname=jobname),
    "parameters": parameters
}

Answer 1

请参阅有关创建以下内容的文档：

https://cloud.google.com/dataflow/docs/guides/templates/creating-templates

和运行模板：

https://cloud.google.com/dataflow/docs/guides/templates/running-templates

请注意，您将只在已经使用ValueProvider的模板代码中传入参数（不更改源代码）。有关具有ValueProvider选项的IOs列表，请参见下面的链接。对于BigQuery，您需要Java SDK 2.0+

https://cloud.google.com/dataflow/docs/guides/templates/creating-templates#pipeline-io-and-runtime-parameters

由于您使用BigQuery作为源代码，我相信遵循上述两个链接中的说明将解决您的问题