无法使用Dataflow+Beam+Python创建模板

Question

python -m main \ --setup_file setup.py \ --runner DataflowRunner \ --project my-test \ --staging_location gs://my-test/staging \ --temp_location gs://my-test/temp \  --template_location gs://my-test/templates/test --output gs://my-test/output

上面的命令只在本地运行（要求在本地安装依赖项），而不创建模板。以下是中的管道选项主.py公司名称：

^{pr2}$

这是设置.py公司名称：

import subprocess

import setuptools
from setuptools.command.bdist_egg import bdist_egg as _bdist_egg


class bdist_egg(_bdist_egg): 
   def run(self):
      self.run_command('CustomCommands')
      _bdist_egg.run(self)
CUSTOM_COMMANDS = [
   ['apt-get', 'update'],
   ['apt-get', '--assume-yes', 'install', 'libproj-dev', 'libgdal-dev'],
   ['export' 'CPLUS_INCLUDE_PATH=/usr/include/gdal'],
   ['export' 'C_INCLUDE_PATH=/usr/include/gdal'],
   ['gdal-config', '--version'],
   ['pip', 'install', 'pygdal==1.11.3.3'],
   ['echo', 'Custom command worked!']]


class CustomCommands(setuptools.Command):


   def initialize_options(self):
      pass

   def finalize_options(self):
      pass

   def RunCustomCommand(self, command_list):

      p = subprocess.Popen(
         command_list,
         stdin=subprocess.PIPE, stdout=subprocess.PIPE, 
         stderr=subprocess.STDOUT)

      stdout_data, _ = p.communicate()

      if p.returncode != 0:
         raise RuntimeError(
          'Command %s failed: exit code: %s' % (command_list, p.returncode))

     def run(self):
        for command in CUSTOM_COMMANDS:
           self.RunCustomCommand(command)




    REQUIRED_PACKAGES = [
       'Shapely',
       'pyshp',
       'beam_utils',
       "google-cloud-storage==1.3.2",
       "google-auth",
       "requests>=2.18.0"
     ]


     setuptools.setup(
        name='ETL',
        version='0.0.1',
        description='',
        install_requires=REQUIRED_PACKAGES,
        packages=setuptools.find_packages(),
        cmdclass={

           'bdist_egg': bdist_egg,
           'CustomCommands': CustomCommands,
        }
    )

如何在具有非Python依赖关系的数据流中创建模板？ 我得到的错误是

WindowsError: [Error 123] The filename, directory name, or volume label syntax is incorrect. 

为什么代码在本地运行而不是创建模板ro在其他时间执行？请帮忙！

Answer 1

我在代码中找到了导致错误消息的位置。 在管道选项中，它应该是：

'setup_file':'./setup.py',