在本地Apache Beam安装上运行SqlTransform Python SDK。收到错误：“FileNotFoundError:[Errno 2]没有这样的文件或目录：'docker'”

Question

我正在尝试使用本地Beam安装测试PythonSDK的SqlTransform。提到Beam安装在虚拟环境中，我使用的是Python 3.8.3。我收到错误“FileNotFoundError:[Errno 2]没有这样的文件或目录：'docker'”

with beam.Pipeline() as p:
pc = (p | beam.Create([
    FruitRecipe("pie", "strawberry", 3, 1.5),
    FruitRecipe("muffin", "blueberry", 2, 2.),
    ])
    | beam.Map(lambda x: beam.Row(recipe = str(x[0]),  # str
                             fruit = str(x[1]),    # str
                             quantity = int(x[2]), # int
                             unit_cost = float(x[3]), # float
                             is_berry = bool(x[1].endswith('berry'))))) # bool
pc | SqlTransform(" SELECT * FROM PCOLLECTION WHERE quantity > 1")

日志：

---

FileNotFoundError回溯（最近的调用

last) in 9 unit_cost = float(x[3]), # float 10 is_berry = bool(x[1].endswith('berry'))))) # bool ---> 11 pc | SqlTransform(" SELECT * FROM PCOLLECTION WHERE quantity > 1")

~/PROJECTS/Apache_Beam/env/lib/python3.8/site-packages/apache_beam/pipeline.py in exit(self, exc_type, exc_val, exc_tb) 580 try: 581 if not exc_type: --> 582 self.result = self.run() 583 self.result.wait_until_finish() 584 finally:

........

FileNotFoundError: [Errno 2] No such file or directory: 'docker'

Answer 1

apachebeam中的SqlTransform在单独的容器中运行。为了运行此转换（以及其他cross-language transforms），您需要让docker installed与root privileges一起运行，或者至少以提升的权限运行管道