Airflow mysql到google云存储，需要bytelike对象

Question

我正在开发一个DAG，它可以查询MySQL数据库，提取数据并将其加载到Google云存储中。在

我试图导出的表包括text、int、float、varchar（20）和varchar（32）数据。在

我使用的是气流v1.8.0。在

default_args = {
    'owner' : 'tia',
    'start_date' : datetime(2018, 1, 4),
    'depends_on_past' : False,
    'retries' : 1,
    'retry_delay':timedelta(minutes=5),
}

dag = DAG('mysql_to_gcs', default_args=default_args)

export_waybills = MySqlToGoogleCloudStorageOperator(
   task_id='extract_waybills',
   mysql_conn_id = 'podiotestmySQL',
   sql = 'SELECT * FROM podiodb.logistics_waybills',
   bucket='podio-reader-storage',
   filename= 'podio-data/waybills{}.json',
   schema_filename='podio-data/schema/waybills.json',
   dag=dag)

我遇到了下面的错误，它似乎与Airflow mysql to gcp Dag error相似

[2018-01-04 11:12:23,372] {models.py:1342} INFO - Executing on 2018-01-04 00:00:00

[2018-01-04 11:12:23,400] {base_hook.py:67} INFO - Using connection to: 35.189.207.140

[2018-01-04 11:12:24,903] {models.py:1417} ERROR - a bytes-like object is required, not 'str'

Traceback (most recent call last):

File "/home/hyperli/.local/lib/python3.5/site-packages/airflow/models.py", line 1374, in run

result = task_copy.execute(context=context)

File "/home/hyperli/.local/lib/python3.5/site-packages/airflow/contrib/operators/mysql_to_gcs.py", line 91, in execute

files_to_upload = self._write_local_data_files(cursor)

File "/home/hyperli/.local/lib/python3.5/site-packages/airflow/contrib/operators/mysql_to_gcs.py", line 136, in _write_local_data_files

json.dump(row_dict, tmp_file_handle)

File "/usr/lib/python3.5/json/init.py", line 179, in dump fp.write(chunk)

File "/usr/lib/python3.5/tempfile.py", line 622, in func_wrapper return func(*args, **kwargs)

TypeError: a bytes-like object is required, not 'str'

[2018-01-04 11:12:24,907] {models.py:1433} INFO - Marking task as UP_FOR_RETRY

[2018-01-04 11:12:25,037] {models.py:1462} ERROR - a bytes-like object is required, not 'str'

有人知道为什么抛出异常吗？在

Answer 1

你在用python3吗？因为它与最新版本（1.90）的mysql_到\u gcs不兼容，所以它与python不兼容。在

很明显，这个操作符被改了here，使mysql与gcs py3兼容。但是latest version(1.90)不包括this change