Pyarrow从s3读/写

Question

是否可以在不使用pyarrow将parquet文件从一个文件夹读写到另一个文件夹，而不需要转换成pandas。在

这是我的代码：

import pyarrow.parquet as pq
import pyarrow as pa
import s3fs

s3 = s3fs.S3FileSystem()

bucket = 'demo-s3'

pd = pq.Parquetdataset('s3://{0}/old'.format(bucket), filesystem=s3).read(nthreads=4).to_pandas()
table = pa.Table.from_pandas(pd)
pq.write_to_dataset(table, 's3://{0}/new'.format(bucket), filesystem=s3, use_dictionary=True, compression='snappy')

Answer 1

如果您不想直接复制文件，那么您确实可以避免使用pandas，因此：

table = pq.ParquetDataset('s3://{0}/old'.format(bucket),
    filesystem=s3).read(nthreads=4)
pq.write_to_dataset(table, 's3://{0}/new'.format(bucket), 
    filesystem=s3, use_dictionary=True, compression='snappy')

Answer 2

为什么不直接复制（S3->；S3）并节省内存和I/O呢？在

import awswrangler as wr

SOURCE_PATH = "s3://..."
TARGET_PATH = "s3://..."

objs = wr.s3.list_objects(path=SOURCE_PATH)
wr.s3.copy_listed_objects(
    objects_paths=objs,
    source_path=SOURCE_PATH,
    target_path=TARGET_PATH,
    mode="append",  # Could be append, overwrite or overwrite_partitions
)

Reference