一个将csv转换为拼花文件的工具
csv2parquet的Python项目详细描述
csv2parquet
将csv转换为拼花文件。你也会发现sqlite-parquet-vtable很有用。
安装
如果您只想使用工具：
sudo pip install pyarrow csv2parquet
如果要克隆repo并使用该工具，请通过pipenv安装其依赖项：
pipenv install
用法
接下来，创建一些拼花文件。该工具支持csv和tsv文件。
usage: csv2parquet [-h] [-n ROWS] [-r ROW_GROUP_SIZE] [-o OUTPUT] [-c CODEC]
[-i INCLUDE [INCLUDE ...] | -x EXCLUDE [EXCLUDE ...]]
[-R RENAME [RENAME ...]] [-t TYPE [TYPE ...]]
csv_file
positional arguments:
csv_file input file, can be CSV or TSV
optional arguments:
-h, --help show this help message and exit
-n ROWS, --rows ROWS The number of rows to include, useful for testing.
-r ROW_GROUP_SIZE, --row-group-size ROW_GROUP_SIZE
The number of rows per row group.
-o OUTPUT, --output OUTPUT
The parquet file
-c CODEC, --codec CODEC
The compression codec to use (brotli, gzip, snappy,
zstd, none)
-i INCLUDE [INCLUDE ...], --include INCLUDE [INCLUDE ...]
Include the given columns (by index or name)
-x EXCLUDE [EXCLUDE ...], --exclude EXCLUDE [EXCLUDE ...]
Exclude the given columns (by index or name)
-R RENAME [RENAME ...], --rename RENAME [RENAME ...]
Rename a column. Specify the column to be renamed and
its new name, eg: 0=age or person_age=age
-t TYPE [TYPE ...], --type TYPE [TYPE ...]
Parse a column as a given type. Specify the column and
its type, eg: 0=bool? or person_age=int8. Parse errors
are fatal unless the type is followed by a question
mark. Valid types are string (default), bool, int8,
int16, int32, int64, float32, float64, timestamp
测试
pylint csv2parquet
pytest
欢迎加入QQ群-->： 979659372
