一种实用程序,它将任务分布在固定数量的进程上,并整理输出,以更好地利用多进程
disttask的Python项目详细描述
disttask是一个实用程序,它提供了跨固定数量的进程分发任务并整理输出的能力,以更好地利用多处理。
使用现有的单线程/进程工具和脚本,充分利用计算机资源。
提供跨固定数量的进程分发任务的能力,以更好地利用多处理。
用法
Usage: disttask [cmd] [concurrent tasks] [argset]
Use a %s in [cmd] where you want the args to go. use %d for the pipe number.
To run a list of commands (job server), have ‘%s’ be your full command.
If you need a literal %s or %d, use %%s or %%d.
Options:
-nc or –no-collate By default, the output will be held until the task is completed, so output is not intermixed.
By providing “-nc” or “–no-collate”, instead each line that comes in from any running task
is printed, prefixed with the argset in square-brackets (e.x. “[arg1] Some message”
来自stdin的参数集
If argset is ‘–’, then the argset items will be read from stdin instead of being provided on the commandline.
Execution begins immediately, so you can use disttask as a job manager with another process feeding in items
as they become available.
最大并发性
You may use “0” or “MAX” as the “concurrent tasks” parameter to execute all items in the argset simultaneously示例用法
disttask “ssh root@%s hostname” 3 host1 host2 host3 host4 host5 host6 # Connect and get hostname on 6 hosts, 3 at a time.disttask 2.2.0版
并发性
应用程序最多运行“并发任务”个进程(很好地匹配这个数字-1可用的处理器总数)。 它捕获stdout和stderr,以确保应用程序之间的任何输出都不会交织在一起。
输出
在完成每个任务后,将完整地写入输出,除非提供“-nc”或“–no collate”参数,在这种情况下,只要输出可用,就会立即打印输出,每行的前缀都是方括号中的项名称。
指定命令
每个命令都应该指定一个“%s”到每个参数的位置(每个应用程序@argset中有一个参数)。若要馈送整个命令,只需使用“%s”作为“cmd”,并在“argset”中提供命令。 这些命令由shell执行,因此可以包含shell表达式。
%d也可用作管道编号,可用于区分正在运行的进程。
还可以通过提供“-”作为参数集,将程序或文件管道输出到disttask。disttask将运行并从stdin读取,将每一行视为argset项,并在stdin关闭且所有子进程完成时终止。
示例
使用disttask并行连接到各个主机并执行命令:
disttask “echo -n ‘pipe %d: ‘ && ssh root@%s hostname” 2 “host1” “host2” “host3” “host4”
pipe 0: host1
pipe 1: host2
pipe 0: host3
pipe 1: host4
同样的例子,使用“no collate”选项:
disttask “echo -n ‘pipe %d: ‘ && ssh root@%s hostname” 2 “host1” “host2” “host3” “host4”
[host1] pipe 0: host1
[host2] pipe 1: host2
[host3] pipe 0: host3
[host4] pipe 1: host4
在子目录中的所有python文件上运行pyflakes,同时使用10个进程(需要shopt-s globstar)。注意倒勾,而不是单引号。)
disttask “pyflakes %s” 10 echo **/*.py
用法:disttask[cmd][concurrent tasks][argset]
在[cmd]中使用一个%s作为参数。使用%d作为管道编号。 若要运行命令列表(作业服务器),请将%s作为完整命令。
Options:
- -nc or –no-collate By default, the output will be held until the task is completed, so output is not intermixed.
- By providing “-nc” or “–no-collate”, instead each line that comes in from any running task is printed, prefixed with the argset in square-brackets (e.x. “[arg1] Some message”
来自stdin的参数集
If argset is ‘–’, then the argset items will be read from stdin instead of being provided on the commandline. Execution begins immediately, so you can use disttask as a job manager with another process feeding in items as they become available.最大并发性
You may use “0” or “MAX” as the “concurrent tasks” parameter to execute all items in the argset simultaneously示例用法
disttask “ssh root@%s hostname” 3 host1 host2 host3 host4 host5 host6 # Connect and get hostname on 6 hosts, 3 at a time.
disttask 2.1.1版