<p>我编写了一个python脚本,在一堆文件上运行一个名为“gtdownload”的多处理命令。“下载”功能是我遇到麻烦的地方。在</p>
<pre><code>#/usr/bin/env python
import os, sys, subprocess
from multiprocessing import Pool
def cghub_dnld_file(file1, file2, file3, np):
<open files>
<read in lines>
p = Pool(int(np))
map_args = [(line.rstrip(),name_lines[i].rstrip(),bar_lines[i].rstrip()) for i, line in enumerate(id_lines)]
p.map(download_wrapper,map_args)
def download(id, name, bar):
<check if file has been downloaded, if not download>
<.....>
link = "https://cghub.ucsc.edu/cghub/data/analysis/download/" + id
dnld_cmd = "gtdownload -c ~/.cghub.key --max-children 4 -vv -d " + link + " > gt.out 2>gt.err"
subprocess.call(dnld_cmd,shell=True)
def download_wrapper(args):
return download(*args)
def main():
<read in arguments>
<...>
cghub_dnld_file(file1,file2,file3,threads)
if __name__ == "__main__":
main()
</code></pre>
<p>如果数据库中不存在此文件,gtdownload将退出,这也会终止我的python作业,并出现以下错误:</p>
^{pr2}$
<p>来自gtdownload的实际错误消息:</p>
<pre><code>Welcome to gtdownload-3.8.5a.
Ready to download
Communicating with GT Executive ...
Headers received from the client: 'HTTP/1.1 100 Continue
HTTP/1.1 404 Not Found
Date: Tue, 29 Jul 2014 18:49:57 GMT
Server: Apache/2.2.15 (Red Hat and CGHub)
Strict-Transport-Security: max-age=31536000
X-Powered-By: PHP/5.3.3
Content-Length: 669
Connection: close
Content-Type: text/xml
'
Error: You have requested to download a uuid which either does not exist within the system, or has not yet reached the 'live' state. The requested action will not be performed. Please double check the supplied uuid or contact thelpdesk for further assistance.
</code></pre>
<p>我希望脚本跳过一个不存在的,并在下一个脚本上开始gtdownload。我试图输出subprocess.call然后查看是否有“error”关键字。但它似乎停止在确切的subprocess.call命令。同样的事情操作系统. 在</p>
<p>我做了一个没有多重处理的MCV案例,子进程根本没有杀死主进程。看起来多处理会把事情搞得一团糟,尽管我只是用一个线程来测试。在</p>
<pre><code>#!/usr/bin/env python
import subprocess
#THis is the id that gtdownload had problem with
id = "df1e073f-4485-4d5f-8659-cd8eaac17329"
link = "https://cghub.ucsc.edu/cghub/data/analysis/download/" + id
dlnd_cmd = "gtdownload -c ~/.cghub.key --max-children 4 -vv -d " + link + " > gt.out 2>gt.err"
print dlnd_cmd
subprocess.call(dlnd_cmd,shell=True)
print "done"
</code></pre>
<p>显然多处理冲突subprocess.call但我不清楚为什么。在</p>