支持gzip的urlgrabber

Question

使用urlgrabber时，建议使用什么方法来处理Content-Encoding: gzip文件？你知道吗

现在我正在用猴子修补它，就像这样：

g = URLGrabber(http_headers=(("Accept-Encoding", "gzip"),))
g.is_compressed = False # I don't know yet if the server will send me compressed data

# Backup current method of handling downloaded headers
try:
    PyCurlFileObject.orig_hdr_retrieve
except AttributeError:
    PyCurlFileObject.orig_hdr_retrieve = PyCurlFileObject._hdr_retrieve

def hdr_retrieve(instance, buf):
    r = PyCurlFileObject.orig_hdr_retrieve(instance, buf)
    if "content-encoding" in buf.lower() and "zip" in buf.lower():
        g.is_compressed = True
    return r
PyCurlFileObject._hdr_retrieve = hdr_retrieve

g.urlgrab(url, dest)

if g.is_compressed:
    # ungzip file here

但它看起来不太干净，我担心它也不安全。。。你知道吗

Answer 1

我想我找到了一个线程安全的解决方案：

g = URLGrabber((http_headers=(("Accept-Encoding", "gzip"),)))
g.opts._set_attributes(grabber=g)
try:
    PyCurlFileObject.orig_setopts
except AttributeError:
    PyCurlFileObject.orig_setopts = PyCurlFileObject._set_opts

    def setopts(instance, opts={}):
        PyCurlFileObject.orig_setopts(instance, opts)
        grabber = instance.opts.grabber
        grabber.is_compressed = False

        def hdr_retrieve(buf):
            r = PyCurlFileObject._hdr_retrieve(instance, buf)
            if "content-encoding" in buf.lower() and "zip" in buf.lower():
                grabber.is_compressed = True
            return r

        instance.curl_obj.setopt(pycurl.HEADERFUNCTION, hdr_retrieve)
    PyCurlFileObject._set_opts = setopts

但感觉还是不太“干净”：