如何使用字典作为输入进行多线程处理？

Question

我是python的新手，正在尝试理解多线程

到目前为止，我掌握的情况如下：

 d_thread = {0:(('instrumentType', 'OPTSTK'), ('symbol', 'INFY'),
                ('expiryDate', 'select'), ('optionType', 'PE'),
                ('strikePrice', '2800'), ('dateRange', 'day'),
                ('fromDate', '11-04-2012'),('toDate', '12-04-2012'),
                ('segmentLink', '9'), ('symbolCount', '')),
            12:(('instrumentType', 'OPTSTK'), ('symbol', 'INFY'), 
                ('expiryDate', 'select'), ('optionType', 'PE'), 
                ('strikePrice', '2400'), ('dateRange', 'day'), 
                ('fromDate', '27-04-2012'), ('toDate', '28-04-2012'), 
                ('segmentLink', '9'), ('symbolCount', ''))}

这本词典大约有500 keys。大熊猫的keys是index。你知道吗

我想创建10 workers来生成requests，然后将数据放入dataframe。我不知道如何让worker在一个线程完成时选择next键。你知道吗

到目前为止我所拥有的：

import threading
from queue import Queue
import requests

hist_lock = threading.Lock()

def opthist_job(worker,d_thread):
      headers = {
    'Pragma': 'no-cache',
    'Accept-Encoding': 'gzip, deflate, br',
    'Accept-Language': 'en-US,en;q=0.9',
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',
    'Accept': '*/*',
    'Referer': 'https://www.nseindia.com/products/content/derivatives/equities/historical_fo.htm',
    'X-Requested-With': 'XMLHttpRequest',
    'Connection': 'keep-alive',
    'Cache-Control': 'no-cache',
}
params = d_threading[0]    # This is where I need to get the value of key
opthistdf = requests.get('https://www.nseindia.com/products/dynaContent/common/productsSymbolMapping.jsp', headers=headers, params=params)

 with hist_lock: # I am not sure if this is required in this instance.

 #### Some more functions ####

 def threader():
      while True:
          worker = q.get()
          opthist_job(worker)
          q.task_done()

 q = Queue()
 for th in range(len(d_threading.keys())):
       t=threading.Thread(target=threader)
       t.daemon = True
       t.start()

提前谢谢。你知道吗

Answer 1

您可能需要使用mulitprocessing.pool库提供的内容。你知道吗

让我们尝试使用map函数：

from multiprocessing import Pool

def f(parameter):   #in parameter you have a tuple (key, value) from your dict
    result =  requests.get('https://www.nseindia.com/products/dynaContent/common/productsSymbolMapping.jsp', headers=headers, params=parameter[1])
    return (parameters[0], result)


if __name__ == '__main__':
    with Pool() as pool:
        result = pool.map(f, d_thread.items())
        print(result)    #this should show you the results as a list of (key, result)
        print(dict(result))   #here you have a dict of your results