有什么方法可以在线程处理时用BeautifulSoup解析吗？

Question

如何解析多线程链接？你知道吗

基本上我是在寻找链接，然后逐一解析这些链接。你知道吗

它是这样做的：

for link in links:
    scrape_for_info(link)

链接包含：

https://www.xtip.co.uk/en/?r=bets/xtra&group=476641&game=312053910
https://www.xtip.co.uk/en/?r=bets/xtra&group=476381&game=312057618
...
https://www.xtip.co.uk/en/bets/xtra.html?group=477374&game=312057263

scrape\u for\u info（url）如下所示：

def scrape_for_info(url):

    scrape = CP_GetOdds(url)

    for x in range(scrape.GameRange()):
     sql_str = "INSERT INTO Scraped_Odds ('"
     sql_str += str(scrape.Time()) + "', '"
     sql_str += str(scrape.Text(x)) + "', '"
     sql_str += str(scrape.HomeTeam()) + "', '"
     sql_str += str(scrape.Odds1(x)) + "', '"
     sql_str += str(scrape.Odds2(x)) + "', '"
     sql_str += str(scrape.AwayTeam()) + "')"

     cursor.execute(sql_str)
    conn.commit()

我看到线程在抓取网站时被使用，但它主要用于抓取而不是解析。你知道吗

我希望有人能教我如何比现在更快地解析。当我看现场赔率时，我必须尽快更新

Answer 1

使用multiprocessing可以考虑使用Queue。你知道吗

通常您会创建两个作业，一个创建URL，另一个使用URL。我们叫它们creator和consumer。我将假设这里的任何信号量都被称为closing_condition（例如使用Value），用于解析url并保存它们的方法分别被称为create_url_method和store_url。你知道吗

from multiprocessing import Queue, Value, Process
import queue


def creator(urls, closing_condition):
    """Parse page and put urls in given Queue."""
    while (not closing_condition):
        created_urls = create_url_method()
        [urls.put(url) for url in created_urls]


def consumer(urls, closing_condition):
    """Consume urls in given Queue."""
    while (not closing_condition):
        try:
            store_url(urls.get(timeout=1))
        except queue.Empty:
            pass


urls = Queue()
semaphore = Value('d', 0)

creators_number = 2
consumers_number = 2

creators = [
    Process(target=creator, args=(urls, semaphore))
    for i in range(creators_number)
]

consumers = [
    Process(target=consumer, args=(urls, semaphore))
    for i in range(consumers_number)
]

[p.start() for p in creators + consumer]
[p.join() for p in creators + consumer]

Answer 2

谢谢你的回答！你知道吗

以下是成功的秘诀：

from multiprocessing import Pool

with Pool(10) as p:
    p.map(scrape_for_info, links))

Answer 3

在中有一个很好的例子，它可以用Python自动化那些无聊的东西。你知道吗

https://automatetheboringstuff.com/chapter15/

基本上，您需要使用threading模块为每个url创建一个不同的线程，然后等待它们全部完成。你知道吗

import threading

def scrape_for_info(url):
    scrape = CP_GetOdds(url)

    for x in range(scrape.GameRange()):
        sql_str = "INSERT INTO Scraped_Odds ('"
        sql_str += str(scrape.Time()) + "', '"
        sql_str += str(scrape.Text(x)) + "', '"
        sql_str += str(scrape.HomeTeam()) + "', '"
        sql_str += str(scrape.Odds1(x)) + "', '"
        sql_str += str(scrape.Odds2(x)) + "', '"
        sql_str += str(scrape.AwayTeam()) + "')"

     cursor.execute(sql_str)
     conn.commit()

# Create and start the Thread objects.
threads = []
for link in links:
    thread = threading.Thread(target=scrape_for_info, args=(link))
    threads.append(thread)
    thread.start()

# Wait for all threads to end.
for thread in threads:
    thread.join()
print('Done.')