如何在XPATH-Python的Span标记之间获取文本

Question

我正在使用这个网站https://www.pealim.com/dict/?page=1。所以我基本上想知道希伯来语的单词和发音

下面是我的代码，它在所有td标记中循环，但是，它产生完全相同的输出，即下面的{'latin': 'av', 'hebrew': u'\u05d0\u05b8\u05d1'}，并且此代码仅用于page=1。我想知道是否有任何自动的方式来循环通过每一页

import requests
from lxml import etree

resp = requests.get("https://www.pealim.com/dict/?page=1")

htmlparser = etree.HTMLParser()
tree = etree.fromstring(resp.text, htmlparser)

for td in tree.xpath('//*//table[@class="table table-hover dict-table-t"]/tbody/tr'):
    print(td)
    data = {
        'hebrew': td.xpath('string(//span[@class="menukad"])'),
        'latin': td.xpath('string(//span[@class="dict-transcription"])'),
    }
    print(data)

我想收集在该网站的每一个条目的信息。请让我知道我能做什么来实现这一点

Answer 1

import requests
from bs4 import BeautifulSoup
from pprint import pprint

for i in range(1, 411):
    data = []
    resp = requests.get("https://www.pealim.com/dict/?page={}".format(i))
    soup = BeautifulSoup(resp.text, 'lxml')
    for m, t in zip(soup.select('.menukad'), soup.select('.dict-transcription')):
        data.append((m.text, t.text))

    print('PAGE {}'.format(i))
    print('*' * 80)
    pprint(data)

印刷品：

PAGE 1
********************************************************************************
[('אָב', 'av'),
 ('אַבָּא', 'aba'),
 ('אָבִיב', 'aviv'),
 ('אֵב', 'ev'),
 ('לֶאֱבוֹד', "le'evod"),
 ('לְהֵיאָבֵד', "lehe'aved"),
 ('לְאַבֵּד', "le'abed"),
 ('לְהִתְאַבֵּד', "lehit'abed"),
 ('לְהַאֲבִיד', "leha'avid"),
 ('הִתְאַבְּדוּת', "hit'abdut"),
 ('אִיבּוּד', 'ibud'),
 ('אֲבֵדָה', 'aveda'),
 ('אָבוּד', 'avud'),
 ('לְאַבְחֵן', "le'avchen"),
 ('אִיבְחוּן', 'ivchun')]
PAGE 2
********************************************************************************
[('לְאַבְטֵחַ', "le'avteach"),
 ('אִיבְטוּחַ', 'ivtuach'),
 ('אֲבַטִּיחַ', 'avatiach'),
 ('לֶאֱבוֹת', "le'evot"),
 ('אֵבֶל', 'evel'),
 ('לֶאֱבוֹל', "le'evol"),
 ('אֲבָל', 'aval'),
 ('לְהִתְאַבֵּל', "lehit'abel"),
 ('לְהִתְאַבֵּן', "lehit'aben"),
 ('אֶבֶן', 'even'),
 ('לְהַאֲבִיס', "leha'avis"),
 ('לְהֵיאָבֵק', "lehe'avek"),
 ('מַאֲבָק', "ma'avak"),
 ('לְאַבֵּק', "le'abek"),
 ('אָבָק', 'avak')]
PAGE 3
********************************************************************************
[('לְהִתְאַבֵּק', "lehit'abek"),
 ('לְהִתְאַבֵּק', "lehit'abek"),
 ('מְאוּבָּק', "me'ubak"),
 ('אִיבּוּק', 'ibuk'),

...and so on.

Answer 2

Andrej击败了我，但是你也可以使用.find()和.get_text()的BeautifulSoup方法：

import bs4
import requests

for page_number in range(1, 411):
    print("-" * 35, page_number, "-" * 35)
    resp = requests.get("https://www.pealim.com/dict/?page={}".format(page_number))
    soup = bs4.BeautifulSoup(resp.text, "html.parser")
    table_elem = soup.find("tbody")
    rows = table_elem.find_all("tr")
    for row in rows:
        hebrew = row.find("span", class_="menukad").get_text()
        latin = row.find("span", class_="dict-transcription").get_text()
        print("{}: {}".format(hebrew, latin))

产生基本相同的结果