如何修复<span>

Question

我正在尝试使用python3.7和beauthoulsoup的web scraper。我已经从下面的html中提取了“发布名称”、“按位置排序发布类别小类别标签”、“按团队发布类别小类别标签排序”的数据，但无法提取“按承诺发布类别小类别标签排序”（无论是否全职），而html结构似乎与其他html结构相同：

<div class="posting" data-qa-posting-id="13f9db2f-7a80-4b50-9a61-005ad322ea2d">
   <div class="posting-apply" data-qa="btn-apply">
      <a href="https://jobs.lever.co/twitch/13f9db2f-7a80-4b50-9a61-005ad322ea2d" class="posting-btn-submit template-btn-submit hex-color">Apply</a>
   </div>
   <a class="posting-title" href="https://jobs.lever.co/twitch/13f9db2f-7a80-4b50-9a61-005ad322ea2d">
      <h5 data-qa="posting-name">Account Director - DACH</h5>
      <div class="posting-categories">
         <span href="#" class="sort-by-location posting-category small-category-label">Hamburg, Germany</span>
         <span href="#" class="sort-by-team posting-category small-category-label">Business Operations &amp; Go-To-Market – Advertising</span>
         <span href="#" class="sort-by-commitment posting-category small-category-label">Full-time</span>
      </div>
   </a>
</div>

我尝试为“发布类别”创建一个单独的汤，但没有成功。在

^{pr2}$

csv中的预期结果将返回职位名称、链接（url）、位置、团队和承诺。在

到目前为止，我收到以下错误：

 commitment = post.find('span',{'class':'sort-by-commitment posting-category small-category-label'}).text
AttributeError: 'NoneType' object has no attribute 'text'

*编辑：数据集缺少最后一行，我不知道为什么：

<a class="posting-title" href="https://jobs.lever.co/twitch/c8cc56e7-75f6-4cac-9983-e0769db9dd2e">
   <h5 data-qa="posting-name">Applied Scientist Intern</h5>
   <div class="posting-categories">
      <span href="#" class="sort-by-location posting-category small-category-label">San Francisco, CA</span>
      <span href="#" class="sort-by-team posting-category small-category-label">University (Internships) – Engineering</span>
      <span href="#" class="sort-by-commitment posting-category small-category-label">Intern</span>

Answer 1

您还可以使用try except：

for post in posts:
    try:
        position = post.find('h5',{'data-qa':'posting-name'}).text
        link = post.find('a')['href']
        location = post.find('span',{'class':'sort-by-location posting-category small-category-label'}).text
        team = post.find('span',{'class':'sort-by-team posting-category small-category-label'}).text
        commitment = post.find('span',{'class':'sort-by-commitment posting-category small-category-label'}).text
        csv_writer.writerow([position, link, location, team, commitment])
    except:
        continue

Answer 2

如果检查html，在某些情况下commitment丢失，则必须提供If条件。试试下面的代码现在。在

for post in posts:
        position = post.find('h5',{'data-qa':'posting-name'}).text
        link = post.find('a')['href']
        location = post.find('span',{'class':'sort-by-location posting-category small-category-label'}).text
        team = post.find('span',{'class':'sort-by-team posting-category small-category-label'}).text
        if post.find('span',{'class':'sort-by-commitment posting-category small-category-label'}):
            commitment = post.find('span',{'class':'sort-by-commitment posting-category small-category-label'}).text
            csv_writer.writerow([position, link, location, team, commitment])

我建议您使用css selector，而不是{}。在

^{pr2}$