For循环返回空列表

Question

我正在尝试使用函数pull_Brand从sptct_links中的链接中提取所有品牌：

from selenium import webdriver
from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options
from bs4 import BeautifulSoup
import pandas as pd

opts = Options()
opts.headless=True
assert opts.headless  # Operating in headless mode
browser = Firefox(options=opts)

sptct_links =['https://www.macys.com/shop/mens-clothing/mens-blazers-sports-coats/Productsperpage/120?id=16499',
        'https://www.macys.com/shop/mens-clothing/mens-blazers-sports-coats/Pageindex,Productsperpage/2,120?id=16499',
        'https://www.macys.com/shop/mens-clothing/mens-blazers-sports-coats/Pageindex,Productsperpage/3,120?id=16499',
        'https://www.macys.com/shop/mens-clothing/mens-blazers-sports-coats/Pageindex,Productsperpage/4,120?id=16499',
        'https://www.macys.com/shop/mens-clothing/mens-blazers-sports-coats/Pageindex,Productsperpage/5,120?id=16499']

def pull_Brand(url):
    browser.get(url)
    html = browser.page_source
    soup = BeautifulSoup(html, 'html.parser')
    brand = []
    for tag in soup.find_all(class_='productBrand'):
        brand.append(tag.text.strip())
    print(brand)

for i in range(len(sptct_links)):
        pull_Brand(sptct_links[i])

browser.quit()

虽然它成功地为sptct_links[0]在列表brand中添加了信息，但随后的列表返回空。有没有关于为什么会发生这种情况以及如何修复的想法

非常感谢你

Answer 1

如果删除opts=headless=True选项，后续链接将出现以下错误：

Access Denied You don't have permission to access the requested URL on this server.

Reference: 18.3d702617.1593528484.1aacf782

如果您要更改列表中链接的顺序，则无论第一个链接是什么，都将起作用，但随后的链接将失败。所以，链接本身并没有什么问题。我的猜测是，该网站检测到浏览器正在由自动化（Selenium）运行，并且只允许您进行一页的web访问

我在页面抓取之间暂停了5秒钟，但除了第一页之外，所有抓取都出现了错误，所以这不是抓取速率的问题

Answer 2

要获得正确的页面，请在请求中设置User-AgentHttp头

例如：

sptct_links =['https://www.macys.com/shop/mens-clothing/mens-blazers-sports-coats/Productsperpage/120?id=16499',
        'https://www.macys.com/shop/mens-clothing/mens-blazers-sports-coats/Pageindex,Productsperpage/2,120?id=16499',
        'https://www.macys.com/shop/mens-clothing/mens-blazers-sports-coats/Pageindex,Productsperpage/3,120?id=16499',
        'https://www.macys.com/shop/mens-clothing/mens-blazers-sports-coats/Pageindex,Productsperpage/4,120?id=16499',
        'https://www.macys.com/shop/mens-clothing/mens-blazers-sports-coats/Pageindex,Productsperpage/5,120?id=16499']

headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0'}

for link in sptct_links:
    soup = BeautifulSoup(requests.get(link, headers=headers).content, 'html.parser')  # <  set headers=

    for brand in soup.select('.productBrand'):
        print(brand.get_text(strip=True))

印刷品：

Michael Kors
MICHAEL Michael Kors
Michael Kors
Bar III
Bar III
Bar III
Unlisted by Kenneth Cole
Tallia
Michael Kors
Bar III

...and so on.