回答此问题可获得 20 贡献值,回答如果被采纳可获得 50 分。
<p>我想刮一个网站,目前,我可以刮网站从第1页到第5页。
唯一让我恼火的是,在第3页,网站改变了div类的信息,我想刮</p>
<p>发件人:“xl价格区间价格”</p>
<p>收件人:“l-price rangePrice”</p>
<p>收件人:“m-price rangePrice”</p>
<pre class="lang-py prettyprint-override"><code>find("div", {"class": "xl-price rangePrice"})
</code></pre>
<p>如何更改此代码以便查询刮取“xl price rangePrice”、“l-price rangePrice”和“m-price rangePrice”</p>
<p>提前感谢您的回答</p>
<p>我的总体代码是:</p>
<pre class="lang-py prettyprint-override"><code>#Fonctionne jusqu à la page 5 mais j'ai pas la page 5
import pandas as pd
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from bs4 import BeautifulSoup
options = Options()
options.add_argument("window-size=1400,600")
from fake_useragent import UserAgent
ua = UserAgent()
a = ua.random
user_agent = ua.random
print(user_agent)
options.add_argument(f'user-agent={user_agent}')
driver = webdriver.Chrome('/Users/raduulea/Documents/chromedriver', options=options)
driver.get('https://www.immoweb.be/fr/recherche/immeuble-de-rapport/a-vendre/liege/4000')
import time
time.sleep(10)
Title = []
address = []
price = []
surface = []
desc = []
page = 2
while True:
time.sleep(10)
html = driver.page_source
soup = BeautifulSoup(html, 'html.parser')
if int(page)<=2:
results = soup.find_all("div", {"class": "result-xl"})
for result in results:
Title.append(result.find("div", {"class": "title-bar-left"}).get_text().strip())
address.append(result.find("span", {"result-adress"}).get_text().strip())
price.append(result.find("div", {"class": "xl-price rangePrice"}).get_text().strip())
surface.append(result.find("div", {"class": "xl-surface-ch"}).get_text().strip())
desc.append(result.find("div", {"class": "xl-desc"}).get_text().strip())
if len(driver.find_elements_by_css_selector("a.next")) > 0:
url = "https://www.immoweb.be/fr/recherche/immeuble-de-rapport/a-vendre/liege/4000/?page={}".format(page)
driver.get(url)
page += 1
elif 3 <= int(page) < 5:
results = soup.find_all("div", {"class": "result-l"})
for result in results:
Title.append(result.find("div", {"class": "title-bar-left"}).get_text().strip())
address.append(result.find("span", {"result-adress"}).get_text().strip())
price.append(result.find("div", {"class": "l-price rangePrice"}).get_text().strip())
surface.append(result.find("div", {"class": "l-surface-ch"}).get_text().strip())
desc.append(result.find("div", {"class": "l-desc"}).get_text().strip())
if len(driver.find_elements_by_css_selector("a.next")) > 0:
url = "https://www.immoweb.be/fr/recherche/immeuble-de-rapport/a-vendre/liege/4000/?page={}".format(page)
driver.get(url)
page += 1
elif 5 <= int(page) <= 6:
results = soup.find_all("div", {"class": "result-m"})
for result in results:
Title.append(result.find("div", {"class": "title-bar-left"}).get_text().strip())
address.append(result.find("span", {"result-adress"}).get_text().strip())
price.append(result.find("div", {"class": "m-price rangePrice"}).get_text().strip())
surface.append(result.find("div", {"class": "m-surface-ch"}).get_text().strip())
desc.append(result.find("div", {"class": "m-desc"}).get_text().strip())
if len(driver.find_elements_by_css_selector("a.next")) > 0:
url = "https://www.immoweb.be/fr/recherche/immeuble-de-rapport/a-vendre/liege/4000/?page={}".format(page)
driver.get(url)
page += 1
else:
break
elif int(page) > 6:
break
df = pd.DataFrame({"Title": Title, "Address": address, "Price:": price, "Surface": surface, "Description": desc})
df.to_csv("immo_scrap.csv")
</code></pre>
<p>如果有人需要它:
我找到了另一个解决方案,部分来自这里的答案,还查看了其他论坛:</p>
<p>下面是我的代码,现在简单多了:</p>
<pre class="lang-py prettyprint-override"><code>#Fonctionne jusqu à la page 5 mais j'ai pas la page 5
import pandas as pd
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from bs4 import BeautifulSoup
options = Options()
options.add_argument("window-size=1400,600")
from fake_useragent import UserAgent
ua = UserAgent()
a = ua.random
user_agent = ua.random
print(user_agent)
options.add_argument(f'user-agent={user_agent}')
driver = webdriver.Chrome('/Users/raduulea/Documents/chromedriver', options=options)
driver.get('https://www.immoweb.be/fr/recherche/immeuble-de-rapport/a-vendre/liege/4000')
import time
time.sleep(10)
Title = []
address = []
price = []
surface = []
desc = []
page = 2
while True:
time.sleep(10)
html = driver.page_source
soup = BeautifulSoup(html, 'html.parser')
if int(page)<=6:
results = soup.find_all(True, {"class": ["result-xl", "result-l","result-m"]})
for result in results:
Title.append(result.find("div", {"class":"title-bar-left"}).get_text().strip())
address.append(result.find("span", {"result-adress"}).get_text().strip())
price.append(result.find("div", {"class": ["xl-price rangePrice", "l-price rangePrice", "m-price rangePrice"]}).get_text().strip())
surface.append(result.find("div", {"class": ["xl-surface-ch", "l-surface-ch", "m-surface-ch"]}).get_text().strip())
desc.append(result.find("div", {"class": ["xl-desc", "l-desc", "m-desc"]}).get_text().strip())
if len(driver.find_elements_by_css_selector("a.next")) > 0:
url = "https://www.immoweb.be/fr/recherche/immeuble-de-rapport/a-vendre/liege/4000/?page={}".format(page)
driver.get(url)
page += 1
elif int(page) > 6:
break
df = pd.DataFrame({"Title": Title, "Address": address, "Price:": price, "Surface": surface, "Description": desc})
df.to_csv("immoweb_no_secret.csv")
</code></pre>