用python搜索网站文章中的关键词

Question

我刚加入你们，从我开始学习python到现在已经一个月了。我想用Python（http://aeconf.com/may2013.htm）搜索这个站点的关键字。你知道吗

通常我手动点击视图abtsract并搜索“关键字：”后面的单词。如何使用python自动执行此操作？我很抱歉我的英语不好

Answer 1

你应该看看Selenium

pip install selenium

我提供了一个示例代码，它可以做什么，你应该测试一下。你知道吗

示例代码：

from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException


caps = DesiredCapabilities().CHROME
caps["pageLoadStrategy"] = "normal"  
driver = webdriver.Chrome(desired_capabilities=caps, executable_path=r'C:\Users\My-PC-Name\AppData\Local\Programs\Python\Python37-32\Scripts\chromedriver.exe')


url = "http://aeconf.com/may2000.htm" #Link
driver.get(url)
links = [x.get_attribute('href') for x in driver.find_elements_by_link_text('View Abstract')]
htmls = []

for link in links:
   driver.get(link)
   Keyword = [y.text for y in driver.find_elements_by_xpath("//font[2]/span[@style = 'mso-bidi-font-size: 1.0pt']")]
   if not Keyword: #If link is a dead link
     continue
   print(Keyword[0])
   htmls.append(driver.page_source)

在本例中，我将url改为http://aeconf.com/may2000.htm我提供的代码主要获取您需要的“关键字”，但在某些情况下，“关键字”的索引位置会根据所述url中的链接而变化。你知道吗

“已更改”链接的输出：

Fiscal decentralization; Corruption; Tax evasion.
Incentive mechanism design; Walrasian allocations; Implementation.
Debt and equity flows; Asymmetric information; Bankruptcy cost; Market failures; 
Corrective taxation.
Transitory volatility; Price formation; Exogenous liquidity demand.
Investment horizon; Beta; Size; Book-to-market equity; CAPM.
G11, G13.                         #At This part you can see that the 'Key Words' printed are not correct
Portfolio constraints; Stochastic income; Relaxation-projection methods.
Foreign aid; Foreign borrowing; Capital accumulation.
Entrepreneurial ability; Asymmetric information; Liquidity constraints.
Contract; Human capital; Labor.
Endogenous structure of the division of labor; Dual economy; Endogenous trade policy regime.

如果我们将我的示例代码中的'url'变量更改为原始的link，那么即使第一个链接是死链接，索引位置也会发生更改。作为一个挑战，我会让你自己解决的：-） 有更多的模块可以像Selenium一样做同样的事情。希望这能让你对浏览器自动化，网页抓取和更多（网络爬虫等）更感兴趣。你知道吗

只是一个提示（可能不是提示）您只需更改'Keyword'变量的索引位置即可获得所需的“Keyword”。