X路径Python错误“list”对象没有属性“xpath”

Question

我正在尝试做一个Web爬网项目，我不断得到一个错误代码，名为“'list'object has no attribute'xpath'”/

以下是目前为止的代码：

import requests 
from lxml import html
import pandas 

url = "http://www.cnbc.com/"
response = requests.get(url)
doc = html.fromstring(response.text)

headlineNode = doc.xpath('//div[@class="headline"]')
len(headlineNode)

headlines = headlineNode.xpath('/a')
len(headlines)

我想把CNBC主页上的头条新闻加起来，但我总是收到一个错误的说法

^{pr2}$

有什么办法解决这个问题吗？在

Answer 1

出现此错误的原因是xpath总是生成一个元素列表，并且可以对元素调用xpath。在

所以为了得到预期的结果，您可以在headlineNode列表上迭代，然后调用xpath到元素节点，比如node.xpath('/a')

下面的代码将为您提供所需的所有url的列表。你可以这样做：

import requests 
from lxml import html
import pandas 

url = "http://www.cnbc.com/"
response = requests.get(url)
doc = html.fromstring(response.text)

headlineNode = doc.xpath('//div[@class="headline"]')
len(headlineNode)

url_list  = []
for node in headlineNode : 
    url_node = node.xpath('./a/@href')
    if url_node : 
        url_list.append(url + url_node[0].strip())
print(url_list)
len(url_list)

这将导致：

^{pr2}$

若要同时向result_list添加标题，可以对其进行如下修改：

import requests 
from lxml import html
import pandas 

url = "http://www.cnbc.com/"
response = requests.get(url)
doc = html.fromstring(response.text)

headlineNode = doc.xpath('//div[@class="headline"]')
len(headlineNode)

result_list  = []
for node in headlineNode : 
    url_node = node.xpath('./a/@href')
    title = node.xpath('./a/text()')
    if (url_node and title) : 
        result_list.append({'url' : url + url_node[0].strip(),
            'title' : title[0].strip()})
print(result_list)
len(result_list)

这将导致：

[{'url': 'http://www.cnbc.com/2017/03/28/asia-markets-to-watch-brexit-news-dollar-movements-oil-prices.html', 'title': 'Asia up higher after Dow breaks 8-day losing streak, Brexit ahead'}, {'url': 'http://www.cnbc.com/2017/03/28/us-markets.html', 'title': 'Dow snaps 8-day losing streak as consumer confidence surges; financials lead'}, {'url': 'http://www.cnbc.com/2017/03/28/european-markets-investors-shrug-off-trump-concerns.html', 'title': 'European stocks close higher as investors shrug off Trump concerns; Wolseley up 5%'}, {'url': 'http://www.cnbc.com/2017/03/27/gold-steady-as-dollar-edges-up-focus-on-trump-agenda.html', 'title': 'Gold rises as investors focus on Trump agenda'}, {'url': 'http://www.cnbc.com/2017/03/27/weaker-dollar-lifts-oil-futures-but-soaring-us-output-weighs.html', 'title': 'US crude settles at $48.37, rising 64 cents with Libyan output disruption'}, {'url': 'http://www.cnbc.com/2017/03/28/us-treasurys-edge-lower-as-investors-eye-data-auctions.html', 'title': 'US Treasurys mostly higher as investors digest data, 5-year note auction'}, {'url': 'http://www.cnbc.com/2017/03/28/dollar-bounces-on-fed-talk-sterling-wobbles-on-brexit-anxiety.html', 'title': 'Dollar bounces on Fed talk, sterling wobbles on Brexit anxiety'}, {'url': 'http://www.cnbc.com/2017/03/28/dollar-bounces-on-fed-talk-sterling-wobbles-on-brexit-anxiety.html', 'title': 'Dollar bounces on Fed talk, sterling wobbles on Brexit anxiety'}, {'url': 'http://www.cnbc.com/2017/03/28/markets-could-see-heartbreak-if-brexit-is-hard-to-do.html', 'title': 'Markets could see heartbreak if Brexit is hard to do'}, {'url': 'http://www.cnbc.com/2017/03/28/former-google-exec-heres-how-to-know-its-time-to-fire-an-employee.html', 'title': "Former Google exec: Here's how to know it's time to fire an employee"}, {'url': 'http://www.cnbc.com/2017/03/28/chinas-secret-plan-to-crush-spacex-and-the-us-space-program.html', 'title': "China's secret plan to crush SpaceX and the US space program"}, {'url': 'http://www.cnbc.com/2017/03/28/le-pen-victory-could-be-five-times-as-dangerous-as-greeces-financial-meltdown-ubs.html', 'title': "Le Pen victory could be five times as dangerous as Greece's financial meltdown: UBS"}, {'url': 'http://www.cnbc.com/2017/03/28/nuclear-nonproliferation-lessons-from-myanmar.html', 'title': 'How this tiny country went from nuclear problem child to role model'}, {'url': 'http://www.cnbc.com/2017/03/28/following-tuesdays-rally-stocks-will-grind-higher-into-2018-analyst-says.html', 'title': "Following Tuesday's rally, stocks will 'grind higher' into 2018, analyst says"}, {'url': 'http://www.cnbc.com/2017/03/28/asia-markets-to-watch-brexit-news-dollar-movements-oil-prices.html', 'title': 'Asia up higher after Dow breaks 8-day losing streak, Brexit ahead'}, {'url': 'http://www.cnbc.com/2017/03/28/made-in-america-impossible-tech-startups.html', 'title': "These three start-ups show why it's so hard to build products in America"}, {'url': 'http://www.cnbc.com/2017/03/28/japans-tepid-retail-sales-raise-concerns-about-consumption-growth.html', 'title': "Japan's tepid retail sales raise concerns about consumption, growth"}, {'url': 'http://www.cnbc.com/2017/03/28/toshibas-westinghouse-to-file-for-us-bankruptcy-sources.html', 'title': "Toshiba's Westinghouse to file for US bankruptcy: Sources"}, {'url': 'http://www.cnbc.com/2017/03/28/early-reports-of-damage-in-wake-of-australias-cyclone-debbie.html', 'title': "Early reports of damage in wake of Australia's Cyclone Debbie"}, {'url': 'http://www.cnbc.com/2017/03/28/messaging-app-parent-tencent-takes-stake-in-tesla.html', 'title': 'Tesla deal boosts Chinese presence in US auto tech'}, {'url': 'http://www.cnbc.com/2017/03/28/rodrigo-duterte-turns-72-a-look-at-his-controversial-presidency-so-far-commentary.html', 'title': 'As Rodrigo Duterte turns 72, a look at his controversial presidency so far'}, {'url': 'http://www.cnbc.com/2017/03/28/north-korea-could-be-in-final-stages-of-nuclear-test-preparations-report.html', 'title': 'North Korea could be in final stages of nuclear test preparations: Report'}, {'url': 'http://www.cnbc.com/2017/03/28/congress-clears-way-for-isps-to-sell-browsing-history.html', 'title': 'Congress just cleared the way for internet providers to sell your web browsing history'}, {'url': 'http://www.cnbc.com/2017/03/28/british-pm-may-to-fire-starting-gun-on-brexit.html', 'title': 'British PM May to fire starting gun on Brexit'}, {'url': 'http://www.cnbc.com/2017/03/28/brexit-is-like-arthritis-it-doesnt-kill-you-but-it-hits-you-says-former-boe-member.html', 'title': u'Brexit is like arthritis, it doesn\u2019t kill you but it hits you, says former BOE member'}, {'url': 'http://www.cnbc.com/2017/03/27/florida-preparesfor-xi-jinpingsvisit.html', 'title': u"Florida authorities prep\xa0for Xi Jinping's\xa0visit"}, {'url': 'http://www.cnbc.com/2017/03/28/us-heading-for-recession-after-2-years-of-unsustainable-growth-economist-says.html', 'title': 'US heading for recession after 2 years of unsustainable growth, economist says'}, {'url': 'http://www.cnbc.com/2017/03/28/this-is-what-immigration-means-to-the-us-economy-in-two-charts.html', 'title': 'This is what immigration means to the U.S. economy in two charts'}, {'url': 'http://www.cnbc.com/2017/03/28/jack-dorsey-twitter-ceo-calls-to-step-down.html', 'title': u'Jack Dorsey brushes off calls to quit as CEO, says will do \u2018whatever it takes\u2019 for Twitter, Square to succeed'}, {'url': 'http://www.cnbc.com/2017/03/28/next-iphone-is-last-hurrah-for-smartphones-apple.html', 'title': 'The next iPhone is the last hurrah for smartphones: Fund manager'}, {'url': 'http://www.cnbc.com/2017/03/28/south-africa-speculates-over-possible-firing-of-finance-minister.html', 'title': 'South Africa faces uncertain wait amid speculation finance minister is to be fired'}, {'url': 'http://www.cnbc.com/2017/03/28/snap-was-a-drunk-at-the-wheel-but-now-looks-like-a-very-real-business-fund-manager-says.html', 'title': "Snap was a 'drunk at the wheel' but now looks like a very real business, fund manager says"}, {'url': 'http://www.cnbc.com/2017/03/28/this-ad-agency-is-rebranding-with-the-name-of-its-jewish-founders.html', 'title': 'This ad agency is standing against xenophobia and rebranding with the name of its Jewish founders'}, {'url': 'http://www.cnbc.com/2017/03/28/siemens-medical-solutions-wins-41-billion-us-defense-contract.html', 'title': 'Siemens Medical Solutions wins $4.1 billion US defense contract'}, {'url': 'http://www.cnbc.com/2017/03/28/google-home-ai-speaker-is-coming-to-the-uk-to-challenge-amazon-echo.html', 'title': 'Google Home A.I. speaker is coming to the UK to challenge Amazon Echo'}, {'url': 'http://www.cnbc.com/2017/03/28/jack-dorsey-twitter-ai-impact-jobs-mnuchin-government.html', 'title': u'Is the US government ready for the impact of AI? \u2018I don\u2019t know,\u2019 Twitter CEO Jack Dorsey says'}, {'url': 'http://www.cnbc.com/2017/03/28/uk-introduces-super-secure-new-one-pound-coin.html', 'title': u'UK introduces super secure new \xa31 coin'}, {'url': 'http://www.cnbc.com/2017/03/28/major-solar-project-gets-the-green-light-in-northern-ireland.html', 'title': "'Major' solar project gets the green light in Northern Ireland"}, {'url': 'http://www.cnbc.com/2017/03/28/trump-to-meet-egyptian-president-at-white-house-on-april-3.html', 'title': 'Trump to meet Egyptian president at White House on April 3'}, {'url': 'http://www.cnbc.com/2017/03/28/tesla-is-rejecting-a-racial-harassment-lawsuit-filed-by-an-assembly-line-worker.html', 'title': 'Tesla rejects racial harassment lawsuit filed by assembly line worker'}, {'url': 'http://www.cnbc.com/2017/03/28/jpmorgan-says-markets-are-underestimating-the-upside-of-asian-banks.html', 'title': 'JPMorgan says markets are underestimating the upside of Asian banks'}, {'url': 'http://www.cnbc.com/https://www.facebook.com/cnbcinternational/', 'title': 'Facebook'}, {'url': 'http://www.cnbc.com/https://twitter.com/cnbcworld', 'title': 'Twitter'}, {'url': 'http://www.cnbc.com/http://www.linkedin.com/company/2966126', 'title': 'LinkedIn'}, {'url': 'http://www.cnbc.com/http://www.youtube.com/user/CNBCInternational', 'title': 'Youtube'}, {'url': 'http://www.cnbc.com/http://www.cnbc.com/id/28295763', 'title': 'RSS'}, {'url': 'http://www.cnbc.com/pre-markets/', 'title': 'Pre-markets'}, {'url': 'http://www.cnbc.com/http://www.cnbc.com/world-markets-heat-map/', 'title': 'World stock markets heat map'}, {'url': 'http://www.cnbc.com/dow-components/', 'title': 'Dow 30: Real-time quotes'}, {'url': 'http://www.cnbc.com/http://data.cnbc.com/quotes/aapl%2Caxp%2Cba%2Ccat%2Ccsco%2Ccvx%2Cdd%2Cdis%2Cge%2Cgs%2Chd%2Cibm%2Cintc%2Cjnj%2Cjpm%2Cko%2Cmcd%2Cmmm%2Cmrk%2Cmsft%2Cnke%2Cpfe%2Cpg%2Ctrv%2Cunh%2Cutx%2Cv%2Cvz%2Cwmt%2Cxom', 'title': 'Dow 30: After-hours quotes'}, {'url': 'http://www.cnbc.com/sectors/', 'title': 'Sector watch'}, {'url': 'http://www.cnbc.com/2017/03/28/dollar-bounces-on-fed-talk-sterling-wobbles-on-brexit-anxiety.html', 'title': 'Dollar bounces on Fed talk, sterling wobbles on Brexit anxiety'}, {'url': 'http://www.cnbc.com/2017/03/28/le-pen-victory-could-be-five-times-as-dangerous-as-greeces-financial-meltdown-ubs.html', 'title': "Le Pen victory could be five times as perilous as Greece's financial crisis: UBS"}, {'url': 'http://www.cnbc.com/2017/03/28/iran-says-russia-can-use-its-military-bases-on-case-by-case-basis.html', 'title': "Iran says Russia can use its military bases 'on case by case basis'"}, {'url': 'http://www.cnbc.com/2017/03/28/uk-introduces-super-secure-new-one-pound-coin.html', 'title': u'UK introduces super secure new \xa31 coin'}, {'url': 'http://www.cnbc.com/2017/03/28/google-home-ai-speaker-is-coming-to-the-uk-to-challenge-amazon-echo.html', 'title': 'Google Home A.I. speaker is coming to the UK to challenge Amazon Echo'}, {'url': 'http://www.cnbc.com/2017/03/28/nuclear-nonproliferation-lessons-from-myanmar.html', 'title': 'Nuclear nonproliferation: Lessons from Myanmar'}, {'url': 'http://www.cnbc.com/2017/03/28/north-korea-could-be-in-final-stages-of-nuclear-test-preparations-report.html', 'title': 'North Korea could be in final stages of nuclear test preparations: Report'}, {'url': 'http://www.cnbc.com/2017/03/28/hootsuite-ceo-talks-social-media-and-trump.html', 'title': 'Companies need top-down approach to social media: Hootsuite CEO'}, {'url': 'http://www.cnbc.com/2017/03/28/messaging-app-parent-tencent-takes-stake-in-tesla.html', 'title': 'Tesla deal boosts Chinese presence in US auto tech'}, {'url': 'http://www.cnbc.com/2017/03/28/dow-trump-dollar-kushner-stoxx-wallstreet.html', 'title': 'Reaction: Global markets stepped past Trump wobble on Obamacare'}, {'url': 'http://www.cnbc.com/2017/03/27/trump-brexit-healthcare-russia-dow-stoxx.html', 'title': 'Reaction: Risk-off hits markets after Trump fails to pass Obamacare hurdle'}, {'url': 'http://www.cnbc.com/2017/03/24/trump-healthcare-markets-us-dow-stocks.html', 'title': 'Reaction: Wall Street stocks move higher as investors countdown to health-care vote'}, {'url': 'http://www.cnbc.com/2017/03/23/trump-london-terrorism-markets-stocks-dow-ftse-may-uk-us.html', 'title': 'Reaction: London police name attacker in Westminster terrorist incident'}, {'url': 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000603723', 'title': "Weekly Agenda: From EU visas to 'Fearless Girl'"}, {'url': 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000602217', 'title': 'Weekly Agenda: From the Canadian budget to the Boao Forum'}, {'url': 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000600643', 'title': "Weekly Agenda: From Merkel's US visit to the FOMC meeting"}, {'url': 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000598102', 'title': "Weekly Agenda: From 'mini' EU summit to US daylight saving"}, {'url': 'http://www.cnbc.com/2017/03/26/trader-poll-which-will-garner-the-most-market-attention.html', 'title': 'Trader poll: Which will garner the most market attention?'}, {'url': 'http://www.cnbc.com/2017/03/20/trader-poll-what-are-chinas-greatest-challenges.html', 'title': u'Trader poll: What are China\u2019s greatest challenges?'}, {'url': 'http://www.cnbc.com/2017/03/12/trader-poll-is-the-fed-definitely-going-to-hike-rates-at-this-meeting.html', 'title': 'Trader Poll: Is the Fed definitely going to hike rates at this meeting?'}, {'url': 'http://www.cnbc.com/2017/03/06/trader-poll-has-the-us-dollar-got-higher-to-go.html', 'title': 'Trader poll: Has the US dollar got higher to go?'}, {'url': 'http://www.cnbc.com/2017/03/28/this-ad-agency-is-rebranding-with-the-name-of-its-jewish-founders.html', 'title': 'This ad agency is standing against xenophobia; using name of its Jewish founders'}, {'url': 'http://www.cnbc.com/advertorial/2017/03/09/breathing-new-life-into-brands.html', 'title': 'Breathing new life into brands'}, {'url': 'http://www.cnbc.com/2017/03/23/make-sure-youre-not-replaced-by-a-machine-microsoft-us-chief-marketer.html', 'title': "Make sure you're not replaced by a machine: Microsoft US chief marketer"}, {'url': 'http://www.cnbc.com/2017/03/23/online-ads-need-to-stop-being-intrusive-eurosport.html', 'title': 'Online ads need to stop being intrusive, especially around 2018 Olympics: Eurosport MD'}, {'url': 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000595689', 'title': 'Why naming a product is so important'}, {'url': 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000595687', 'title': u'\u2018Building up a sofa\u2019 that\u2019s fits customer needs: Matthew\u2019s brand proposal'}, {'url': 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000595688', 'title': u'A men\u2019s \u2018grooming brand with an organic twist\u2019: Connor\u2019s proposal'}, {'url': 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000595690', 'title': 'Why passion is the true driver of a successful entrepreneur'}, {'url': 'http://www.cnbc.com/2017/03/28/major-solar-project-gets-the-green-light-in-northern-ireland.html', 'title': "'Major' solar project gets the green light in Northern Ireland"}, {'url': 'http://www.cnbc.com/2017/03/23/engineers-develop-solar-powered-skin-which-could-transform-lives.html', 'title': "Engineers develop solar powered 'skin' which could transform lives"}, {'url': 'http://www.cnbc.com/2017/03/23/palm-oil-its-uses-and-ways-to-limit-its-impact.html', 'title': 'Palm oil: Its uses and ways to limit its impact'}, {'url': 'http://www.cnbc.com/advertorial/2017/03/13/access-to-energy-when-business-issues-meet-social-impacts.html', 'title': 'Access to energy: When business issues meet social impacts'}, {'url': 'http://www.cnbc.com/2017/03/28/jack-dorsey-twitter-ceo-calls-to-step-down.html', 'title': u"Jack Dorsey: I'll do \u2018whatever it takes\u2019 for Twitter, Square to succeed"}, {'url': 'http://www.cnbc.com/2017/03/28/jack-dorsey-twitter-ai-impact-jobs-mnuchin-government.html', 'title': u'Is the US government ready for the impact of AI? \u2018I don\u2019t know,\u2019 Twitter CEO Jack Dorsey says'}, {'url': 'http://www.cnbc.com/2017/03/28/square-uk-launch-ceo-jack-dorsey-ai-acquisitions.html', 'title': 'Square launches in UK, CEO Jack Dorsey says on the lookout for AI firms to acquire'}, {'url': 'http://www.cnbc.com/2017/03/28/amazon-to-acquire-souqcom-a-large-e-commerce-site-in-the-middle-east.html', 'title': 'Amazon reaches deal to acquire Middle East e-commerce site Souq.com'}, {'url': 'http://www.cnbc.com/advertorial/2016/12/13/a-new-era-of-intelligent-machinery.html', 'title': 'A new era of intelligent machinery'}, {'url': 'http://www.cnbc.com/advertorial/2016/12/05/the-reimagining-of-cities.html', 'title': 'The reimagining of cities'}, {'url': 'http://www.cnbc.com/advertorial/2016/11/22/could-mining-asteroids-save-earth.html', 'title': 'Could mining asteroids save Earth?'}, {'url': 'http://www.cnbc.com/advertorial/2016/11/02/smarter-ships-safer-seas.html', 'title': 'At sea, every hour is rush hour. Are autonomous ships the solution?'}, {'url': 'http://www.cnbc.com/2017/03/28/jack-dorsey-twitter-ai-impact-jobs-mnuchin-government.html', 'title': u'Is the US government ready for the impact of AI? \u2018I don\u2019t know,\u2019 Twitter CEO says'}]

您可以参考scraping和lxml的文档以了解更多信息。在