Python中文
首页
教程
问答
标签
搜索
登录
注册
如何使用selenium刮取整个页面?
回答此问题可获得
20
贡献值,回答如果被采纳可获得
50
分。
<p>我的目标是能够读取一吨div中的某个嵌套深度。唯一的问题是,它们似乎依赖于javascript,所以据我所知,仅使用driver.page_source无法获得它们</p> <p>这是我的密码:</p> <pre><code>import requests # for making standard html requests from bs4 import BeautifulSoup # magical tool for parsing html data import json # for parsing data from pandas import DataFrame as df # premier library for data organization import time import lxml from selenium import webdriver from webdriver_manager.chrome import ChromeDriverManager url = "https://www.challengermode.com/dota2/tournaments?state=upcoming" driver = webdriver.Chrome(ChromeDriverManager().install()) driver.get(url) time.sleep(5) # To let the page load in soup_ID = BeautifulSoup(driver.page_source, 'html.parser') print(soup_ID.prettify) </code></pre> <p><a href="https://i.stack.imgur.com/pqt9i.jpg" rel="nofollow noreferrer">Here is an image of the span of information I want to be included in the print</a></p> <p>这是我的输出:</p> <pre><code><bound method Tag.prettify of <html class="arena-html mod_flexbox mod_flexwrap mod_cssscrollbar mod_eventlistener mod_scriptasync mod_localstorage mod_sessionstorage mod_websockets mod_eventsource" id="html" lang="en" style="margin: 0px; padding: 0px;"><head> <base href="/"/> <link href="https://fonts.googleapis.com/css?family=Roboto:300,400,400i,500,700&amp;display=swap" rel="stylesheet"/> <link as="style" href="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/light.43d62e718e19239b66ac.css" rel="preload"/> <link href="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/light.43d62e718e19239b66ac.css" media="all" onload="this.media='all'" rel="stylesheet"/> <noscript><link href="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/light.43d62e718e19239b66ac.css" rel="stylesheet"/></noscript> <link as="style" href="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/arena-paypal.26f2c9c2acd9b96ba93b.css" rel="preload"/> <link href="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/arena-paypal.26f2c9c2acd9b96ba93b.css" media="all" onload="this.media='all'" rel="stylesheet"/> <noscript><link href="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/arena-paypal.26f2c9c2acd9b96ba93b.css" rel="stylesheet"/></noscript> <script async="" src="https://widget.intercom.io/widget/yxk7m4ye" type="text/javascript"></script><script async="" src="https://www.google-analytics.com/gtm/js?id=GTM-MHVMG4G&amp;t=gtag_UA_63855440_1&amp;cid=2113228608.1596037460" type="text/javascript"></script><script async="" src="https://www.google-analytics.com/plugins/ua/linkid.js" type="text/javascript"></script><script async="" src="https://www.googleadservices.com/pagead/conversion_async.js" type="text/javascript"></script><script async="" src="https://connect.facebook.net/signals/config/1363905500304531?v=2.9.22&amp;r=stable"></script><script async="" crossorigin="anonymous" src="https://connect.facebook.net/en_US/sdk.js?hash=4c7217325ae946d41396c9d017814623&amp;ua=modern_es6"></script><script async="" src="https://www.googletagmanager.com/gtag/js?id=AW-969263990&amp;l=dataLayer&amp;cx=c" type="text/javascript"></script><script async="" src="https://www.google-analytics.com/analytics.js" type="text/javascript"></script><script async="" src="https://www.gstatic.com/recaptcha/releases/AFBwIe6h0oOL7MOVu88LHld-/recaptcha__en.js" type="text/javascript"></script><script id="facebook-jssdk" src="//connect.facebook.net/en_US/sdk.js"></script><script async="" src="https://connect.facebook.net/en_US/fbevents.js"></script><script async="true" crossorigin="anonymous" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/manifest.2aa5da30056e9cc4eae7.bundle.js"></script> <title>Dota 2 Tournaments | Challengermode</title> <meta charset="utf-8"/> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"/> <meta content="width=750, user-scalable=no" name="viewport"/> <meta content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" name="viewport"/> <link href="/pwa-manifest.json" rel="manifest"/> <link href="/opensearch" rel="search" title="Challengermode" type="application/opensearchdescription+xml"/> <meta content="#252730" name="theme-color"/> <meta content="#252730" name="msapplication-navbutton-color"/> <meta content="yes" name="apple-mobile-web-app-capable"/> <meta content="black-translucent" name="apple-mobile-web-app-status-bar-style"/> <meta content="Challengermode" name="apple-mobile-web-app-title"/> <link href="https://challengermode-permanent-assets.azureedge.net/app/cm-192-logo.png" rel="apple-touch-icon" sizes="192x192"/> <link href="https://challengermode-permanent-assets.azureedge.net/app/cm-512-logo.png" rel="apple-touch-icon" sizes="512x512"/> <link href="https://challengermode-permanent-assets.azureedge.net/app/splashscreens/iphone6_splash.png" rel="apple-touch-startup-image"/> <link href="https://challengermode-permanent-assets.azureedge.net/app/splashscreens/iphonex_splash.png" media="(device-width: 375px) and (device-height: 812px) and (-webkit-device-pixel-ratio: 3)" rel="apple-touch-startup-image"/> <link href="https://challengermode-permanent-assets.azureedge.net/app/splashscreens/iphone6_splash.png" media="(device-width: 375px) and (device-height: 667px) and (-webkit-device-pixel-ratio: 2)" rel="apple-touch-startup-image"/> <link href="https://challengermode-permanent-assets.azureedge.net/app/splashscreens/iphoneplus_splash.png" media="(device-width: 414px) and (device-height: 736px) and (-webkit-device-pixel-ratio: 3)" rel="apple-touch-startup-image"/> <link href="https://challengermode-permanent-assets.azureedge.net/app/splashscreens/iphone5_splash.png" media="(device-width: 320px) and (device-height: 568px) and (-webkit-device-pixel-ratio: 2)" rel="apple-touch-startup-image"/> <link href="https://www.challengermode.com/tournaments/feed" rel="alternate" type="application/atom+xml"/> <link href="https://www.challengermode.com/spaces/feed" rel="alternate" type="application/atom+xml"/> <link href="https://www.challengermode.com/classifieds/feed" rel="alternate" type="application/atom+xml"/> <meta content="Leading platform for Dota 2 esports tournaments. Compete in quality tournaments from the best organizers or create your own space &amp; monetize your community." name="description"/> <meta content="challengermode esports competitions tournaments leagues skills solo team organize host lol league of legends csgo counter-strike: global offensive pubg playerunknowns battlegrounds dota 2 teamfight tactics tft valorant" name="keywords"/> <meta content="index,follow" name="robots"/> <meta content="English" name="language"/> <link href="https://www.challengermode.com/dota2/tournaments?state=upcoming" rel="canonical"/> <link href="https://api.challengermode.com" rel="dns-prefetch"/> <link crossorigin="" href="https://api.challengermode.com" rel="preconnect"/> <link href="https://syndication.twitter.com" rel="preconnect"/> <link href="https://widget.intercom.io" rel="preconnect"/> <link href="https://js.intercomcdn.com" rel="preconnect"/> <link href="https://www.facebook.com" rel="preconnect"/> <link crossorigin="" href="https://connect.facebook.net" rel="preconnect"/> <link href="https://api-iam.intercom.io" rel="preconnect"/> <link crossorigin="" href="https://fonts.gstatic.com" rel="preconnect"/> <link href="https://az416426.vo.msecnd.net" rel="preconnect"/> <link href="https://stats.g.doubleclick.net" rel="preconnect"/> <link crossorigin="" href="https://fonts.googleapis.com" rel="preconnect"/> <link href="https://dc.services.visualstudio.com" rel="preconnect"/> <meta content="https://www.challengermode.com/dota2/tournaments?state=upcoming" property="og:url"/> <meta content="Dota 2 Tournaments" property="og:title"/> <meta content="Leading platform for Dota 2 esports tournaments. Compete in quality tournaments from the best organizers or create your own space &amp; monetize your community." property="og:description"/> <meta content="https://challengermode-permanent-assets.azureedge.net/app/og_image.png" property="og:image"/> <meta content="image/png" property="og:image:type"/> <meta content="1200" property="og:image:width"/> <meta content="630" property="og:image:height"/> <meta content="website" property="og:type"/> <meta content="Challengermode" property="og:site_name"/> <meta content="cm:game_info_slug:f52a42ce-3425-4dca-ab1d-e425ea1e71ea" property="og:cm_resource"/> <meta content="3625f24494c7ac4f0ad3" name="wot-verification"/> <meta content="1179483245396310" property="fb:app_id"/> <style> body::after { content: "none"; display: none !important } @media (max-width:1920px) { body::after { content: "breakpoint--full-hd" } } @media (max-width:1280px) { body::after { content: "breakpoint--hd" } } @media (max-width:1024px) { body::after { content: "breakpoint--tablet" } } @media (max-width:414px) { body::after { content: "breakpoint--mobile" } } </style> <script src="//az416426.vo.msecnd.net/scripts/a/ai.0.js"></script><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/0.1d1eb0a321bfe9aa47ee.bundle.js"></script><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/1.97217bf357c5de4a751a.bundle.js"></script><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/2.3240916b8c45c6c77a5b.bundle.js"></script><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/3.966cc108df5a7515bf50.bundle.js"></script><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/7.ed08c498b552166708b9.bundle.js"></script><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/175.f6ae048c521d527a8f53.bundle.js"></script><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/282.a0ab5b4c130061ae89b3.bundle.js"></script><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/323.86bf89e818dd1c06cf21.bundle.js"></script><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/337.c989accb4d8622d946e5.bundle.js"></script><style data-emotion=""></style><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/5.da829e90054bb31c6591.bundle.js"></script><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/4.fc75798185acc24a996a.bundle.js"></script><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/6.ba3b4ef40d494de88ed8.bundle.js"></script><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/8.0a8441153a17e1c20931.bundle.js"></script><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/9.92e08e43b5aeab83b11a.bundle.js"></script><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/11.75d6926838e4e7c55f20.bundle.js"></script><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/17.0c42d6a55e624fc36e4c.bundle.js"></script><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/51.79196085aeb507e3486e.bundle.js"></script><link href="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/10.5df7cf3cfa886d3230a3.css" rel="stylesheet" type="text/css"/><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/10.8c3b8aef15bdf341e192.bundle.js"></script><link href="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/13.8ddd5b6f8bfee769c14a.css" rel="stylesheet" type="text/css"/><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/13.d20ed356ddb838ab76ce.bundle.js"></script><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/16.9da04cea0e07cef002f4.bundle.js"></script><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/22.13bf9d744401ea38a0bd.bundle.js"></script><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/30.803fc5a3967c13785bb5.bundle.js"></script><link href="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/71.ab772642f9c8624e736d.css" rel="stylesheet" type="text/css"/><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/71.e7da16d37e16b62bf79b.bundle.js"></script><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/158.818c18197b42c18410d9.bundle.js"></script><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/284.8b5c95597f8814f01390.bundle.js"></script><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/12.404ebfb3d2a9e09d5abc.bundle.js"></script><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/75.9fddbb16d492adbd2ab5.bundle.js"></script><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/161.1c91a2c2545bb21d7e20.bundle.js"></script><script src="https://googleads.g.doubleclick.net/pagead/viewthroughconversion/969263990/?random=1596037459961&amp;cv=9&amp;fst=1596037459961&amp;num=1&amp;bg=ffffff&amp;guid=ON&amp;resp=GooglemKTybQhCsO&amp;u_h=1080&amp;u_w=1920&amp;u_ah=1080&amp;u_aw=1920&amp;u_cd=24&amp;u_his=2&amp;u_tz=120&amp;u_java=false&amp;u_nplug=3&amp;u_nmime=4&amp;gtm=2oa7m1&amp;sendb=1&amp;ig=1&amp;data=event%3Dpage_view&amp;frm=0&amp;url=https%3A%2F%2Fwww.challengermode.com%2Fdota2%2Ftournaments%3Fstate%3Dupcoming&amp;tiba=Dota%202%20Tournaments%20%7C%20Challengermode&amp;hn=www.googleadservices.com&amp;async=1&amp;rfmt=3&amp;fmt=4"></script><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/14.eb76c66c32e99864e5ad.bundle.js"></script><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedge.net/dist2/15.1379135acdc99c059dcd.bundle.js"></script><script charset="utf-8" src="https://cmp-edge-webapp-cdn2.azureedg </code></pre> <p>所需的输出将在输出中显示以蓝色和红色标记的<a href="https://i.stack.imgur.com/jdmKW.png" rel="nofollow noreferrer">all the source code</a></p> <p>哦,如果你有任何问题或需要更多信息,我很乐意提供</p>
0 条评论
分类:
Python问答
请先
登录
后评论
默认排序
时间排序
1 个回答
匿名
1天前
擅长:python、mysql、java
<p>我找到了一个解决办法。我似乎无法打印所有内容,但它仍在存储中。所以,如果我使用driver.find_element_by_class_name(“link white”),它非常适合我的目标</p>
请先
登录
后评论
针对此问题:
更多的回答
关注
89
关注
收藏
1
收藏,
216
浏览
网友 提问于 2天前
相关Python问题
当用户用PYTHON设置一个或一个不带值的URL时,他们怎么能输入一个/a的代码呢?
9 回答
当用户登录到站点时,如何显示不同的导航栏
8 回答
当用户登录时,在Flask中向用户显示处理结果
8 回答
当用户的Flask会话结束时,我如何从Redis后端中移除所有Celery结果?
3 回答
当用户的Okta配置文件字段当前为blan时,更新该字段
6 回答
当用户的付款逾期2天时,从Django模型检索数据
7 回答
当用户的消息以问号结尾时,如何让机器人说些什么?
8 回答
当用户的系统上可能也安装了Python 2.7时,如何在用户的系统上运行Python 3脚本?
8 回答
当用户确定打印数量时,使用Matplotlib打印动画
2 回答
当用户离开时是否可以删除整个网页?
2 回答
当用户给出一个单词时如何打印?
6 回答
当用户继续更改TKin中的值(使用trace方法)时,使用Entry并更新输入的条目
8 回答
当用户编辑表单字段时,从Django时间字段中删除秒数
9 回答
当用户被更改时,消息不会来自web套接字
4 回答
当用户访问表单时,如何使表单为只读,而不具有更改权限
6 回答
当用户试图更改对象的值时,使用描述符类引发RuntimeError
9 回答
当用户调整GUI的大小时,是否有方法更改GUI内容的大小?
4 回答
当用户调整风的大小时,pythontkinter小部件的大小会不均匀
6 回答
当用户购买某个类别时,是否查找其他类别的销售?
10 回答
当用户转到上一页时,Django和芹菜插入操作
2 回答