Selenium Python循环只获取第一项

Question

我很难得到全部的汽车清单。我只拿到每页的第一辆车。我喜欢字典的组织方式，但我愿意接受任何建议。下一个功能只需发送有效载荷。我想使用executemany（sqlCode，payload），但可能会使事情变得复杂 我希望能够

这是循环迭代的相关部分：

Form = driver.find_elements_by_xpath('//*[@id="compareForm"]')
cars = driver.find_elements_by_xpath('*//div[@class="description"]')
time.sleep(10)
for car in Form:

剩余o代码：

^{pr2}$

新增：

def scrape(driver):


    # for
    #get data from first pagebs
    cars = driver.find_elements_by_xpath('//*[@id="compareForm"]/div/div[2]/ul')
    print cars;
    time.sleep(10)
    for car in cars: #car.find_element_by_xpat('//li[contains(text(),"low to high")]h()')
        car_element = car.find_element_by_xpath(".//*[starts-with(@class, 'item')]")
        payload = [{
        "Year" : car_element.get_attribute("data-year"),
        "Make" :  car_element.get_attribute("data-make"),
        "Model" : car_element.get_attribute("data-model"),
        "Trim" : car_element.get_attribute("data-trim")
        "Engine" : car.get_attribute("data-engine"),
        "Transmission" : car.get_attribute("data-transmission"),
        "Exterior_Color" : car.get_attribute("data-exteriorcolor"),
        "Interior_Color" : car.get_attribute("data-interiorcolor"),
        "VIN" : car.get_attribute("data-vin"),
        "Type" : car.get_attribute("data-type"),
        # "Stock_No" : driver.find_element_by_xpath(".//dt[contains(text(),'Stock #:')]/following-sibling::dd").text,
        # "Mileage" : driver.find_element_by_xpath(".//dt[contains(text(),'Mileage:')]/following-sibling::dd").text,
        # "MPG_Range" : driver.find_element_by_xpath(".//dt[contains(text(),'MPG Range:')]/following-sibling::dd").text,
        # "Price" : driver.find_element_by_xpath(".//*[starts-with(@class, 'internetPrice')]/span[2]").text,
        # "classification" : driver.find_element_by_xpath(".//*[starts-with(@class, 'hproduct')]").get_attribute("data-classification")
        }]

        Final.append(payload)
        print Final;
    time.sleep(15)

谢谢，我已经做了调整，安德森 我在传递元素之前打印了这些元素，结果很成功：

selenium.webdriver.remote.webelement.WebElement (session="d5b2d93b4e8ad78be19a080cc34d548b", element="0.271760196680197-12")>, 

在

element=“0.271760196680197-13”）>；，

element=“0.271760196680197-14”）>；，

element=“0.271760196680197-15”）>；，

element=“0.271760196680197-16”）>；，

element=“0.271760196680197-17”）>；]

等等

由于某种原因，在第二次执行循环时，它找不到该项。在

这是HTML，我确保使用（.）以便从当前节点查看。在

我使用了元素，我希望遍历以item开头的

元素

cars = driver.find_elements_by_xpath('//*[@id="compareForm"]/div/div[2]/ul')
print cars;
time.sleep(10)
for car in cars: #car.find_element_by_xpat('//li[contains(text(),"low to high")]h()')
    car_element = car.find_element_by_xpath(".//*[starts-with(@class, 'item')]")
    payload = [{
    "Year" : car_element.get_attribute("data-year"),

这是HTML

<div class="hproduct auto volkswagen clearfix" data-classification="primary" data-type="certified" data-internetprice="MTI5MDMuMA==" data-msrp="MC4w" data-exteriorcolor="Reflex Silver Metallic" data-bodystyle="Sedan" data-trim="Auto SEL Pzev" data-year="2014" data-model="Jetta" data-vin="3VWL17AJ8EM394570" data-make="Volkswagen" data-index-position="7">
<div>
<div class="media">
<a href="/certified/Volkswagen/2014-Volkswagen-Jetta-afc6c4970a0e0adf279214712962aff7.htm">
<img class="photo thumb" src="https://pictures.dealer.com/ddc/resize/320x/quality/70/sharpen/1/ddc/c/carolinavolkswagenvw/0554/c9f0a2adc8cdb8a719d211c045fa1e6bx.jpg" data-src="https://pictures.dealer.com/ddc/resize/320x/quality/70/sharpen/1/ddc/c/carolinavolkswagenvw/0554/c9f0a2adc8cdb8a719d211c045fa1e6bx.jpg" alt="2014 Volkswagen Jetta Auto SEL Pzev Sedan" title="2014 Volkswagen Jetta Auto SEL Pzev Sedan">
</a>
<a class="dialog video ddc-btn ddc-btn-link ddc-btn-small" href="#" title="2014 Volkswagen Jetta Auto SEL Pzev Sedan" data-title=" 2014 Volkswagen Jetta Auto SEL Pzev Sedan" data-href="/carflix-popup.htm?hideVcard=false&amp;videoUrl=//videos2.dealer.com/clients/c/carolinavolkswagenvw/carflix2/afc6c4970a0e0adf279214712962aff7/dfbd71dd0a0a00f7116d789e7f2f80ae.xml&amp;videoHeight=400&amp;videoWidth=680&amp;autoPlay=true&amp;itemId=afc6c4970a0e0adf279214712962aff7&amp;category=AUTO" data-min-width="730" data-min-height="500" data-draggable="true" data-modal="false" data-destroy-on-close="true">
<i class="ddc-icon ddc-icon-video"></i>
Watch Video
</a>

</div>


<h3 class="fn  "><a class="url" href="/certified/Volkswagen/2014-Volkswagen-Jetta-afc6c4970a0e0adf279214712962aff7.htm"> 2014 Volkswagen Jetta Auto SEL Pzev Sedan</a></h3>
<div class="pricing-area has-buttons">
    <ul class="pricing  multiple-prices has-eprice list-unstyled">
        <li style="margin-bottom: 10px;"><span class="retailValue"><span class="label">Average Market Price<span class="separator">:</span></span><span class="value">$17,950</span></span></li>

        <li style="margin-bottom: 10px;"><span class="abSub"><span class="label">Savings<span class="separator">:</span></span><span class="value">$5,047</span></span></li>

        <li style="margin-bottom: 10px;"><span class="internetPrice final-price"><span class="label">Carolina VW Value Price*<span class="separator">:</span></span><span class="value">$12,903</span></span></li>

        <li style="margin-bottom: 10px;"><a data-href="/eprice-form.htm?itemId=afc6c4970a0e0adf279214712962aff7&amp;vehicleId=afc6c4970a0e0adf279214712962aff7&amp;category=AUTO&amp;parentPageAlias=INVENTORY_LISTING_DEFAULT_AUTO_ALL" class="dialog pointer epriceLink" data-eprice="true" data-title="Get Your ePrice"><img class="customEpriceImage" src="https://pictures.dealer.com/c/carolinavolkswagenvw/0001/25b1b5172b1c8523a5b83e0ec8b14cebx.jpg"></a></li>

        <li style="margin-bottom: 10px;">
        <a class="dialog pointer ddc-btn-block ui-button ui-widget ui-state-default ui-corner-all ui-button-text-only ddc-btn ddc-btn-primary sms-button custom3pcTextUsBtn" href="sms:+17042766637" data-make="Volkswagen" data-model="Jetta" data-vin="3VWL17AJ8EM394570" data-year="2014" data-status="Used" style="visibility: visible;" data-sdk-attached="true"><span class="ui-button-text">Text Us</span></a>
        </li>
    </ul>
</div>

<div class="description">
<dl> <dt>Engine:</dt> <dd>1.8L I-4 cyl<span class="separator">,</span></dd> <dt>Transmission:</dt> <dd>Automatic<span class="separator">,</span></dd> <dt>Mileage:</dt> <dd>45,084 miles<span class="separator">,</span></dd> <dt>MPG Range:</dt> <dd>25/36<span class="separator">,</span></dd></dl><dl class="last"> <dt>Exterior Color:</dt> <dd>Reflex Silver Metallic<span class="separator">,</span></dd> <dt>Interior Color:</dt> <dd>Titan Black<span class="separator">,</span></dd> <dt>Stock #:</dt> <dd>P3237</dd></dl> <span class="ddc-more">More<span class="hellip">…</span></span> 
<div class="calloutDetails">
<ul class="list-unstyled">
<li class="certified" style="margin-bottom: 10px;"><div class="badge "><img class="align-center" src="https://static.dealer.com/v8/global/images/franchise/white/en_US/logo-certified-volkswagen.gif?r=1356028132000" alt="Certified"></div></li><li class="carfax" style="margin-bottom: 10px;"><a href="http://www.carfax.com/cfm/ccc_displayhistoryrpt.cfm?partner=DLR_3&amp;vin=3VWL17AJ8EM394570" class="badge carfax-one-owner pointer" target="_blank"><img class="align-center" src="https://static.dealer.com/v8/global/images/franchise/white/logo-certified-carfax-one-owner-lrg.png?r=1405027620000" alt="Carfax One Owner"></a></li>
</ul>
</div>
<div class="hproductDynamicArea"></div>
</div>
</div>
<div class="mycars-vlp ddc-btn-mycars">
<a class="ddc-btn ddc-btn-default mycars-btn mycars-save-btn ddc-btn-mycars" href="#" data-id="afc6c4970a0e0adf279214712962aff7" data-mycars-save="">
Save This Car!
</a>
<a class="ddc-btn ddc-btn-default mycars-btn mycars-saved-btn mycars-action-set ddc-btn-mycars   hide" href="#" data-id="afc6c4970a0e0adf279214712962aff7" label-remove="Remove Car" label-set="Saved!" data-mycars-remove="">
Saved!
</a>
<a class="ddc-btn ddc-btn-default mycars-btn mycars-add-alert-btn ddc-btn-mycars" href="#" data-id="afc6c4970a0e0adf279214712962aff7" data-mycars-get-alerts="">
Get Price Alerts
</a>
<a class="ddc-btn ddc-btn-default mycars-btn mycars-alerts-isset mycars-action-set ddc-btn-mycars  hide" href="#" data-id="afc6c4970a0e0adf279214712962aff7" label-remove="Remove Alerts" label-set="Alert Set!">
Alert Set!
</a>
</div></div>

在

Answer 1

似乎第一个问题在您的循环中-您可能需要使用for car in cars而不是{}

第二个问题-您需要在XPath中指定对当前节点（.）的引用。在

简单的说这个

car.find_element_by_xpath("//*[starts-with(@class, 'item')]")

意味着从根html节点开始查找class属性值以"item"开头的元素的第一个出现位置

在这期间

^{pr2}$

表示从当前的class节点开始查找class属性值以"item"开头的元素的第一个出现位置