我想浏览网页，但是找不到html代码

Question

你好，我是刮削网站的beginer。我希望有人能帮助我。 我想从https://www.bhinneka.com/jual?cari=iphone上搜刮有关物品销售的网站，当我分析它们时，找不到价格

我将代码写在下面：

import bs4
import requests
url = 'http://www.bhinneka.com/jual?cari=iphone'
r = requests.get(url)
soup = BeautifulSoup(r, "html.parser")
produk = soup.findAll('div', {'class' : 'product-title'})

非常感谢你帮助我

Answer 1

数据通过Ajax从外部URL加载。您可以使用requests模块来加载它，例如：

import json
import requests

api_url = "https://api.bhinneka.com/gws/graphql"

payload = {
    "operationName": "getSearchPageData",
    "query": 'query getSearchPageData($keyword: String, $page: Int, $range: String, $sort: String, $filter: String) {\n  search(\n    include: "category,brand,price,shippingMethod,merchant,storeCity"\n    keyword: $keyword\n    page: $page\n    limit: 40\n    filter: $filter\n    range: $range\n    aggs: "variants.activePrice"\n    sort: $sort\n    exclude: ""\n  ) {\n    code\n    success\n    message\n    meta {\n      took\n      totalRecords\n      totalPages\n      limit\n      page\n      aggregation {\n        price {\n          min\n          max\n          __typename\n        }\n        brand {\n          id\n          name\n          slug\n          count: docCount\n          __typename\n        }\n        intervalPrice {\n          start\n          count: docCount\n          __typename\n        }\n        shipping: shippingMethod {\n          code\n          name\n          count: docCount\n          __typename\n        }\n        location: storeCity {\n          id\n          name\n          count: docCount\n          __typename\n        }\n        categories: category {\n          ...FilterCategory\n          children: child {\n            ...FilterCategory\n            children: child {\n              ...FilterCategory\n              children: child {\n                ...FilterCategory\n                __typename\n              }\n              __typename\n            }\n            __typename\n          }\n          __typename\n        }\n        merchant {\n          id\n          name\n          count: docCount\n          __typename\n        }\n        __typename\n      }\n      __typename\n    }\n    result {\n      merchant {\n        name\n        vanityUrl\n        storeDistrict {\n          name\n          __typename\n        }\n        __typename\n      }\n      description\n      isPreOrder\n      durationPreOrder\n      productWarranty\n      isSold\n      name\n      image {\n        small\n        thumbnail\n        __typename\n      }\n      media {\n        variant {\n          name\n          thumbnail\n          small\n          __typename\n        }\n        __typename\n      }\n      brand {\n        id\n        name\n        __typename\n      }\n      category {\n        id\n        name\n        __typename\n      }\n      variants {\n        promoPrice {\n          percentage\n          nominal\n          specialPrice\n          startDate\n          endDate\n          __typename\n        }\n        id\n        skuInternal\n        name\n        fullName\n        slug\n        nameVariantMedia\n        stockAvailable\n        stockMinimum\n        priceAfterTax\n        priceNormal\n        isActive\n        sellingPrice\n        activePrice\n        __typename\n      }\n      __typename\n    }\n    __typename\n  }\n}\n\nfragment FilterCategory on AggrCategoryType {\n  id\n  name\n  slug\n  level: category\n  count: docCount\n  __typename\n}\n',
    "variables": {
        "filter": "variants.isActive:true,isSold:true,merchant.isActive:true",
        "keyword": "iphone",  # <  change keyword here
        "range": "",
        "sort": "",
    },
}

data = requests.post(api_url, json=payload).json()

# uncomment this to print all data:
# print(json.dumps(data, indent=4))

for r in data["data"]["search"]["result"]:
    print(r["name"])

印刷品：

Ferrari - Power Case iPhone 11 Pro Max 6.5 Full Cover - Powerbank 4000 mAh
Ferrari - Power Case iPhone 11 Pro Max 6.5 Full Cover - Powerbank 4000 mAh
CASEOLOGY Parallax for iPhone X/ iPhone Xs
CASEOLOGY Waterfall Series for iPhone 7/ iPhone 8
PROTEGO Back Protector for iPhone 7 / iPhone 8
PROTEGO Tempered Glass for Apple iPhone 7 / iPhone 8
SPIGEN iPhone Stand S315
VIVAN Paket Fast Charging iPhone VIVAN Charger USB 3.0 + Kabel iPhone
APPLE iPhone 11
APPLE iPhone 12
APPLE iPhone 12
APPLE iPhone 12
APPLE iPhone 12
APPLE iPhone 12
APPLE iPhone 12
APPLE iPhone 12
APPLE iPhone 12
APPLE iPhone 12
APPLE iPhone 12
APPLE iPhone 12 Pro
APPLE iPhone 12 Pro
APPLE iPhone 12 Pro
APPLE iPhone 12 mini
APPLE iPhone 12 mini
APPLE iPhone 12 mini
APPLE iPhone 12 mini
APPLE iPhone 12 mini
APPLE iPhone 12 mini
APPLE iPhone SE 2020
APPLE iPhone SE 2020
APPLE iPhone SE 2020
APPLE iPhone SE 2020
APPLE iPhone SE 2020
APPLE iPhone SE 2020
APPLE USB Adaptor Charger iPhone
Ailun Screen Protector for iPhone X
INDOSCREEN Anti Break Apple iPhone 11
PROTEGO Tempered Glass iPhone 6 - 6s
PROTEGO Tempered Glass for iPhone XR
CAPDASE Crystal Jacket for iPhone X

Answer 2

正在传递响应对象，而不是内容：soup = BeautifulSoup(r, "html.parser")

供参考：https://docs.python-requests.org/en/master/user/quickstart/#response-content