无法从某些json内容中提取不同深度的所有可用URL

Question

我试图解析一些json内容中不同深度的URL的所有值。我附上a file包含不同深度的URL供您考虑

这就是它们的结构（截断）：

{'hasSub': True,
 'navigationTitle': 'Products',
 'nodeName': 'products',
 'pages': [{'hasSub': True,
            'navigationTitle': 'Enclosures',
            'nodeName': 'PG0002SCHRANK1',
            'pages': [{'hasSub': True,
                       'navigationTitle': 'Hygienic Design',
                       'nodeName': 'PG0125SCHRANK1',
                       'pages': [{'hasSub': False,
                                  'navigationTitle': 'Hygienic Design Terminal '
                                                     'box HD',
                                  'nodeName': 'PRO0130',
                                  'target': '_self',
                                  'url': '/com-en/products/PG0002SCHRANK1/PG0125SCHRANK1/PRO0130'},
                                 {'hasSub': False,
                                  'navigationTitle': 'Hygienic Design Compact '
                                                     'enclosure HD, '
                                                     'single-door',
                                  'nodeName': 'PRO0131',
                                  'target': '_self',
                                  'url': '/com-en/products/PG0002SCHRANK1/PG0125SCHRANK1/PRO0131'},

如果我考虑上面的内容，我的输出是：

/com-en/products/PG0002SCHRANK1/PG0125SCHRANK1/PRO0130
/com-en/products/PG0002SCHRANK1/PG0125SCHRANK1/PRO0131

我编写的用于生成json内容的脚本：

import requests
from pprint import pprint

url = 'https://www.rittal.com/.rest/nav/menu/tree?'
params = {
    'path': 'com',
    'locale': 'en',
    'deep': '10'
}
with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36'
    s.headers['Accept'] = 'application/json, text/plain, */*'
    r = s.get(url,params=params)
    pprint(r.json()['pages'][0])

How can I scrape all the urls from different depth out of the json content?

Answer 1

好的，看来我在别处找到了一个解决方案，可以从任何嵌套的json中获取所有可用的链接

import requests
from pprint import pprint

url = 'https://www.rittal.com/.rest/nav/menu/tree?'
params = {
    'path': 'com',
    'locale': 'en',
    'deep': '10'
}

def json_extract(obj, key):
    arr = []

    def extract(obj, arr, key):
        if isinstance(obj, dict):
            for k, v in obj.items():
                if isinstance(v, (dict, list)):
                    extract(v, arr, key)
                elif k == key:
                    arr.append(v)
        elif isinstance(obj, list):
            for item in obj:
                extract(item, arr, key)
        return arr

    values = extract(obj, arr, key)
    return values

with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36'
    s.headers['Accept'] = 'application/json, text/plain, */*'
    r = s.get(url, params=params).json()
    for item in json_extract(r,'url'):
        print(item)

脚本生成的链接数约为3500