如何将BeautifulSoup4爬取结果中的循环字典保存为JSON文件格式..?

Question

我用beautifulsoup4从一个网站页面上截取数据，并将截取结果保存到一个列表字典中，如下所示：

DATA = [
            TITLE, {
                'IMAGES': IMAGE,
                'URL_VIDEOS': URL_VIDEOS,
                'DESCRIPTIONS': DESCRIPTIONS,
                'SYNOPSIS': SYNOPSIS
            }
        ]

即值IMAGE、URL videos、DESCRIPTIONS和sympsis用于变量刮取结果。你知道吗

我尝试用以下代码将变量数据保存到.json文件扩展名中：

json_file = open('result.json', 'w')
json.dump(DATA, json_file)
json_file.close()

结果是这样的：

["Action Fruits", {"IMAGES": "http://animeindo.video/wp-content/uploads/2017/07/rsz_heroin.jpg", "URL_VIDEOS": "http://www.mp4upload.com/embed-q7xxgge1yu1c.html", "DESCRIPTIONS": {"Japanese": " \u30a2\u30af\u30b7\u30e7\u30f3\u30d2\u30ed\u30a4\u30f3 \u30c1\u30a2\u30d5\u30eb\u30fc\u30c4", "\nProducer": " Diomedea", "\nType": " TV Series", "\nStatus": " Ongoing", "\nGenre": " Comedy, School, Slice of Life", "\nDurasi": " 24 min", "\nEpisode": " \u2013", "\nRating": " 6.11", "\nAdded On": " July 12th, 2017"}, "SYNOPSIS": "Japanese: \u30a2\u30af\u30b7\u30e7\u30f3\u30d2\u30ed\u30a4\u30f3 \u30c1\u30a2\u30d5\u30eb\u30fc\u30c4\nProducer: Diomedea\nType: TV Series\nStatus: Ongoing\nGenre: Comedy, School, Slice of Life\nDurasi: 24 min\nEpisode: \u2013\nRating: 6.11\nAdded On: July 12th, 2017\nSinopsis:\nPerjuangan pahlawan lokal dalam menyelamatkan daerahnya.\n"}]

但是在那次循环中，.json文件的结果总是被覆盖，这并没有添加新的数据，只是用新的数据覆盖，就像这样：

["Happy", {"IMAGES": "https://1.bp.blogspot.com/-SUq5_dpoIlM/VwpKqqsEzNI/AAAAAAAAM50/H81MUyDLZA0ctj8zo8JbuUVPPz4sxQulw/s1600/77219__1460292250_36.80.228.117.jpg", "URL_VIDEOS": "http://www.mp4upload.com/embed-ptj9hmeefar8.html", "DESCRIPTIONS": {"Japanese": " \u3042\u3093\u30cf\u30d4\u266a", "\nProducer": " Silver Link", "\nType": " TV Series", "\nStatus": " Ongoing", "\nGenre": " Comedy, School, Slice of Life", "\nDurasi": " 23 min. per ep.", "\nEpisode": " 12", "\nRating": " 7.06", "\nAdded On": " April 10th, 2016"}, "SYNOPSIS": "Japanese: \u3042\u3093\u30cf\u30d4\u266a\nProducer: Silver Link\nType: TV Series\nStatus: Ongoing\nGenre: Comedy, School, Slice of Life\nDurasi: 23 min. per ep.\nEpisode: 12\nRating: 7.06\nAdded On: April 10th, 2016\nSinopsis:\nMenceritakan kelas 1-7 di Akademi Tennomifune, di mana semua murid yang suka sial berkumpul. Hibari, salah satu murid di kelas ini, bertemu dengan si sial Hanako di hari pertama sekolah, dan bersama-sama mereka berjuang mencari hidup bahagia di sekolah mereka.\n"}]

下一个结果也覆盖了。。。你知道吗

我想添加新数据，并用一个.json文件保存所有刮取结果。所以。。怎么做。。？你知道吗

Answer 1

'w'文件模式会在每次写入时重写文件。你知道吗

'a'在这里不起作用，因为它会导致一个无效的JSON文件。你知道吗

你应该做的是收集结果，而你刮（到一个列表？）然后在完成数据循环后，将数据转储到JSON文件中一次。你知道吗

Answer 2

我已通过以下代码解决此问题：

 with open('result.json', 'a') as outfile:
      outfile.write(json.dumps(DATA, sort_keys=True, indent=4))

我从here那里得到了答案。你知道吗

Answer 3

如何选择IMAGES或URL_VIDEOS的TITLE？我认为你的json是不正确的，因为Title是value而不是key，也许它应该是这样的格式

{
  "title A" : {"IMAGES" : "IMAGE A"},
  "title B" : {"IMAGES" : "IMAGE B"}
}

或者

[
  {"Title" : "title A", "IMAGES" : "IMAGE A"},
  {"Title" : "title B", "IMAGES" : "IMAGE B"}
]

让我们试试第一个示例，您需要读取以前的json和update()新数据，首先确保删除result.json

import os.path

....
DATA = {"Action Fruits": {"IMAGES": "a.jpg", "URL_VIDEOS" : "http://a.mp4"}}

OLD_DATA = {} # set old data to this if file not exist
if os.path.isfile('result.json'):
    with open('result.json', 'r') as f:
        OLD_DATA = json.load(f)
        # {"Happy" : {"IMAGES" : "b.jpg", "URL_VIDEOS" : "http://b.mp4"}}

# concatenate old and new data
DATA.update(OLD_DATA)
with open('result.json', 'w') as f:
    json.dump(DATA, f)

结果.json

{
  "Action Fruits": {"IMAGES": "a.jpg", "URL_VIDEOS": "http://a.mp4"}, 
  "Happy": {"IMAGES": "b.jpg", "URL_VIDEOS": "http://b.mp4"}
}