刮取跳过部分页面url，不会将所有值写入CSV

Question

我对Python还不熟悉。我昨天才开始。我想浏览一个网站，收集字典里的数据。 所有导入都添加在python脚本的开头

title_and_urls = {} #dictionary
totalNumberOfPages = 12
for x in range(1,int(totalNumberOfPages)+1):
    url_pages = 'https://abd.com/api?&page=' +str(x)+'&year=2017'
    resp = requests.get(url_pages, timeout=60)
    soup = BeautifulSoup(resp.text, 'lxml')
    for div in soup.find_all('div', {"class": "block2"}):
        a = div.find('a')
        h3 = a.find('h3')
        print(h3,url_pages) #prints correct
        title_and_urls[h3.text] = base_enthu_url+a.attrs['href']

print(title_and_urls)


with open('dict.csv', 'wb') as csv_file:
    writer = csv.writer(csv_file)
    for key, value in title_and_urls.items():
       writer.writerow([key, value])

这里有几个问题 1.我总共有12页，但它跳过了第7页和第8页 2.打印行print(h3,url_pages)打印了60项，而csv文件只有36项

我感谢所有的帮助和解释。请建议最佳做法

Answer 1

使用try函数

**title_and_urls = {} #dictionary
totalNumberOfPages = 12
for x in range(1,int(totalNumberOfPages)+1):
    try:
        url_pages = 'https://abd.com/api?&page=' +str(x)+'&year=2017'
        resp = requests.get(url_pages, timeout=60)
        soup = BeautifulSoup(resp.text, 'lxml')
        for div in soup.find_all('div', {"class": "block2"}):
            a = div.find('a')
            h3 = a.find('h3')
            print(h3,url_pages) #prints correct
            title_and_urls[h3.text] = base_enthu_url+a.attrs['href']
    except:
        pass


with open('dict.csv', 'wb') as csv_file:
    writer = csv.writer(csv_file)
    for key, value in title_and_urls.items():
       writer.writerow([key, value])**