网页刮擦天气标签

Question

我正在尝试网络抓取天气数据，需要采取表和转换成csv格式。但并非表中的所有条目都填充了相同数量的列。所以当我以这种格式输入时

for h in airports:

    for i in range(1,3):
          if(i==1):
              for j in range(1,32):
                  url="https://www.wunderground.com/history/airport/"+str(h)+"/2018/"+str(i)+"/"+str(j)+"/DailyHistory.html?req_city=&req_state=&req_statename=&reqdb.zip=&reqdb.magic=&reqdb.wmo="
                  www= urllib3.PoolManager()
                  page=www.urlopen("GET",url)
                  bs= BeautifulSoup(page.data,"lxml")
                  x=bs.find('div',{"class":"high-res"})
                  for tr in x.findAll('tr'):
                         weather.append([td for td in tr.stripped_strings])

          else: 
              for k in range(1,29):
                  url="https://www.wunderground.com/history/airport/"+str(h)+"/2018/"+str(i)+"/"+str(k)+"/DailyHistory.html?req_city=&req_state=&req_statename=&reqdb.zip=&reqdb.magic=&reqdb.wmo="
                  www= urllib3.PoolManager()
                  page=www.urlopen("GET",url)
                  bs= BeautifulSoup(page.data,"lxml")
                  x=bs.find('div',{"class":"high-res"})
                  for tr in x.findAll('tr'):
                          weather.append([td for td in tr.stripped_strings])

输出的csv文件到处都是，逗号分隔的变量每个都将进入一个新的列，而不考虑标题。 有没有一个简单的方法来做这件事，并得到一个更清晰的日期？

所以我不断地在一个列表中添加表中的行，而不考虑列。如何确保列中的数据位于右标题下？

这是我用来将数据写入csv文件的：

^{pr2}$

Answer 1

所以，下面这些人似乎解决了我在正确的列标题下获取正确数据的问题：

for tr in x.findAll('tr'):
                     cols=tr.findAll('td')
                     cols=[ele.text.strip() for ele in cols]
                     weather.append([ele for ele in cols if ele])

result=pd.DataFrame(weather,columns=["Time(EST)","Temp.","Windchill","Dew Point","Humidity","Pressure","Visibility","Wind Dir","Wind Speed","Gust Speed","Precip","Events","Conditions"])

But i got a new problem that is when i strip the text there are some missing values in the table which the code neglects and continues to fill in the wrong header. please help