使用BeautifulSoup和Python在poorlyformed表中获取一列

Question

我正在遍历一个.csv的合同，试图从网站中提取一个列

这里有一个网站的例子：https://www.austintexas.gov/financeonline/contract_catalog/OCCViewMA.cfm?cd=CT&dd=6100&id=13060600641

我想从网页末尾的表中抓取标有“商品描述”的列。但是，我不知道如何获取列—只获取行

这是我目前正在使用的代码

def scraper(first, second, third):
    url = "https://www.austintexas.gov/financeonline/contract_catalog/OCCViewMA.cfm?cd=%s&dd=%d&id=%s" % (first, second, third)
    soup = BeautifulSoup(urllib2.urlopen(url).read())
    foundtext = soup.find('td',text="Commodity Description")
    table = foundtext.findPrevious('table')
    rows = table.findAll('tr')
    second_column = []
    for row in rows:
        print row.contents

我希望最终的输出返回是该列中所有行的文本，行之间有返回框

有什么想法吗

Answer 1

对于找到的每一行，查找所有td元素并按索引获取所需的元素：

table = soup.find('td', text="Commodity Description").find_parent("table")
for row in table.select("tr")[2:]:  # skipping the header rows
    cell = row.find_all("td")[1]
    print(cell.get_text())
    print("  ")

印刷品：

WATERLINE REPLACEMENTCONSTRUCTION, PIPELINEPER YUEJIAO LIU, ADD THE REMAINING FUNDS BACK INTO THIS FUNDING LINE  //   PEMBERTON HEIGHTS PHASE III PROJECT  ++   ENC.  $53,209.97
  
WATERLINE REPLACEMENTCONSTRUCTION, PIPELINEPEMBERTON HEIGHTS PHASE III PROJECT
  
WATERLINE REPLACEMENTCONSTRUCTION, PIPELINEPEMBERTON HEIGHTS PHASE III PROJECT