在使用BS4、Python和Selenium之后解析文本

Question

在使用我的scrape脚本之后：

from selenium import webdriver
from bs4 import BeautifulSoup
import csv

browser = webdriver.Firefox()
browser.get('http://dyn.com/about/events/')
html = browser.page_source
soup = BeautifulSoup(html)
titles = [tag.text for tag in soup.find_all('p','pubdate')]

我得到的结果是：

[u'\n\n\t\t\tWEBINAR: How To Expand Your Global Reach To China\xa0\n\t\t\t\n\t\t\tOct 22, 2014\t\t\t\nspeak \n', u'\n\n\t\t\tLAUNCH Scale \u2013 San Francisco, CA\xa0\n\t\t\t\n\t\t\tOct 23 - 24, 2014\t\t\t\nattend \n', u'\n\n\t\t\tAcquia Engage User Conference \u2013 Boston, MA\xa0\n\t\t\t\n\t\t\tNov 3 - 5, 2014\t\t\t\nexhibitattend \n', u'\n\n\t\t\tCloud Expo \u2013 Santa Clara, CA\xa0\n\t\t\t\n\t\t\tNov 4 - 6, 2014\t\t\t\nexhibit \n', u'\n\n\t\t\tThe Global Carrier Awards 2014 \u2013 Amsterdam\xa0\n\t\t\t\n\t\t\tNov 4, 2014\t\t\t\n\n', u'\n\n\t\t\tWeb Summit \u2013 Dublin, Ireland\xa0\n\t\t\t\n\t\t\tNov 4 - 6, 2014\t\t\t\nspeak \n', u'\n\n\t\t\tVelocity Europe \u2013 Barcelona, Spain\xa0\n\t\t\t\n\t\t\tNov 17 - 19, 2014\t\t\t\nexhibit \n', u'\n\n\t\t\tNH/VT FIRST LEGO League Championship Event\xa0\n\t\t\t\n\t\t\tDec 6, 2014\t\t\t\nspeak \n']

我是python新手，您能建议我如何从这个结果中获得事件名称、日期和事件类型吗？在

谢谢！在

Answer 1

您可以使用stripped_strings属性提取每个组件并构造字典列表，如下所示：

from selenium import webdriver
from bs4 import BeautifulSoup
import csv

browser = webdriver.Firefox()
browser.get('http://dyn.com/about/events/')
html = browser.page_source
soup = BeautifulSoup(html)

events = []
titles = soup.find_all('p','pubdate')
for title in titles:
    event = list(title.stripped_strings)
    events.append({
        'event_name': event[0],
        'event_date': event[1],
        'event_type': event[2:]    # there can be multiple event types so use a list
    })

>>> from pprint import pprint
>>> for event in events:
...     pprint(event)
{'event_date': u'Oct 22, 2014',
 'event_name': u'WEBINAR: How To Expand Your Global Reach To China',
 'event_type': [u'speak']}
{'event_date': u'Oct 23 - 24, 2014',
 'event_name': u'LAUNCH Scale \u2013 San Francisco, CA',
 'event_type': [u'attend']}
{'event_date': u'Nov 3 - 5, 2014',
 'event_name': u'Acquia Engage User Conference \u2013 Boston, MA',
 'event_type': [u'exhibit', u'attend']}
{'event_date': u'Nov 4 - 6, 2014',
 'event_name': u'Cloud Expo \u2013 Santa Clara, CA',
 'event_type': [u'exhibit']}
{'event_date': u'Nov 4, 2014',
 'event_name': u'The Global Carrier Awards 2014 \u2013 Amsterdam',
 'event_type': []}
{'event_date': u'Nov 4 - 6, 2014',
 'event_name': u'Web Summit \u2013 Dublin, Ireland',
 'event_type': [u'speak']}
{'event_date': u'Nov 17 - 19, 2014',
 'event_name': u'Velocity Europe \u2013 Barcelona, Spain',
 'event_type': [u'exhibit']}
{'event_date': u'Dec 6, 2014',
 'event_name': u'NH/VT FIRST LEGO League Championship Event',
 'event_type': [u'speak']}

您可能还想在字符串u' \u2013 '上拆分事件名称和事件位置event[0]

Answer 2

您可以替换输出中的\n和\t

例如，您可以使用re.sub函数来替换。在

import re

for val in titles:
     print re.sub("[\n\t]", "", val)

会产生输出

^{pr2}$

功能

re.sub("[\n\t]", "", val)

将\n和{}替换为空字符串。在