解析机器人.txt在python中

Question

我要解析机器人.txtpython文件。 我已经研究过robotParser和roboteExclusionParser，但是没有什么真正满足我的条件。我想获取所有的diallowDurls和allowedURL，而不是手动检查每个url，如果允许或不允许。有图书馆可以做这个吗？在

Answer 1

您可以使用curl命令读取机器人.txt将文件拆分为一个字符串，用新行检查允许和不允许的URL。在

import os
result = os.popen("curl https://fortune.com/robots.txt").read()
result_data_set = {"Disallowed":[], "Allowed":[]}

for line in result.split("\n"):
    if line.startswith('Allow'):    # this is for allowed url
        result_data_set["Allowed"].append(line.split(': ')[1].split(' ')[0])    # to neglect the comments or other junk info
    elif line.startswith('Disallow'):    # this is for disallowed url
        result_data_set["Disallowed"].append(line.split(': ')[1].split(' ')[0])    # to neglect the comments or other junk info

print (result_data_set)

Answer 2

为什么你必须手动检查你的网址？ 您可以在Python3中使用urllib.robotparser，并执行以下操作

import urllib.robotparser as urobot

url = "example.com"
rp = urobot.RobotFileParser()
rp.set_url(url + "/robots.txt")
rp.read()
if rp.can_fetch("*", url):
    site = urllib.request.urlopen(url)
    sauce = site.read()
    soup = BeautifulSoup(sauce, "html.parser")
    actual_url = site.geturl()[:site.geturl().rfind('/')]

    my_list = soup.find_all("a", href=True)
    for i in my_list:
        # rather than != "#" you can control your list before loop over it
        if i != "#":
            newurl = str(actual_url+"/"+i)
            try:
                if rp.can_fetch("*", newurl):
                    site = urllib.request.urlopen(newurl)
                    # do what you want on each authorized webpage
            except:
                pass
else:
    print("cannot scrap")