Pythonの単純なWebクローラー・エラー（無限ループ・クロール）

Question

私はPythonで単純なクローラーを作成しました。それは正常に動作し、新しいリンクを見つけるように見えるが、同じリンクの発見を繰り返し、それは新しいWebページをダウンロードしていない見つけた。設定されたクロールの深さの限界に達しても、無限にクロールするようです。私は何の誤りもありません。それはただ永遠に実行されます。ここにコードと実行があります。私はWindows 7 64ビットでPython 2.7を使用しています。

C:\Users\Hussam-Den\Desktop>python test.py https://www.yahoo.com/ 4 
Entered URL: 
https://www.yahoo.com/ 
Final URL: 
https://www.yahoo.com/ 
******************* 
1 

そして、クロールされたURLを格納するための出力ファイルは、このいずれかです：

https://www.yahoo.com/ 
https://www.yahoo.com/lifestyle/horoscope/libra/daily-20170924.html 
https://policies.yahoo.com/us/en/yahoo/terms/utos/index.htm 
https://policies.yahoo.com/us/en/yahoo/privacy/adinfo/index.htm 
https://www.oath.com/careers/work-at-oath/ 
https://help.yahoo.com/kb/account 
https://www.yahoo.com/ 
https://www.yahoo.com/lifestyle/horoscope/libra/daily-20170924.html 
https://policies.yahoo.com/us/en/yahoo/terms/utos/index.htm 
https://policies.yahoo.com/us/en/yahoo/privacy/adinfo/index.htm 
https://www.oath.com/careers/work-at-oath/ 
https://help.yahoo.com/kb/account 
https://www.yahoo.com/ 
https://www.yahoo.com/lifestyle/horoscope/libra/daily-20170924.html 
https://policies.yahoo.com/us/en/yahoo/terms/utos/index.htm 
https://policies.yahoo.com/us/en/yahoo/privacy/adinfo/index.htm 
https://www.oath.com/careers/work-at-oath/ 
https://help.yahoo.com/kb/account 
https://www.yahoo.com/ 
https://www.yahoo.com/lifestyle/horoscope/libra/daily-20170924.html 
https://policies.yahoo.com/us/en/yahoo/terms/utos/index.htm 
https://policies.yahoo.com/us/en/yahoo/privacy/adinfo/index.htm 
https://www.oath.com/careers/work-at-oath/ 
https://help.yahoo.com/kb/account 
https://www.yahoo.com/ 
https://www.yahoo.com/lifestyle/horoscope/libra/daily-20170924.html 
https://policies.yahoo.com/us/en/yahoo/terms/utos/index.htm 
https://policies.yahoo.com/us/en/yahoo/privacy/adinfo/index.htm 
https://www.oath.com/careers/work-at-oath/ 
https://help.yahoo.com/kb/account 
https://www.yahoo.com/ 
https://www.yahoo.com/lifestyle/horoscope/libra/daily-20170924.html 
https://policies.yahoo.com/us/en/yahoo/terms/utos/index.htm 
https://policies.yahoo.com/us/en/yahoo/privacy/adinfo/index.htm 
https://www.oath.com/careers/work-at-oath/ 
https://help.yahoo.com/kb/account 
https://www.yahoo.com/ 
https://www.yahoo.com/lifestyle/horoscope/libra/daily-20170924.html 
https://policies.yahoo.com/us/en/yahoo/terms/utos/index.htm 
https://policies.yahoo.com/us/en/yahoo/privacy/adinfo/index.htm 
https://www.oath.com/careers/work-at-oath/ 
https://help.yahoo.com/kb/account 
https://www.yahoo.com/ 
https://www.yahoo.com/lifestyle/horoscope/libra/daily-20170924.html 
https://policies.yahoo.com/us/en/yahoo/terms/utos/index.htm 
https://policies.yahoo.com/us/en/yahoo/privacy/adinfo/index.htm 
https://www.oath.com/careers/work-at-oath/ 
https://help.yahoo.com/kb/account 
https://www.yahoo.com/ 
https://www.yahoo.com/lifestyle/horoscope/libra/daily-20170924.html 
https://policies.yahoo.com/us/en/yahoo/terms/utos/index.htm 
https://policies.yahoo.com/us/en/yahoo/privacy/adinfo/index.htm 
https://www.oath.com/careers/work-at-oath/ 
https://www.yahoo.com/ 
https://www.yahoo.com/lifestyle/horoscope/libra/daily-20170924.html 
https://policies.yahoo.com/us/en/yahoo/terms/utos/index.htm 
https://policies.yahoo.com/us/en/yahoo/privacy/adinfo/index.htm 
https://www.oath.com/careers/work-at-oath/ 
https://help.yahoo.com/kb/account 

間違っているものを任意のアイデアここで

import sys 
import time 
from bs4 import * 
import urllib2 
import re 
from urlparse import urljoin 

def crawl(url): 
    url = url.strip() 
    page_file_name = str(hash(url)) 
    page_file_name = page_file_name + ".html" 
    fh_page = open(page_file_name, "w") 
    fh_urls = open("urls.txt", "a") 
    fh_urls.write(url + "\n") 
    html_page = urllib2.urlopen(url) 
    soup = BeautifulSoup(html_page, "html.parser") 
    html_text = str(soup) 
    fh_page.write(url + "\n") 
    fh_page.write(page_file_name + "\n") 
    fh_page.write(html_text) 
    links = [] 
    for link in soup.findAll('a', attrs={'href': re.compile("^http://")}): 
    links.append(link.get('href')) 
    rs = [] 
    for link in links: 
    try: 
      #r = urllib2.urlparse.urljoin(url, link) 
      r = urllib2.urlopen(link) 
      r_str = str(r.geturl()) 
      fh_urls.write(r_str + "\n") 
      #a = urllib2.urlopen(r) 
      if r.headers['content-type'] == "html" and r.getcode() == 200: 
       rs.append(r) 
       print "Extracted link:" 
     print link 
     print "Extracted link final URL:" 
     print r 
    except urllib2.HTTPError as e: 
      print "There is an error crawling links in this page:" 
      print "Error Code:" 
      print e.code 
    return rs 
    fh_page.close() 
    fh_urls.close() 

if __name__ == "__main__": 
    if len(sys.argv) != 3: 
    print "Usage: python crawl.py <seed_url> <crawling_depth>" 
    print "e.g: python crawl.py https://www.yahoo.com/ 5" 
    exit() 
    url = sys.argv[1] 
    depth = sys.argv[2] 
    print "Entered URL:" 
    print url 
    html_page = urllib2.urlopen(url) 
    print "Final URL:" 
    print html_page.geturl() 
    print "*******************" 
    url_list = [url, ] 
    current_depth = 0 
    while current_depth < depth: 
     for link in url_list: 
      new_links = crawl(link) 
      for new_link in new_links: 
       if new_link not in url_list: 
        url_list.append(new_link) 
      time.sleep(5) 
      current_depth += 1 
      print current_depth 

は、私はそれを実行したときに、私が得たものでしょうか？

Answer 1

1. あなたがここにエラーがあります：depth = sys.argv[2]、sysリターンstrないintを。あなたは1点のdepth = int(sys.argv[2])
2. Becouse、条件while current_depth < depth:常にintに変換argv[2]でそれを修正してみTrue

を返すを記述する必要があります。薄いエラーがあります