Pythonでウェブページからリンクを取得する

Question

こんにちは！私はこのスクリプトを持っています：

URL = "http://www.hitmeister.de/" 

page = urllib2.urlopen(URL).read() 
soup = BeautifulSoup(page) 

links = soup.findAll('a') 

for link in links: 
    print link['href'] 

これはウェブページからのリンクを得るはずですが、何が問題なのでしょうか？私もUser-Agentヘッダで試してみましたが、結果はありませんが、このスクリプトは他のWebページでも動作します。

Answer 1

BeautifulSoupからの本当に素晴らしいエラーメッセージがあります。あなたはそれを読んでそれに従ったのですか？

/Library/Python/2.7/site-packages/bs4/builder/_htmlparser.py:149: RuntimeWarning: Python's built-in HTMLParser cannot parse the given document. This is not a bug in Beautiful Soup. The best solution is to install an external parser (lxml or html5lib), and use Beautiful Soup with that parser. See http://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-parser for help.

"Python's built-in HTMLParser cannot parse the given document. This is not a bug in Beautiful Soup. The best solution is to install an external parser (lxml or html5lib), and use Beautiful Soup with that parser. See http://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-parser for help."))

Traceback (most recent call last):

File "", line 1, in

File "/Library/Python/2.7/site-packages/bs4/init.py", line 172, in init self._feed()

File "/Library/Python/2.7/site-packages/bs4/init.py", line 185, in _feed self.builder.feed(self.markup)

File "/Library/Python/2.7/site-packages/bs4/builder/_htmlparser.py", line 150, in feed raise e

HTMLParser.HTMLParseError: malformed start tag, at line 57, column 872

Answer 2

import urllib 
import lxml.html 
import urlparse 

def get_dom(url): 
    connection = urllib.urlopen(url) 
    return lxml.html.fromstring(connection.read()) 

def get_links(url): 
    return resolve_links((link for link in get_dom(url).xpath('//a/@href'))) 

def guess_root(links): 
    for link in links: 
     if link.startswith('http'): 
      parsed_link = urlparse.urlparse(link) 
      scheme = parsed_link.scheme + '://' 
      netloc = parsed_link.netloc 
      return scheme + netloc 

def resolve_links(links): 
    root = guess_root(links) 
    for link in links: 
     if not link.startswith('http'): 
      link = urlparse.urljoin(root, link) 
     yield link 


for link in get_links('http://www.google.com'): 
    print link