BeautifulSupとPythonがjsタグの周りを掻き回っていますか？

Question

複数の見出し、リンク、日付を取得しようとしています。最初のものだけを取得する。なぜBS4がすべての項目をフェッチしないのか分かりません...それはJavaScriptの問題ですか？

from bs4 import BeautifulSoup 
from urllib import urlopen 

html = urlopen("http://www.fiercepharma.com/news") 
soup = BeautifulSoup(html.read().decode('utf-8'),"lxml") 
main_div = soup.select_one("div#content") 
div_sub = main_div.select("div.region.region-content") 

for d in div_sub: 
    date = d.time.get_text() 
    headline = d.h2.a.get_text() 
    url = d.a["href"] 
    print headline, url, date 

Answer 1

両方 div.card.horizontal.views-row

とは@citra_amarillo動作するはずです。私はこれを走らせて、両道で働いています。

from bs4 import BeautifulSoup 
from urllib import urlopen 


html = urlopen("http://www.fiercepharma.com/news") 
soup = BeautifulSoup(html.read().decode('utf-8'),"lxml") 
main_div = soup.select_one("div#content") 
div_sub = main_div.select(".card.horizontal.views-row") 
#div_sub = main_div.select("div.card.horizontal.views-row") 

for d in div_sub: 
    date = d.time.get_text() 
    headline = d.h2.a.get_text() 
    url = d.a["href"] 
    print headline, url, date 

Answer 2

メインページにリンク、著者、投稿日を含むすべての記事をキャプチャするには、次のようにします。これを辞書に保存するか、簡単な操作のためにパンダのデータフレームに保存することができます。

from bs4 import BeautifulSoup 
import requests 

baseurl = 'http://www.fiercepharma.com' 
response = requests.get(baseurl) 

soup = BeautifulSoup(response.content) 

cdict = {} 

for group in soup.find_all('div', {'class' : 'card horizontal views-row'}): 
    try: 
     title = group.find('h2', {'class' : 'field-content list-title'}).text 
     link = baseurl + group.find('h2', {'class' : 'field-content list-title'}).find('a', href=True)['href'] 
     author = group.find('span', {'class' : 'field-content'}).find('a').text 
     time = group.find('span', {'class' : 'field-content'}).find('time').text 
     content = group.find('p', {'class' : 'field-content card-text'}).text 
     cdict[link] = {'title' : title, 'author' : author, 'time' : time, 'content' : content} 
    except AttributeError as e: 
     print('[-] Unable to parse {}'.format(e)) 

print(cdict) 
#{'http://www.fiercepharma.com/manufacturing/lonza-bulks-up-5-5b-deal-for-capsugel': {'author': u'Eric Palmer', 
# 'content': u'Swiss CDMO Lonza has pulled the trigger on a $5.5 billion deal to acquire the U.S.-based contract capsule and drug producer Capsugel to create another sizable\u2026', 
# 'time': u'Dec 15, 2016 8:45am', 
# 'title': u'Lonza bulks up with $5.5B deal for Capsugel'},