BeautifulSoupで掻き集めるときにリンクが見つかりません

Question

すべてのプレーヤーのリンクをwhoscoredから取得したいと考えています。問題は、BSはクラスplayer-linkとのリンクを見つけることができないということです。これらのリンクを取得する最良の方法は何ですか？彼らは検査官に現れてどこかに存在します。

これは、これまでの私のコードです：助けるため

import requests 
from bs4 import BeautifulSoup as BS 
import re 
from incapsula import IncapSession 
import json 

session = IncapSession() 
links = "https://www.whoscored.com/Matches/1080633/Live/England-Premier-League-2016-2017-Manchester-United-Arsenal" 
try: 
    response = session.get(links).text 
except IncapBlocked as e: 
    raise 
soup = BS(response, "html.parser") 
player_site = soup.find("a", text = "Player Statistics") 
player_link = player_site.get("href") 
player_link = "https://www.whoscored.com" + player_link 
try: 
    response_players = session.get(player_link).text 
except IncapBlocked as e: 
    raise 
soup_players = BS(response_players, "html.parser") 
soup_players.find_all("a", class_ ="player-link") 

感謝。

EDIT：

今、私は、ページが部分的にJavaScriptでロードされているので、私は別の結果を得ることをお読みください。どうすれば解決できますか？

Answer 1

これにはセレンを使用します。

>>> from selenium import webdriver 
>>> driver = webdriver.Chrome() 
>>> driver.get('https://www.whoscored.com/Matches/1080633/LiveStatistics/England-Premier-League-2016-2017-Manchester-United-Arsenal') 
>>> links = driver.find_elements_by_xpath('.//a[@class="player-link"]') 
>>> for link in links: 
...  link.get_attribute('href') 
...  
'https://www.whoscored.com/Players/79554/Show/David-de-Gea' 
'https://www.whoscored.com/Players/18296/Show/Antonio-Valencia' 
'https://www.whoscored.com/Players/70050/Show/Marcos-Rojo' 
'https://www.whoscored.com/Players/81726/Show/Phil-Jones' 
'https://www.whoscored.com/Players/23220/Show/Matteo-Darmian' 
'https://www.whoscored.com/Players/97752/Show/Paul-Pogba' 
'https://www.whoscored.com/Players/71174/Show/Ander-Herrera' 
'https://www.whoscored.com/Players/2115/Show/Michael-Carrick' 
'https://www.whoscored.com/Players/25363/Show/Juan-Mata' 
'https://www.whoscored.com/Players/300299/Show/Marcus-Rashford' 
'https://www.whoscored.com/Players/122366/Show/Anthony-Martial' 
'https://www.whoscored.com/Players/29544/Show/Morgan-Schneiderlin' 
'https://www.whoscored.com/Players/70033/Show/Daley-Blind' 
'https://www.whoscored.com/Players/3859/Show/Wayne-Rooney' 
'https://www.whoscored.com/Players/33831/Show/Sergio-Romero' 
'https://www.whoscored.com/Players/110154/Show/Memphis-Depay' 
'https://www.whoscored.com/Players/109000/Show/Jesse-Lingard' 
'https://www.whoscored.com/Players/8166/Show/Ashley-Young' 
'https://www.whoscored.com/Players/6775/Show/Petr-Cech' 
'https://www.whoscored.com/Players/91879/Show/Carl-Jenkinson' 
'https://www.whoscored.com/Players/30051/Show/Laurent-Koscielny' 
'https://www.whoscored.com/Players/80921/Show/Shkodran-Mustafi' 
'https://www.whoscored.com/Players/23072/Show/Nacho-Monreal' 
'https://www.whoscored.com/Players/125209/Show/Mohamed-Elneny' 
'https://www.whoscored.com/Players/69738/Show/Francis-Coquelin' 
'https://www.whoscored.com/Players/13796/Show/Theo-Walcott' 
'https://www.whoscored.com/Players/13756/Show/Mesut-%C3%96zil' 
'https://www.whoscored.com/Players/26820/Show/Aaron-Ramsey' 
'https://www.whoscored.com/Players/25244/Show/Alexis-S%C3%A1nchez' 
'https://www.whoscored.com/Players/24444/Show/Olivier-Giroud' 
'https://www.whoscored.com/Players/89401/Show/Granit-Xhaka' 
'https://www.whoscored.com/Players/84146/Show/Alex-Oxlade-Chamberlain' 
'https://www.whoscored.com/Players/76810/Show/Gabriel' 
'https://www.whoscored.com/Players/36745/Show/David-Ospina' 
'https://www.whoscored.com/Players/136824/Show/Alex-Iwobi' 
'https://www.whoscored.com/Players/27550/Show/Kieran-Gibbs'