dryscrape「もっとボタンをロード」をクリックしてください

Question

私はwww.ratemyprofessors.comを掻き取ろうとしていましたが、必要なすべてのデータをスクラップするために「もっと読み込み」ボタンをクリックする必要があります。しかし、今使用してイムコードが

loadButton = session.at_xpath(path) 
    loadButton.click() 

が動作していないloadButton.text()は「より多くの負荷」に等しいので、パスは間違いなく正しいです、しかし、それは「あるため、重複要素をクリックすることができなかった」基本的に言って、私にエラーを与えます。

これを解決する方法や回避策を知っている人はいますか？私が読んできたことから、javascriptがネットワークタブで実行されている機能をシミュレートすることもできます。しかし、私はいくつかの問題に直接関数を呼び出していないのonclick以来の機能を見つけることがありますが、その代わりに

onclick="javascript:mtvn.btg.Controller.sendLinkEvent({ linkName:\'PROFMIDPANE:LoadMore\', linkType:\'o\' }); 

ところで私はPythonと、ボタンが教授のリストの下左側にあり、「より多くの負荷を」使用していますあなたは学校

の検索を実行した後、私はいくつかの関連記事を読んでいるが、任意の助けをいただければ幸い便利

クロスは何も来ていません！

my network/params tab

Answer 1

あなたは、負荷に多くのボタンをクリックすると、要求が行われ、requestsとbs4を使用してそれをすべて行うことができます。

ページを持ってたらそう、あなたが得ることができますすべての教師と評価は、jsonの形式で、次のURLを使用してhttp://www.ratemyprofessors.com/ShowRatings.jsp?tid=881718となります。

import requests 
from bs4 import BeautifulSoup 


params = {"solrformat": "true", 
      "rows": "1000", # set it high number to always get all rows. 
      "q": "", 
      "defType": "edismax", 
      "qf": "teacherfullname_t^1000 autosuggest", 
      "bf": "pow(total_number_of_ratings_i,2.1)", 
      "sort": "total_number_of_ratings_i desc", 
      "siteName": "rmp", 
      "fl": "pk_id teacherfirstname_t teacherlastname_t total_number_of_ratings_i averageratingscore_rf schoolid_s"} 

url = "http://search.mtvnservices.com/typeahead/suggest/" 
query = '*:* AND schoolid_s:{id} AND teacherdepartment_s:"{subject}"' 
with requests.Session() as s: 
    s.headers.update({"User-Agent": "Mozilla/5.0 (X11; Linux x86_64)"}) 
    soup = BeautifulSoup(s.get("http://www.ratemyprofessors.com/ShowRatings.jsp?tid=881718").content) 
    # pass the school id which we can parse from the page. 
    params["q"] = query.format(id=soup.select_one("[data-schoolid]")["data-schoolid"], subject="History") 
    res = s.get(url, params=params) 

    json_data = res.json() 


from pprint import pprint as pp 
pp(json_data["response"]["docs"]) 

たちを与える：

[{u'averageratingscore_rf': 4.6, 
    u'pk_id': 1347824, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'JP', 
    u'teacherlastname_t': u'Godwin', 
    u'total_number_of_ratings_i': 88}, 
{u'averageratingscore_rf': 3.38, 
    u'pk_id': 692471, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'James', 
    u'teacherlastname_t': u'Page', 
    u'total_number_of_ratings_i': 49}, 
{u'averageratingscore_rf': 3.5, 
    u'pk_id': 555487, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Kevin', 
    u'teacherlastname_t': u'Davis', 
    u'total_number_of_ratings_i': 44}, 
{u'averageratingscore_rf': 4.4, 
    u'pk_id': 1289399, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Jane', 
    u'teacherlastname_t': u'England', 
    u'total_number_of_ratings_i': 33}, 
{u'averageratingscore_rf': 3.46, 
    u'pk_id': 1230841, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Simone', 
    u'teacherlastname_t': u'De Santiago Ramos', 
    u'total_number_of_ratings_i': 24}, 
{u'averageratingscore_rf': 3.15, 
    u'pk_id': 701257, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Jack', 
    u'teacherlastname_t': u'Pyle', 
    u'total_number_of_ratings_i': 23}, 
{u'averageratingscore_rf': 4.13, 
    u'pk_id': 1466455, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Chris', 
    u'teacherlastname_t': u'Politz', 
    u'total_number_of_ratings_i': 20}, 
{u'averageratingscore_rf': 4.67, 
    u'pk_id': 1218949, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'James', 
    u'teacherlastname_t': u'Hathcock', 
    u'total_number_of_ratings_i': 18}, 
{u'averageratingscore_rf': 3.93, 
    u'pk_id': 1648329, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Joshua', 
    u'teacherlastname_t': u'Montandon', 
    u'total_number_of_ratings_i': 15}, 
{u'averageratingscore_rf': 2.79, 
    u'pk_id': 1543864, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'M', 
    u'teacherlastname_t': u'Antle', 
    u'total_number_of_ratings_i': 14}, 
{u'averageratingscore_rf': 3.83, 
    u'pk_id': 1096585, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Scotty', 
    u'teacherlastname_t': u'Edler', 
    u'total_number_of_ratings_i': 12}, 
{u'averageratingscore_rf': 3.92, 
    u'pk_id': 1260089, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'James', 
    u'teacherlastname_t': u'Reynolds', 
    u'total_number_of_ratings_i': 12}, 
{u'averageratingscore_rf': 4.42, 
    u'pk_id': 1418409, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Steve', 
    u'teacherlastname_t': u'Wolfrum', 
    u'total_number_of_ratings_i': 12}, 
{u'averageratingscore_rf': 4.45, 
    u'pk_id': 899881, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Karen', 
    u'teacherlastname_t': u'Stewart', 
    u'total_number_of_ratings_i': 11}, 
{u'averageratingscore_rf': 3.2, 
    u'pk_id': 592508, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Crystal', 
    u'teacherlastname_t': u'Wright', 
    u'total_number_of_ratings_i': 10}, 
{u'averageratingscore_rf': 4.5, 
    u'pk_id': 891457, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Lisa', 
    u'teacherlastname_t': u'Morales', 
    u'total_number_of_ratings_i': 10}, 
{u'averageratingscore_rf': 2.9, 
    u'pk_id': 1329058, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Mark', 
    u'teacherlastname_t': u'Thompson', 
    u'total_number_of_ratings_i': 10}, 
{u'averageratingscore_rf': 4.0, 
    u'pk_id': 1339373, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Charles', 
    u'teacherlastname_t': u'Williams', 
    u'total_number_of_ratings_i': 10}, 
{u'averageratingscore_rf': 4.5, 
    u'pk_id': 1587880, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Noelle', 
    u'teacherlastname_t': u'Depperschmidt', 
    u'total_number_of_ratings_i': 10}, 
{u'averageratingscore_rf': 4.39, 
    u'pk_id': 1426470, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Adrien', 
    u'teacherlastname_t': u'Ivan', 
    u'total_number_of_ratings_i': 9}, 
{u'averageratingscore_rf': 5.0, 
    u'pk_id': 1871677, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Kevin', 
    u'teacherlastname_t': u'Eades', 
    u'total_number_of_ratings_i': 9}, 
{u'averageratingscore_rf': 4.81, 
    u'pk_id': 393151, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Sharon', 
    u'teacherlastname_t': u'Romero', 
    u'total_number_of_ratings_i': 8}, 
{u'averageratingscore_rf': 3.69, 
    u'pk_id': 1377603, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Joseph', 
    u'teacherlastname_t': u'Ialenti', 
    u'total_number_of_ratings_i': 8}, 
{u'averageratingscore_rf': 3.43, 
    u'pk_id': 1752608, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'James', 
    u'teacherlastname_t': u'Jones', 
    u'total_number_of_ratings_i': 7}, 
{u'averageratingscore_rf': 3.43, 
    u'pk_id': 1782369, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Sara', 
    u'teacherlastname_t': u'Ruppel', 
    u'total_number_of_ratings_i': 7}, 
{u'averageratingscore_rf': 3.33, 
    u'pk_id': 1096000, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Scott', 
    u'teacherlastname_t': u'Harp', 
    u'total_number_of_ratings_i': 6}, 
{u'averageratingscore_rf': 2.17, 
    u'pk_id': 2061535, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'David', 
    u'teacherlastname_t': u'Powell', 
    u'total_number_of_ratings_i': 6}, 
{u'averageratingscore_rf': 4.1, 
    u'pk_id': 556560, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'', 
    u'teacherlastname_t': u'English', 
    u'total_number_of_ratings_i': 5}, 
{u'averageratingscore_rf': 3.9, 
    u'pk_id': 2032232, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Robin', 
    u'teacherlastname_t': u'Jett', 
    u'total_number_of_ratings_i': 5}, 
{u'averageratingscore_rf': 3.3, 
    u'pk_id': 1242893, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Dennis', 
    u'teacherlastname_t': u'Spillman', 
    u'total_number_of_ratings_i': 5}, 
{u'averageratingscore_rf': 5.0, 
    u'pk_id': 1209837, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Jared', 
    u'teacherlastname_t': u'Sutton', 
    u'total_number_of_ratings_i': 4}, 
{u'averageratingscore_rf': 3.38, 
    u'pk_id': 1587886, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Arianna', 
    u'teacherlastname_t': u'Warren', 
    u'total_number_of_ratings_i': 4}, 
{u'averageratingscore_rf': 4.4, 
    u'pk_id': 1643053, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Kimberly', 
    u'teacherlastname_t': u'Lacoco', 
    u'total_number_of_ratings_i': 4}, 
{u'averageratingscore_rf': 2.5, 
    u'pk_id': 1857299, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Kevin', 
    u'teacherlastname_t': u'Pyle', 
    u'total_number_of_ratings_i': 4}, 
{u'averageratingscore_rf': 2.33, 
    u'pk_id': 892723, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Keith', 
    u'teacherlastname_t': u'Mitchener', 
    u'total_number_of_ratings_i': 3}, 
{u'averageratingscore_rf': 3.5, 
    u'pk_id': 1448008, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Sally', 
    u'teacherlastname_t': u'Stratso', 
    u'total_number_of_ratings_i': 3}, 
{u'averageratingscore_rf': 3.25, 
    u'pk_id': 680381, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Todd', 
    u'teacherlastname_t': u'Venable', 
    u'total_number_of_ratings_i': 2}, 
{u'averageratingscore_rf': 5.0, 
    u'pk_id': 1256069, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Amanda', 
    u'teacherlastname_t': u'Campbell-Wyatt', 
    u'total_number_of_ratings_i': 2}, 
{u'averageratingscore_rf': 5.0, 
    u'pk_id': 2142326, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Jeremy', 
    u'teacherlastname_t': u'Godwin', 
    u'total_number_of_ratings_i': 2}, 
{u'averageratingscore_rf': 1.5, 
    u'pk_id': 697421, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Woody', 
    u'teacherlastname_t': u'Paige', 
    u'total_number_of_ratings_i': 1}, 
{u'averageratingscore_rf': 1.0, 
    u'pk_id': 881718, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'M', 
    u'teacherlastname_t': u'Sullivan', 
    u'total_number_of_ratings_i': 1}, 
{u'averageratingscore_rf': 1.5, 
    u'pk_id': 1607181, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Nancy', 
    u'teacherlastname_t': u'Coffelt', 
    u'total_number_of_ratings_i': 1}, 
{u'averageratingscore_rf': 5.0, 
    u'pk_id': 1710114, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Jason', 
    u'teacherlastname_t': u'Scheller', 
    u'total_number_of_ratings_i': 1}, 
{u'averageratingscore_rf': 4.0, 
    u'pk_id': 2164391, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'James', 
    u'teacherlastname_t': u'Paige', 
    u'total_number_of_ratings_i': 1}, 
{u'pk_id': 2083511, 
    u'schoolid_s': u'4873', 
    u'teacherfirstname_t': u'Stephen ', 
    u'teacherlastname_t': u'Wolfrum', 
    u'total_number_of_ratings_i': 0}] 

必要なのは、その学校のIDとクエリ文字列の対象を渡し、あなたは何が好き得ることができているし。