JavaScriptを使用してウェブページからのリンクを掻き集めるR

Question

私は個々のプロバイダのURLをhttp://www.childrenshospital.org/directory?state=%7B%22showLandingContent%22%3Afalse%2C%22model%22%3A%7B%22search_specialist%22%3Afalse%2C%22search_type%22%3A%5B%22directoryphysician%22%2C%22directorynurse%22%5D%7D%2C%22customModel%22%3A%7B%22nurses%22%3Atrue%7D%7Dから削り取ろうとしていました。

私はページのソースを確認し、関心のあるURLを特定しました。例えば、私は、「http://www.childrenshospital.org/doctors/mirna-aeschlimann」私は、次のコードを試み、以下のノード

<a data-layer-event="searchClick" data-bind="attr: {href: model.Url}" href="http://www.childrenshospital.org/doctors/mirna-aeschlimann"><!--ko text: model.FirstName-->Mirna<!--/ko--><!--ko text: ' ' + model.LastName--> Aeschlimann<!--/ko--><!--ko if: model.Suffix-->, <!--ko text: model.Suffix-->MD<!--/ko--><!--/ko--></a> 

を形成こすりたいです。しかし、何らかの理由で、上記のノードが返されませんでした。

base_html <- "http://www.childrenshospital.org/directory?state=%7B%22showLandingContent%22%3Afalse%2C%22model%22%3A%7B%22search_specialist%22%3Afalse%2C%22search_type%22%3A%5B%22directoryphysician%22%2C%22directorynurse%22%5D%7D%2C%22customModel%22%3A%7B%22nurses%22%3Atrue%7D%7D" 
doc <- htmlTreeParse(base_html, useInternal = TRUE) 

ご協力いただければ幸いです。詳細をお知らせください。

Answer 1

サイトでデータを取得するXHRリクエストをターゲットに設定してみましたか？

library(httr) 
library(purrr) 
library(xml2) 
library(dplyr) 
library(jsonlite) 

map_df(1:17, function(i) { 

    POST("http://www.childrenshospital.org/searchdirectory.ajax", 
     body = list(search_query = "", 
        search_specialties = "", 
        search_languages = "", 
        search_gender = "", search_departments = "", 
        search_programs = "", search_userlocation = "", 
        search_radius = "10", search_pcp = "true", 
        search_specialist = "false", 
        search_type = "directorynurse|directoryphysician", 
        search_letter = "", search_querygroup = "dirnametext", 
        search_page = "10"), 
     encode = "form") -> res 

    content(res, as="text") %>% 
    fromJSON() %>% 
    .$Records %>% 
    mutate(Address=xml2::xml_text(xml2::read_html(paste0("<x>", Address, "</x>")))) %>% 
    tbl_df() 

}) -> tmp_df 

glimpse(tmp_df) 
## Observations: 408 
## Variables: 21 
## $ ID    <chr> "{E8ECAF3B-B49C-4CD8-AB16-6CE63F0379C0}", "{1E1... 
## $ FirstName  <chr> "Jonathan", "Barbara", "Mark", "Maura", "Sarah"... 
## $ LastName  <chr> "Schwab", "Seagle", "Shapira", "Shea", "Sheldon... 
## $ Image   <chr> "/~/media/directory/physicians/schwab_jonathan.... 
## $ Suffix   <chr> "MD", "MD", "MD", "MD", "MD", "MD", "MD", "MD",... 
## $ Url   <chr> "http://www.childrenshospital.org/doctors/jonat... 
## $ Gender   <chr> "male", "female", "male", "female", "female", "... 
## $ Latitude  <chr> "42.3344382", "42.326435", "41.559642", "42.423... 
## $ Longitude  <chr> "-72.6618324", "-71.149499", "-70.939315", "-71... 
## $ Address  <chr> "{\"practice_name\":\"Northampton Area Pediatri... 
## $ Distance  <chr> "", "", "", "", "", "", "", "", "", "", "", "",... 
## $ OtherLocations <chr> "", "", "", "", "", "", "", "", "", "", "Westwo... 
## $ AcademicTitle <chr> "", "", "", "", "", "", "", "", "", "", "", "",... 
## $ HospitalTitle <chr> "Pediatrician", "Pediatrician", "Pediatrician",... 
## $ Specialties <chr> "Pediatrics", "General Pediatrics, Pediatrics, ... 
## $ Departments <chr> "", "General Pediatrics", "General Pediatrics",... 
## $ Languages  <chr> "", "English", "", "English", "English", "", "E... 
## $ PPOCLink  <chr> "http://www.childrenshospital.org/patient-resou... 
## $ Gallery  <chr> "", "", "", "", "", "", "", "", "", "", "", "",... 
## $ Phone   <chr> "(413) 584-8700", "(617) 731-0200", "(508) 996-... 
## $ Fax   <chr> "(413) 584-1714", "(617) 731-0289", "(508) 992-...