クロールデータに情報がありません

Question

XXXでAlphaGoに関連するすべてのニュース（タイトル、URL、テキスト）をクロールするにはRを使い、ページURLはhttp://www.xxxxxx.com/search/?q=AlphaGoです。ここに私のコードは次のとおりです。私は私がターゲットにコードを発見したかどうかを確認するためのコード

xpathSApply(parsedpage,"//h3//a",xmlGetAttr,"href") 

を使用する場合

url <- "http://www.xxxxxx.com/search/?q=AlphaGo" 
info <- debugGatherer() 
handle <- getCurlHandle(cookiejar ="", 
         #turn the page 
         followlocation = TRUE, 
         autoreferer = TRUE, 
         debugfunc = info$update, 
         verbose = TRUE, 
         httpheader = list(
          from = "eddie@r-datacollection.com", 
          'user-agent' = str_c(R.version$version.string, 
               ",",R.version$platform) 
         )) 
html <- getURL(url,curl=handle,header = TRUE) 
parsedpage <- htmlParse(html) 

しかし、私は、関連ニュースの情報のすべてのコンテンツが不足していることがわかります。それから、F12には、私が望む情報が含まれていて、sourcesには何も入っていない（実際には、すべての要素が一緒に積み上げられているように、ちょっと混乱している）DOM elements（Chromeは私が使ったもの）だから私は自分のコードを次のように変更します：

parsed_page <- htmlTreeParse(file = url,asTree = T) 

代わりにDOMツリーを取得したいと考えています。 まだ、情報が欠落していますが、私が見つけたものはすべて、DOM elementsで折りたたまれた情報です（前にこのような状況に会ったことはありません）。

どのように問題が発生し、どのように私はこれを修正することができますか？

Answer 1

@Colin提供の考えでは、私は元のコードに沿って続くことを試みました。したがって、パッケージのJSONファイルの動的コンテンツのコードは、RJSONIO

url <- "https://search.xxxxxx.io/content?q=AlphaGo" 
content <- fromJSON(url) 
content1 <- content$result 
content_result <- matrix(NA,10,5) 
for(i in 1:length(content1)){ 
    content_result[i,] <- c("CNN", content1[[i]]$firstPublishDate,ifelse(class(content1[[i]]$headline) != "NULL",content1[[i]]$headline,"NA"), 
         content1[[i]]$body,content1[[i]]$url) 
} 

Answer 2

問題はあなたのコードから来ていません。結果ページは動的に生成されるため、リンクやテキストは結果ページには表示されません（ソースコードを見ると分かります）。

結果が10件しかないので、手動でURLのリストを作成することをお勧めします。

このコードで使用したパッケージについてはわかりません。しかし、私はあなたがrvestに行くことをお勧めします。これはあなたが使ったパッケージよりも単純なようです。

の場合：

url <- "http://money.cnn.com/2017/05/25/technology/alphago-china-ai/index.html" 

library(rvest) 
library(tidyverse) 

url %>% 
    read_html() %>% 
    html_nodes(xpath = '//*[@id="storytext"]/p') %>% 
    html_text() 

[1] " A computer system that Google engineers trained to play the game Go beat the world's best human player Thursday in China. The victory was AlphaGo's second this week over Chinese professional Ke Jie, clinching the best-of-three series at the Future of Go Summit in Wuzhen. "         
[2] " Afterward, Google engineers said AlphaGo estimated that the first 50 moves -- by both players -- were virtually perfect. And the first 100 moves were the best anyone had ever played against AlphaGo's master version. "                       
[3] " Related: Google's man-versus-machine showdown is blocked in China "                                                             
[4] " \"What an amazing and complex game! Ke Jie pushed AlphaGo right to the limit,\" said DeepMind CEO Demis Hassabis on Twitter. DeepMind is a British artificial intelligence company that developed AlphaGo and was purchased by Google in 2014. "                  
[5] " DeepMind made a stir in January 2016 when it first announced it had used artificial intelligence to master Go, a 2,500-year-old game. Computer scientists had struggled for years to get computers to excel at the game. "                       
[6] " In Go, two players alternate placing white and black stones on a grid. The goal is to claim the most territory. To do so, you surround your opponent's pieces so that they're removed from the board. "                            
[7] " The board's 19-by-19 grid is so vast that it allows a near infinite combination of moves, making it tough for machines to comprehend. Games such as chess have come quicker to machines. "                               
[8] " Related: Elon Musk's new plan to save humanity from AI "                                                                
[9] " The Google engineers at DeepMind rely on deep learning, a trendy form of artificial intelligence that's driving remarkable gains in what computers are capable of. World-changing technologies that loom on the horizon, such as autonomous vehicles, rely on deep learning to effectively see and drive on roads. " 
[10] " AlphaGo's achievement is also a reminder of the steady improvement of machines' ability to complete tasks once reserved for humans. As machines get smarter, there are concerns about how society will be disrupted, and if all humans will be able to find work. "             
[11] " Historically, mankind's development of tools has always created new jobs that never existed before. But the gains in artificial intelligence are coming at a breakneck pace, which will likely accentuate upheaval in the short term. "                    
[12] " Related: Google uses AI to help diagnose breast cancer "                                                                
[13] " The 19-year-old Ke and AlphaGo will play a third match Saturday morning. The summit will also feature a match Friday in which five human players will team up against AlphaGo. "  

ベスト

コリン・