scrapy-splashで従属ドロップダウンを選択する

Question

私は次のウェブサイトを削っています：https://www.climatempo.com.br/climatologia/558/saopaulo-sp。 2番目のドロップダウンメニューには最初のものに応じて2番目のメニューがあるので、私はscrapyとsplashを使ってscrapy-splashを選択します。

最初に状態を選択し、次に都市を選択することで、場所の変更を自動化する必要があります。 SplashFormRequestを試しましたが、都市リストを変更できません。これは私があなたが絶対にサイトのメニューを使用しなければならない場合のためのセレンを示唆している仕事です

import scrapy 
from scrapy_splash import SplashRequest, SplashFormRequest 


class ExampleSpider(scrapy.Spider): 
    name = 'climatologia' 

    def start_requests(self): 
     urls = ['https://www.climatempo.com.br/climatologia/558/saopaulo-sp'] 
     for url in urls: 
      yield SplashRequest(url=url, callback=self.parse, 
           endpoint='render.html', 
           args={'wait': 0.5},) 

    def parse(self, response): 
     print(response.url) 
     state = response.css("select.slt-geo")[0].css("option::attr(value)").extract() 
     print(state) 

     return SplashFormRequest(response.url, method='POST', 
           formdata={'sel-state-geo': 'SP'}, 
           callback=self.state_selected, 
           args={'wait': 0.5}) 

    def state_selected(self, response): 
     print('\t:+)\t:+)\t:+)\t:+)\t:+)\t:+)') 
     print(response.css("select.slt-geo")[0].css("option::text").extract()) 
     print(response.css("select.slt-geo")[1].css("option::text").extract()) 

Answer 1

：私のクモは、（デバッグのためのプリント）です。 Splashをスクリプト化する唯一の方法は、LUAスクリプトによるものです。実行エンドポイントに送信し、LUAスクリプトを作成する必要があります。あなたが選択しようとしていたオプションを見つけましたが、フォームをどこに提出するのか、またはサイト上でどのように機能するのか分かりませんでした。私は英語に翻訳しなければならなかった。

私の提案は、このようなエンドポイントのブラウザインスペクタで見て、特に興味深い見ているいくつかのいずれかです。

{"success":true,"message":"Resultados encontrados","time":"2017-11-30 16:05:20","totalRows":null,"totalPages":null,"page":null,"data":[{"idlocale":338,"idstate":31,"uf":"AC","state":"Acre","region":"N","latitude":null,"longitude":null},{"idlocale":339,"idstate":49,"uf":"AL","state":"Alagoas","region":"NE","latitude":null,"longitude":null},{"idlocale":340,"idstate":41,"uf":"AM","state":"Amazonas","region":"N","latitude":null,"longitude":null},{"idlocale":341,"idstate":30,"uf":"AP","state":"Amap\u00e1","region":"N","latitude":null,"longitude":null},{"idlocale":342,"idstate":56,"uf":"BA","state":"Bahia","region":"NE","latitude":null,"longitude":null},{"idlocale":343,"idstate":44,"uf":"CE","state":"Cear\u00e1","region":"NE","latitude":null,"longitude":null},{"idlocale":344,"idstate":47,"uf":"DF","state":"Distrito Federal","region":"CO","latitude":null,"longitude":null},{"idlocale":345,"idstate":45,"uf":"ES","state":"Esp\u00edrito Santo","region":"SE","latitude":null,"longitude":null},{"idlocale":346,"idstate":54,"uf":"GO","state":"Goi\u00e1s","region":"CO","latitude":null,"longitude":null},{"idlocale":347,"idstate":52,"uf":"MA","state":"Maranh\u00e3o","region":"NE","latitude":null,"longitude":null},{"idlocale":348,"idstate":53,"uf":"MG","state":"Minas Gerais","region":"SE","latitude":null,"longitude":null},{"idlocale":349,"idstate":39,"uf":"MS","state":"Mato Grosso do Sul","region":"CO","latitude":null,"longitude":null},{"idlocale":350,"idstate":40,"uf":"MT","state":"Mato Grosso","region":"CO","latitude":null,"longitude":null},{"idlocale":351,"idstate":50,"uf":"ND","state":"N\u00e3o Aplic\u00e1vel","region":"ND","latitude":null,"longitude":null},{"idlocale":352,"idstate":55,"uf":"PA","state":"Par\u00e1","region":"N","latitude":null,"longitude":null},{"idlocale":353,"idstate":37,"uf":"PB","state":"Para\u00edba","region":"NE","latitude":null,"longitude":null},{"idlocale":354,"idstate":29,"uf":"PE","state":"Pernambuco","region":"NE","latitude":null,"longitude":null},{"idlocale":355,"idstate":33,"uf":"PI","state":"Piau\u00ed","region":"NE","latitude":null,"longitude":null},{"idlocale":356,"idstate":32,"uf":"PR","state":"Paran\u00e1","region":"S","latitude":null,"longitude":null},{"idlocale":357,"idstate":46,"uf":"RJ","state":"Rio de Janeiro","region":"SE","latitude":null,"longitude":null},{"idlocale":358,"idstate":35,"uf":"RN","state":"Rio Grande do Norte","region":"NE","latitude":null,"longitude":null},{"idlocale":359,"idstate":38,"uf":"RO","state":"Rond\u00f4nia","region":"N","latitude":null,"longitude":null},{"idlocale":360,"idstate":43,"uf":"RR","state":"Roraima","region":"N","latitude":null,"longitude":null},{"idlocale":361,"idstate":48,"uf":"RS","state":"Rio Grande do Sul","region":"S","latitude":null,"longitude":null},{"idlocale":362,"idstate":36,"uf":"SC","state":"Santa Catarina","region":"S","latitude":null,"longitude":null},{"idlocale":363,"idstate":51,"uf":"SE","state":"Sergipe","region":"NE","latitude":null,"longitude":null},{"idlocale":364,"idstate":34,"uf":"SP","state":"S\u00e3o Paulo","region":"SE","latitude":null,"longitude":null},{"idlocale":365,"idstate":42,"uf":"TO","state":"Tocantins","region":"N","latitude":null,"longitude":null}]} 

うまくいけば、これは別であるが、以下のようにこのエンドポイントは、JSONを与える https://www.climatempo.com.br/json/busca-estados

あなたが探しているデータを取得する方法？

通常のリクエストを使用してデータを取得できます。あなたは同じ要求を同じようにしなければなりません。通常、accept、useragent、およびrequested withヘッダーを追加するだけで十分です。