Scrapy FormRequestリダイレクト不要リンク

Question

私は基本的なScrapy Loginに従っています。それはいつもうまくいきますが、このケースではいくつか問題がありました。 FormRequest.from_responseはhttps://www.crowdfunder.com/user/validateLoginを要求せず、代わりに常にhttps://www.crowdfunder.com/user/signupにペイロードを送信しました。私はpayloadでvalidateLoginを直接リクエストしようとしましたが、404エラーで応答しました。この問題を解決するのに役立つどんな考えですか？前もって感謝します！！！ここで

class CrowdfunderSpider(InitSpider): 
    name = "crowdfunder" 
    allowed_domains = ["crowdfunder.com"] 
    start_urls = [ 
     'http://www.crowdfunder.com/', 
    ] 

    login_page = 'https://www.crowdfunder.com/user/login/' 
    payload = {} 

    def init_request(self): 
     """This function is called before crawling starts.""" 
     return scrapy.Request(url=self.login_page, callback=self.login) 

    def login(self, response): 
     """Generate a login request.""" 
     self.payload = {'email': 'my_email', 
         'password': 'my_password'} 

     # scrapy login 
     return scrapy.FormRequest.from_response(response, formdata=self.payload, callback=self.check_login_response) 

    def check_login_response(self, response): 
     """Check the response returned by a login request to see if we are 
     successfully logged in. 
     """ 
     if 'https://www.crowdfunder.com/user/settings' == response.url: 
      self.log("Successfully logged in. :) :) :)") 
      # start the crawling 
      return self.initialized() 
     else: 
      # login fail 
      self.log("login failed :(:(:(") 

は、ブラウザでログインをクリックして送信されたペイロードとリクエストのリンクです：

2016-10-21 21:56:21 [scrapy] INFO: Scrapy 1.1.0 started (bot: crowdfunder_crawl) 
2016-10-21 21:56:21 [scrapy] INFO: Overridden settings: {'AJAXCRAWL_ENABLED': True, 'NEWSPIDER_MODULE': 'crowdfunder_crawl.spiders', 'SPIDER_MODULES': ['crowdfunder_crawl.spiders'], 'ROBOTSTXT_OBEY': True, 'BOT_NAME': 'crowdfunder_crawl'} 
2016-10-21 21:56:21 [scrapy] INFO: Enabled extensions: 
['scrapy.extensions.logstats.LogStats', 
'scrapy.extensions.telnet.TelnetConsole', 
'scrapy.extensions.corestats.CoreStats'] 
2016-10-21 21:56:21 [scrapy] INFO: Enabled downloader middlewares: 
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware', 
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 
'scrapy.downloadermiddlewares.retry.RetryMiddleware', 
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 
'scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware', 
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 
'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware', 
'scrapy.downloadermiddlewares.stats.DownloaderStats'] 
2016-10-21 21:56:21 [scrapy] INFO: Enabled spider middlewares: 
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 
'scrapy.spidermiddlewares.referer.RefererMiddleware', 
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 
'scrapy.spidermiddlewares.depth.DepthMiddleware'] 

2016-10-21 21:56:21 [scrapy] INFO: Enabled item pipelines: 
[] 
2016-10-21 21:56:21 [scrapy] INFO: Spider opened 

2016-10-21 21:56:21 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 

2016-10-21 21:56:21 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6024 

2016-10-21 21:56:21 [scrapy] DEBUG: Crawled (200) <GET https://www.crowdfunder.com/robots.txt> (referer: None) 

2016-10-21 21:56:21 [scrapy] DEBUG: Redirecting (301) to <GET http://www.crowdfunder.com/user/login> from <GET https://www.crowdfunder.com/user/login/> 

2016-10-21 21:56:22 [scrapy] DEBUG: Redirecting (301) to <GET https://www.crowdfunder.com/user/login> from <GET http://www.crowdfunder.com/user/login> 

2016-10-21 21:56:22 [scrapy] DEBUG: Crawled (200) <GET https://www.crowdfunder.com/user/login> (referer: None) 

2016-10-21 21:56:23 [scrapy] DEBUG: Crawled (200) <POST https://www.crowdfunder.com/user/signup> (referer: https://www.crowdfunder.com/user/login) 

2016-10-21 21:56:23 [crowdfunder] DEBUG: login failed :(:(:(
2016-10-21 21:56:23 [scrapy] INFO: Closing spider (finished) 
2016-10-21 21:56:23 [scrapy] INFO: Dumping Scrapy stats: 
{'downloader/request_bytes': 1569, 
'downloader/request_count': 5, 
'downloader/request_method_count/GET': 4, 
'downloader/request_method_count/POST': 1, 
'downloader/response_bytes': 16313, 
'downloader/response_count': 5, 
'downloader/response_status_count/200': 3, 
'downloader/response_status_count/301': 2, 
'finish_reason': 'finished', 
'finish_time': datetime.datetime(2016, 10, 22, 4, 56, 23, 232493), 
'log_count/DEBUG': 7, 
'log_count/INFO': 7, 
'request_depth_max': 1, 
'response_received_count': 3, 
'scheduler/dequeued': 4, 
'scheduler/dequeued/memory': 4, 
'scheduler/enqueued': 4, 
'scheduler/enqueued/memory': 4, 
'start_time': datetime.datetime(2016, 10, 22, 4, 56, 21, 180030)} 
2016-10-21 21:56:23 [scrapy] INFO: Spider closed (finished) 

Answer 1

FormRequest.from_response(response)デフォルトで最初のフォームを使用しています。ここでは

payload and request url sent by clicking login button

は、ログ情報がありますそれは見つける。どのフォームがページに表示されているかを確認すると、次のように表示されます：

In : response.xpath("//form") 
Out: 
[<Selector xpath='//form' data='<form action="/user/signup" method="post'>, 
<Selector xpath='//form' data='<form action="/user/login" method="POST"'>, 
<Selector xpath='//form' data='<form action="/user/login" method="post"'>] 

あなたが探しているフォームは1番目ではありません。これを修正する方法は、多くの方法パラメータの1つを使用して、使用するフォームを指定することです。 formxpathを使用して

は最も柔軟性があり、私の個人的な好み：

In : FormRequest.from_response(response, formxpath='//*[contains(@action,"login")]') 
Out: <POST https://www.crowdfunder.com/user/login>