ウェブサイトの強制的なリダイレクト

Question

私はwww.caribbeanjobs.comからリダイレクトされ続けています。私はスパイダーがrobot.txtに従わないようにプログラミングしました。クッキーは無効になりました。メタ= dont_redirectを試しました。他に何ができますか？

これは、以下の私のクモです：

import scrapy 

from tutorial.items import CaribbeanJobsItem 

class CaribbeanJobsSpider(scrapy.Spider): 
     name = "caribbeanjobs" 
     allowed_domains = ["caribbeanjobs.com/"] 
     start_urls = [ 
     "http://www.caribbeanjobs.com/" 
     ] 
     def start_requests(self): 
      for url in self.start_urls: 
       yield scrapy.Request(url, meta={'dont_redirect':True}) 

     def parse(self, response): 
       if ".com" in response.url: 
         from scrapy.shell import inspect_response 
         inspect_response(response, self) 

は、これらは私の設定です：

BOT_NAME = 'tutorial' 

SPIDER_MODULES = ['tutorial.spiders'] 
NEWSPIDER_MODULE = 'tutorial.spiders' 


# Crawl responsibly by identifying yourself (and your website) on the user-agent 
#USER_AGENT = 'tutorial (+http://www.yourdomain.com)' 

# Obey robots.txt rules 
ROBOTSTXT_OBEY = False 

# Configure maximum concurrent requests performed by Scrapy (default: 16) 
#CONCURRENT_REQUESTS = 32 

# Configure a delay for requests for the same website (default: 0) 
# See http://scrapy.readthedocs.org/en/latest/topics/settings.html#download-delay 
# See also autothrottle settings and docs 
DOWNLOAD_DELAY = 3 
# The download delay setting will honor only one of: 
#CONCURRENT_REQUESTS_PER_DOMAIN = 16 
#CONCURRENT_REQUESTS_PER_IP = 16 

# Disable cookies (enabled by default) 
COOKIES_ENABLED = False 

# Disable Telnet Console (enabled by default) 
#TELNETCONSOLE_ENABLED = False 

# Override the default request headers: 
#DEFAULT_REQUEST_HEADERS = { 
# 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 
# 'Accept-Language': 'en', 
#} 

# Enable or disable spider middlewares 
# See http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html 
#SPIDER_MIDDLEWARES = { 
# 'tutorial.middlewares.MyCustomSpiderMiddleware': 543, 
#} 

# Enable or disable downloader middlewares 
# See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html 
#DOWNLOADER_MIDDLEWARES = { 
# 'tutorial.middlewares.MyCustomDownloaderMiddleware': 543, 
#} 

# Enable or disable extensions 
# See http://scrapy.readthedocs.org/en/latest/topics/extensions.html 
#EXTENSIONS = { 
# 'scrapy.extensions.telnet.TelnetConsole': None, 
#} 

# Configure item pipelines 
# See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.html 
#ITEM_PIPELINES = { 
# 'tutorial.pipelines.SomePipeline': 300, 
#} 

# Enable and configure the AutoThrottle extension (disabled by default) 
# See http://doc.scrapy.org/en/latest/topics/autothrottle.html 
#AUTOTHROTTLE_ENABLED = True 
# The initial download delay 
#AUTOTHROTTLE_START_DELAY = 5 
# The maximum download delay to be set in case of high latencies 
#AUTOTHROTTLE_MAX_DELAY = 60 
# The average number of requests Scrapy should be sending in parallel to 
# each remote server 
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0 
# Enable showing throttling stats for every response received: 
#AUTOTHROTTLE_DEBUG = False 

# Enable and configure HTTP caching (disabled by default) 
# See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings 
#HTTPCACHE_ENABLED = True 
#HTTPCACHE_EXPIRATION_SECS = 0 
#HTTPCACHE_DIR = 'httpcache' 
#HTTPCACHE_IGNORE_HTTP_CODES = [] 
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage' 

Answer 1

あなたの設定で明示的USER_AGENTを設定してみてくださいましたか？

http://doc.scrapy.org/en/latest/topics/settings.html#user-agent

このような何かを出発点として働くかもしれない：

USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.102 Safari/537.36"` 

Answer 2

あなたはhandle_http_statusを指定することができます。このリストは、start_urlsで初期化できます。

handle_http_status = ['303', '301']