治療クローラはクロールしていないか、csvファイルに書き込めませんか？

Question

プロジェクト：https://github.com/smalik/resume_scraper

私はコマンドを使用して、このボットを実行します。

sudo scrapy crawl indeed_resume -o items.csv 

問題は、それが（0ページ）適切にクロールされていないかということであるか、何らかの理由でcsvファイルを書き込むことができません。エラーは償却されたモジュールに起因する可能性がありますか？それは実行時にここの端子出力は次のようになります。

/home/shap/Desktop/resume_crawlers/resume_scraper-master/resume_data/spiders/resume_indeed.py:10: ScrapyDeprecationWarning: Module `scrapy.contrib.spiders` is deprecated, use `scrapy.spiders` instead 
from scrapy.contrib.spiders import CrawlSpider, Rule 
/home/shap/Desktop/resume_crawlers/resume_scraper-master/resume_data/spiders/resume_indeed.py:11: ScrapyDeprecationWarning: Module `scrapy.contrib.linkextractors` is deprecated, use `scrapy.linkextractors` instead 
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor 
/home/shap/Desktop/resume_crawlers/resume_scraper-master/resume_data/spiders/resume_indeed.py:11: ScrapyDeprecationWarning: Module `scrapy.contrib.linkextractors.sgml` is deprecated, use `scrapy.linkextractors.sgml` instead 
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor 
/home/shap/Desktop/resume_crawlers/resume_scraper-master/resume_data/spiders/resume_indeed.py:33: ScrapyDeprecationWarning: SgmlLinkExtractor is deprecated and will be removed in future releases. Please use scrapy.linkextractors.LinkExtractor 
rules = (Rule (SgmlLinkExtractor(restrict_xpaths = ('//a[@class="instl confirm-nav next"]')), callback = "parse_item", follow = True),) 
2017-01-09 14:39:22 [scrapy.utils.log] INFO: Scrapy 1.3.0 started (bot: resume_data) 
2017-01-09 14:39:22 [scrapy.utils.log] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'resume_data.spiders', 'FEED_URI': 'items.csv', 'DEPTH_LIMIT': 50, 'SPIDER_MODULES': ['resume_data.spiders'], 'BOT_NAME': 'resume_data', 'AJAXCRAWL_ENABLED': True, 'COOKIES_ENABLED': False, 'FEED_FORMAT': 'csv', 'MEMUSAGE_REPORT': True, 'DOWNLOAD_DELAY': 2} 
2017-01-09 14:39:22 [scrapy.middleware] INFO: Enabled extensions: 
['scrapy.extensions.feedexport.FeedExporter', 
'scrapy.extensions.logstats.LogStats', 
'scrapy.extensions.telnet.TelnetConsole', 
'scrapy.extensions.corestats.CoreStats'] 
2017-01-09 14:39:22 [scrapy.middleware] INFO: Enabled downloader middlewares: 
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 
'scrapy.downloadermiddlewares.retry.RetryMiddleware', 
'scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware', 
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 
'scrapy.downloadermiddlewares.stats.DownloaderStats'] 
2017-01-09 14:39:22 [scrapy.middleware] INFO: Enabled spider middlewares: 
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 
'scrapy.spidermiddlewares.referer.RefererMiddleware', 
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 
'scrapy.spidermiddlewares.depth.DepthMiddleware'] 
2017-01-09 14:39:22 [scrapy.middleware] INFO: Enabled item pipelines: 
[] 
2017-01-09 14:39:22 [scrapy.core.engine] INFO: Spider opened 
2017-01-09 14:39:22 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 
2017-01-09 14:39:22 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023 
2017-01-09 14:39:22 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (307) to <GET https://www.indeed.com/resumes/mechanical-engineer> from <GET http://www.indeed.com/resumes/mechanical-engineer> 
2017-01-09 14:39:24 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (307) to <GET https://www.indeed.com/resumes/mechanical-engineering> from <GET http://www.indeed.com/resumes/mechanical-engineering> 
2017-01-09 14:39:26 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (307) to <GET https://www.indeed.com/resumes/piping-engineer> from <GET http://www.indeed.com/resumes/piping-engineer> 
2017-01-09 14:39:28 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (307) to <GET https://www.indeed.com/resumes/java> from <GET http://www.indeed.com/resumes/java> 
2017-01-09 14:39:30 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (307) to <GET https://www.indeed.com/resumes/SAS> from <GET http://www.indeed.com/resumes/SAS> 
2017-01-09 14:39:32 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (307) to <GET https://www.indeed.com/resumes/javascript> from <GET http://www.indeed.com/resumes/javascript> 
2017-01-09 14:39:35 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (307) to <GET https://www.indeed.com/resumes/marketing-analyst> from <GET http://www.indeed.com/resumes/marketing-analyst> 
2017-01-09 14:39:40 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.indeed.com/resumes/mechanical-engineering> (referer: None) 
2017-01-09 14:39:43 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.indeed.com/resumes/piping-engineer> (referer: None) 
2017-01-09 14:39:45 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.indeed.com/resumes/java> (referer: None) 
2017-01-09 14:39:47 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.indeed.com/resumes/SAS> (referer: None) 
2017-01-09 14:39:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.indeed.com/resumes/javascript> (referer: None) 
2017-01-09 14:39:52 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.indeed.com/resumes/marketing-analyst> (referer: None) 
2017-01-09 14:40:00 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.indeed.com/resumes/mechanical-engineer> (referer: None) 
2017-01-09 14:40:00 [scrapy.core.engine] INFO: Closing spider (finished) 
2017-01-09 14:40:00 [scrapy.statscollectors] INFO: Dumping Scrapy stats: 
{'downloader/request_bytes': 3260, 
'downloader/request_count': 14, 
'downloader/request_method_count/GET': 14, 
'downloader/response_bytes': 127215, 
'downloader/response_count': 14, 
'downloader/response_status_count/200': 7, 
'downloader/response_status_count/307': 7, 
'finish_reason': 'finished', 
'finish_time': datetime.datetime(2017, 1, 9, 22, 40, 0, 662954), 
'log_count/DEBUG': 15, 
'log_count/INFO': 7, 
'response_received_count': 7, 
'scheduler/dequeued': 14, 
'scheduler/dequeued/memory': 14, 
'scheduler/enqueued': 14, 
'scheduler/enqueued/memory': 14, 
'start_time': datetime.datetime(2017, 1, 9, 22, 39, 22, 288258)} 
2017-01-09 14:40:00 [scrapy.core.engine] INFO: Spider closed (finished) 

Answer 1

クモがstart_urlsで定義されたURL以外のものをクロールしていないようです。

あなたのコードの問題は、RulesとLinkextractorsが何も抽出していないことです。
あなたlinkextractorsをテストする良い方法はscrapy shellを使用することです：

$ scrapy shell "https://www.indeed.com/resumes/javascript" 
from scrapy.linkextractors import LinkExtractor 
le = LinkExtractor(restrict_xpaths='//a[contains(@class,"app_link")]') 
le.extract_links(response) 
#< returns 50 scrapy.link.Link objects here> 

の例では、上記の求人へのURLを取得するための正しいXPathを含んでいます。このアプローチを使用してスパイダーを修正することができます。