スクレーパーが起動するときに、そのスレッド自身が呼び出すparseメソッドを呼び出す必要があります。スクレイピングが完了した後に手動で呼び出すことは可能ですか？コメントで示唆したようにspider_closedメソッドの呼び出し解析メソッド
from scrapy import signals
from scrapy.xlib.pydispatch import dispatcher
class MySpider(CrawlSpider):
def __init__(self):
dispatcher.connect(self.spider_closed, signals.spider_closed)
def parse(self, response):
# something here
def spider_closed(self, spider):
# CALL PARSE METHOD AGAIN
、@StevenAlmeroth's answerspider_idleを使用すると、あなたが必要なものと考えられます。
はここで二回再起動する例のクモです：
import scrapy
class IdleRestartSpider(scrapy.Spider):
name = "idlerestart"
restarts = 0
max_restarts = 2
start_urls = ['http://httpbin.org/html']
@classmethod
def from_crawler(cls, crawler, *args, **kwargs):
spider = super(IdleRestartSpider, cls).from_crawler(crawler, *args, **kwargs)
crawler.signals.connect(spider.idle, signal=scrapy.signals.spider_idle)
return spider
def parse(self, response):
self.logger.info("Got response %r" % response)
yield scrapy.Request('http://httpbin.org/get?restarts=%d' % self.restarts,
callback=self.parse_response)
def parse_response(self, response):
self.logger.info("Got response %r" % response)
def idle(self):
self.logger.info("Spider is idle: %d restarts left" % (
self.max_restarts - self.restarts))
if self.restarts < self.max_restarts:
self.logger.info("Spider is restarting")
self.restarts += 1
self.crawler.engine.crawl(
scrapy.Request(self.start_urls[0],
dont_filter=True),
self)
そして、ここでscrapy runspider（scrapy 1.1.0で）使用して出力
$ scrapy runspider idlerestart.py
2016-05-19 10:49:18 [scrapy] INFO: Scrapy 1.1.0 started (bot: scrapybot)
(...)
2016-05-19 10:49:18 [scrapy] INFO: Spider opened
2016-05-19 10:49:18 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-05-19 10:49:18 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-05-19 10:49:19 [scrapy] DEBUG: Crawled (200) <GET http://httpbin.org/html> (referer: None)
2016-05-19 10:49:19 [idlerestart] INFO: Got response <200 http://httpbin.org/html>
2016-05-19 10:49:19 [scrapy] DEBUG: Crawled (200) <GET http://httpbin.org/get?restarts=0> (referer: http://httpbin.org/html)
2016-05-19 10:49:19 [idlerestart] INFO: Got response <200 http://httpbin.org/get?restarts=0>
2016-05-19 10:49:19 [idlerestart] INFO: Spider is idle: 2 restarts left
2016-05-19 10:49:19 [idlerestart] INFO: Spider is restarting
2016-05-19 10:49:19 [scrapy] DEBUG: Crawled (200) <GET http://httpbin.org/html> (referer: None)
2016-05-19 10:49:19 [idlerestart] INFO: Got response <200 http://httpbin.org/html>
2016-05-19 10:49:19 [scrapy] DEBUG: Crawled (200) <GET http://httpbin.org/get?restarts=1> (referer: http://httpbin.org/html)
2016-05-19 10:49:19 [idlerestart] INFO: Got response <200 http://httpbin.org/get?restarts=1>
2016-05-19 10:49:19 [idlerestart] INFO: Spider is idle: 1 restarts left
2016-05-19 10:49:19 [idlerestart] INFO: Spider is restarting
2016-05-19 10:49:20 [scrapy] DEBUG: Crawled (200) <GET http://httpbin.org/html> (referer: None)
2016-05-19 10:49:20 [idlerestart] INFO: Got response <200 http://httpbin.org/html>
2016-05-19 10:49:20 [scrapy] DEBUG: Crawled (200) <GET http://httpbin.org/get?restarts=2> (referer: http://httpbin.org/html)
2016-05-19 10:49:20 [idlerestart] INFO: Got response <200 http://httpbin.org/get?restarts=2>
2016-05-19 10:49:20 [idlerestart] INFO: Spider is idle: 0 restarts left
2016-05-19 10:49:20 [scrapy] INFO: Closing spider (finished)
2016-05-19 10:49:20 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 1410,
'downloader/request_count': 6,
'downloader/request_method_count/GET': 6,
'downloader/response_bytes': 13542,
'downloader/response_count': 6,
'downloader/response_status_count/200': 6,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2016, 5, 19, 8, 49, 20, 438174),
'log_count/DEBUG': 7,
'log_count/INFO': 18,
'request_depth_max': 1,
'response_received_count': 6,
'scheduler/dequeued': 6,
'scheduler/dequeued/memory': 6,
'scheduler/enqueued': 6,
'scheduler/enqueued/memory': 6,
'start_time': datetime.datetime(2016, 5, 19, 8, 49, 18, 865649)}
2016-05-19 10:49:20 [scrapy] INFO: Spider closed (finished)
必要な場合にはあなたが必要なdont_filter=True引数を追加していることを確認します。
あなたは（）を解析するために渡すために応答オブジェクトがありませんようそれはそうは思えないこの
return super(YourSpider, self).parse(response)
を使用して機能を解析するために制御を渡すことができます。あなたは何を達成しようとしていますか？ – Steve
スクレーピングが完了した後にスクレーパーを再起動したい –
この[リンク]をご覧ください（http://doc.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-スクリプト）は、スクリプトから実行中の治療を記述します。 – Steve