JSONのサブアイテムとしての治療の収穫アイテム

Question

すべての産出アイテムを2つのリストに分割するにはどうすればよいですか？たとえば、私がアイテムの2つの主要なタイプ - articleとauthorを持っているとしましょう。私はそれらを2つの別々のリストに入れたいと思う。今すぐ出力を取得しています。

[ 
    { 
    "article_title":"foo", 
    "article_published":"1.1.1972", 
    "author": "John Doe" 
    }, 
    { 
    "name": "John Doe", 
    "age": 42, 
    "email": "foo@example.com" 
    } 
] 

どうすればこのように変換できますか？これらを出力するための

{ 
    "articles": [ 
    { 
     "article_title": "foo", 
     "article_published": "1.1.1972", 
     "author": "John Doe" 
    } 
    ], 
    "authors": [ 
    { 
     "name": "John Doe", 
     "age": 42, 
     "email": "foo@example.com" 
    } 
    ] 
} 

私の機能は、これに似た、単純です：

def parse_author(self, response): 
     name = response.css('div.author-info a::text').extract_first() 
     print("Parsing author: {}".format(name)) 

     yield { 
      'author_name': name 
     } 

Answer 1

アイテム別途パ​​イプラインに到達し、この設定をそれに応じて、それぞれを追加します。

items.py

class Article(scrapy.Item): 
    title = scrapy.Field() 
    published = scrapy.Field() 
    author = scrapy.Field() 

class Author(scrapy.Item): 
    name = scrapy.Field() 
    age = scrapy.Field() 

spider.py

def parse(self, response): 

    author = items.Author() 
    author['name'] = response.css('div.author-info a::text').extract_first() 
    print("Parsing author: {}".format(author['name'])) 
    yield author 

    article = items.Article() 
    article['title'] = response.css('article css').extract_first() 
    print("Parsing article: {}".format(article['title'])) 

    yield article 

pipelines.py

process_item(self, item, spider): 
    if isinstance(item, items.Author): 
     # Do something to authors 
    elif isinstance(item, items.Article): 
     # Do something to articles 

---

は、私は、このアーキテクチャかかわら勧め：

[{ 
    "title": "foo", 
    "published": "1.1.1972", 
    "authors": [ 
     { 
     "name": "John Doe", 
     "age": 42, 
     "email": "foo@example.com" 
     }, 
     { 
     "name": "Jane Doe", 
     "age": 21, 
     "email": "bar@example.com" 
     }, 
    ] 
}] 

これは、それが一つの項目ですべてを行かせます。

items.py

class Article(scrapy.Item): 
    title = scrapy.Field() 
    published = scrapy.Field() 
    authors = scrapy.Field() 

spider.py

def parse(self, response): 

    authors = [] 
    author = {} 
    author['name'] = "John Doe" 
    author['age'] = 42 
    author['email'] = "foo@example.com" 
    print("Parsing author: {}".format(author['name'])) 
    authors.append(author) 

    article = items.Article() 
    article['title'] = "foo" 
    article['published'] = "1.1.1972" 
    print("Parsing article: {}".format(article['title'])) 
    article['authors'] = authors 
    yield article 

Answer 2

raw = [ 
    { 
     "article_title":"foo", 
     "article_published":"1.1.1972", 
     "author": "John Doe" 
    }, 
    { 
     "name": "John Doe", 
     "age": 42, 
     "email": "foo@example.com" 
    } 
] 

data = {'articles':[], "authors":[]} 

for a in raw: 

    if 'article_title' in a: 
     data['articles'].extend([ a ]) 

    else: 
     data['articles'].extend([ a ])