Pythonで大きなテキストファイル（906.2MB）を反復処理する方法は？

Question

これは私がここで私はむしろ、外出先でwhloeファイル以外の塊を通る線でループラインにPythonでitertoolsパッケージからisliceを使用していた

tweets_data=[] 
with open('/home/surya/tweet2.txt','r') as f: 
    while True: 
     next_n_lines= list(islice(f,10000)) 
     if not next_n_lines: 
     break 
     for line in next_n_lines: 
     try: 
      tweet= json.loads(line) 
      tweets_data.append(tweet) 
     except: 
      continue 
print(len(tweets_data)) 

を使用していたコードスニペットです。後でmatplotlibを使って分析するために配列tweets_dataを使用しています。

同じファイルの119.2 MBでコードをテストしましたが、約30秒かかっていますが、906.2 MBファイルでは私のUbuntuがハングアップし、長い時間待ってからプロセスを強制終了します。

これは、私は私のコードで実現しようとしていますがtweet.txtに私が持っているつぶやきの数をFINGしようとしている私のtweet.txtファイルの形式

{"created_at":"Thu Jun 16 15:44:22 +0000 2016","id":743469273415311360,"id_str":"743469273415311360","text":"RT @itvnews: This is how the fans in Lens reacted as Daniel Sturridge scored the winning #ENG goal #ENGWAL #EURO2016\nhttps:\/\/t.co\/7uta44gRh1","source":"\u003ca href=\"http:\/\/twitter.com\/download\/android\" rel=\"nofollow\"\u003eTwitter for Android\u003c\/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":29464323,"id_str":"29464323","name":"Darren Roberts","screen_name":"Darren081969","location":"Kent England","url":null,"description":null,"protected":false,"verified":false,"followers_count":14,"friends_count":49,"listed_count":0,"favourites_count":9,"statuses_count":9,"created_at":"Tue Apr 07 14:58:50 +0000 2009","utc_offset":3600,"time_zone":"London","geo_enabled":false,"lang":"en","contributors_enabled":false,"is_translator":false,"profile_background_color":"1A1B1F","profile_background_image_url":"http:\/\/abs.twimg.com\/images\/themes\/theme9\/bg.gif","profile_background_image_url_https":"https:\/\/abs.twimg.com\/images\/themes\/theme9\/bg.gif","profile_background_tile":false,"profile_link_color":"2FC2EF","profile_sidebar_border_color":"181A1E","profile_sidebar_fill_color":"252429","profile_text_color":"666666","profile_use_background_image":true,"profile_image_url":"http:\/\/pbs.twimg.com\/profile_images\/742699562217943041\/e87qqP_5_normal.jpg","profile_image_url_https":"https:\/\/pbs.twimg.com\/profile_images\/742699562217943041\/e87qqP_5_normal.jpg","profile_banner_url":"https:\/\/pbs.twimg.com\/profile_banners\/29464323\/1465908742","default_profile":false,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"retweeted_status":{"created_at":"Thu Jun 16 15:39:45 +0000 2016","id":743468111282659328,"id_str":"743468111282659328","text":"This is how the fans in Lens reacted as Daniel Sturridge scored the winning #ENG goal #ENGWAL #EURO2016\nhttps:\/\/t.co\/7uta44gRh1","source":"\u003ca href=\"http:\/\/twitter.com\" rel=\"nofollow\"\u003eTwitter Web Client\u003c\/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":21866939,"id_str":"21866939","name":"ITV News","screen_name":"itvnews","location":"United Kingdom","url":"http:\/\/www.itv.com\/news","description":"Breaking news and the biggest stories from the UK and around the world. Bulletins weekdays at 1:30pm, 6:30pm and 10pm on ITV. Email yourstory@itv.com","protected":false,"verified":true,"followers_count":1447643,"friends_count":1371,"listed_count":5097,"favourites_count":245,"statuses_count":142011,"created_at":"Wed Feb 25 13:51:23 +0000 2009","utc_offset":3600,"time_zone":"London","geo_enabled":true,"lang":"en","contributors_enabled":false,"is_translator":false,"profile_background_color":"006473","profile_background_image_url":"http:\/\/pbs.twimg.com\/profile_background_images\/766810747\/a53a8d9148ca55ff82b713b6a9631fe7.jpeg","profile_background_image_url_https":"https:\/\/pbs.twimg.com\/profile_background_images\/766810747\/a53a8d9148ca55ff82b713b6a9631fe7.jpeg","profile_background_tile":true,"profile_link_color":"006473","profile_sidebar_border_color":"FFFFFF","profile_sidebar_fill_color":"DDFFCC","profile_text_color":"333333","profile_use_background_image":true,"profile_image_url":"http:\/\/pbs.twimg.com\/profile_images\/3105933720\/17bab587f151975ca4ae9f6a1cab7c56_normal.png","profile_image_url_https":"https:\/\/pbs.twimg.com\/profile_images\/3105933720\/17bab587f151975ca4ae9f6a1cab7c56_normal.png","profile_banner_url":"https:\/\/pbs.twimg.com\/profile_banners\/21866939\/1398434757","default_profile":false,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"is_quote_status":false,"retweet_count":53,"favorite_count":67,"entities":{"hashtags":[{"text":"ENG","indices":[76,80]},{"text":"ENGWAL","indices":[86,93]},{"text":"EURO2016","indices":[94,103]}],"urls":[{"url":"https:\/\/t.co\/7uta44gRh1","expanded_url":"https:\/\/amp.twimg.com\/v\/a4cdb597-bde4-4523-8cf6-591d823723c8","display_url":"amp.twimg.com\/v\/a4cdb597-bde\u2026","indices":[104,127]}],"user_mentions":[],"symbols":[]},"favorited":false,"retweeted":false,"possibly_sensitive":false,"filter_level":"low","lang":"en"},"is_quote_status":false,"retweet_count":0,"favorite_count":0,"entities":{"hashtags":[{"text":"ENG","indices":[89,93]},{"text":"ENGWAL","indices":[99,106]},{"text":"EURO2016","indices":[107,116]}],"urls":[{"url":"https:\/\/t.co\/7uta44gRh1","expanded_url":"https:\/\/amp.twimg.com\/v\/a4cdb597-bde4-4523-8cf6-591d823723c8","display_url":"amp.twimg.com\/v\/a4cdb597-bde\u2026","indices":[117,140]}],"user_mentions":[{"screen_name":"itvnews","name":"ITV News","id":21866939,"id_str":"21866939","indices":[3,11]}],"symbols":[]},"favorited":false,"retweeted":false,"possibly_sensitive":false,"filter_level":"low","lang":"en","timestamp_ms":"1466091862430"} 

ですファイル。より迅速な反復方法や別のアプローチの提案は大歓迎です。事前に感謝します。

Answer 1

メモリ使用量が問題である場合は、ファイル自体を反復：

for line in f.readlines(): 
    # your code here