Pythonでセレンを使って新しいWebページを開く方法（古いものを閉じる）

Question

Python 2.7でセレンを使ってWebサイトからいくつかのWebスクレイピングをしたい、ちょっと待ってからブラウザを閉じてgeckodriver.exeを閉じてください。何百万ものブラウザページを開きたくないからです。 exeファイル）

私はこれを行う方法はありますか？

コメントと私のコード：

from bs4 import BeautifulSoup 
 
from selenium import webdriver 
 
import time 
 
import urllib2 
 
import unicodecsv as csv 
 
import os 
 
import sys 
 
import io 
 
import time 
 
import datetime 
 
import pandas as pd 
 
from bs4 import BeautifulSoup 
 
import MySQLdb 
 
import re 
 
import contextlib 
 
import selenium.webdriver.support.ui as ui 
 

 
#I am create a new csv file 
 
filename=r'output.csv' 
 

 
resultcsv=open(filename,"wb") 
 
output=csv.writer(resultcsv, delimiter=';',quotechar = '"', quoting=csv.QUOTE_NONNUMERIC, encoding='latin-1') 
 

 
#I am opening the website with selenium (js website) 
 
profile=webdriver.FirefoxProfile() 
 
profile.set_preference("intl.accept_languages","en-us") 
 
driver = webdriver.Firefox(firefox_profile=profile) 
 
driver.get("https://www.flightradar24.com/data/airports/bud/arrivals") 
 
time.sleep(10) 
 
html_source=driver.page_source 
 
soup=BeautifulSoup(html_source,"html.parser") 
 
print soup 
 

 
#HERE I AM WEBSCRAPING THE INFORMATIONS WHAT I NEEDED AND 
 
#AFTER I AM WRITING IT INTO THIS CSV FILE. 
 
    
 
output.writerows(datatable) 
 
    
 
resultcsv.close() 
 

 
#AND MY QUESTION START HERE. I WANT TO CLOSE THIS SESSEION, 
 
#WAIT A LITTLE, FOR EXAMPLE 10 SEC, BECAUSE IT IS NEEDED SOME TIME TO WEB- 
 
#SCRAPING DATAES AFTER THIS CLOSE THE GECKODRIVER + FIREFOX, AND AFTER 
 
#REPEAT THIS CODE WITH A NEW WEBSITE. IS IT POSSIBLE?

更新コード - nutmeg64

私は、このエラーメッセージが出ます：

File "C:/Python27/air17.py", line 43, in scrape(urls) File "C:/Python27/air17.py", line 28, in scrape table = soup.find('table', { "class" : "table table-condensed table-hover data-table m-n-t-15" }) NameError: global name 'soup' is not defined

from bs4 import BeautifulSoup 
 
from selenium import webdriver 
 
import time 
 
import urllib2 
 
import unicodecsv as csv 
 
import os 
 
import sys 
 
import io 
 
import time 
 
import datetime 
 
import pandas as pd 
 
from bs4 import BeautifulSoup 
 
import MySQLdb 
 
import re 
 
import contextlib 
 
import selenium.webdriver.support.ui as ui 
 

 
filename=r'output.csv' 
 

 
resultcsv=open(filename,"wb") 
 
output=csv.writer(resultcsv, delimiter=';',quotechar = '"', quoting=csv.QUOTE_NONNUMERIC, encoding='latin-1') 
 

 
def scrape(urls): 
 
    browser = webdriver.Firefox() 
 
    for url in urls: 
 
     browser.get(url) 
 
     html = browser.page_source 
 
     soup=BeautifulSoup(html,"html.parser") 
 
     table = soup.find('table', { "class" : "table table-condensed table-hover data-table m-n-t-15" }) 
 
     datatable=[] 
 
     for record in table.find_all('tr', class_="hidden-xs hidden-sm ng-scope"): 
 
      temp_data = [] 
 
      for data in record.find_all("td"): 
 
       temp_data.append(data.text.encode('latin-1')) 
 
      datatable.append(temp_data) 
 
    
 
     output.writerows(datatable) 
 
    
 
     resultcsv.close() 
 
     time.sleep(10) 
 
     browser.quit() 
 

 
urls = ["https://www.flightradar24.com/data/airports/bud/arrivals", "https://www.flightradar24.com/data/airports/fco/arrivals"] 
 
scrape(urls) 
 

を

Answer 1

これはセレンの一部を関数に入れ、別のURLで呼び出すことができます。反復の間に10秒間スリープします。

これは理想的な解決策ではありません。一度セレンを開くだけで、ソースを読んでからbrowser.get(new_url)を実行するだけです。すべての掻き取りが完了したら、browser.quit()を解放してください。例えば

（非常に非常に単純化された）：

def scrape(urls): 
    browser = webdriver.Firefox() 
    for url in urls: 
     browser.get(url) 
     html = browser.page_source 
     # scrape the html as you like 
     # create a csv file for that specific url 
     # write results to csv and close it 
     time.sleep(10) # <-- not really necessary. scraping and writing to csv is a long enough break 
    browser.quit() 

urls = ["http://example.com", "http://notarealwebsite.co.uk", "http://lastwebpagetoscrape.com" ] 
scrape(urls)