CSVファイルに書き込むときのUnicodeの問題

Question

私はいくつかのガイダンスが必要です。

import requests 
import bs4 
import csv 

results = requests.get('http://grad-schools.usnews.rankingsandreviews.com/best-graduate-schools/top-engineering-schools/eng-rankings?int=a74509') 

reqSoup = bs4.BeautifulSoup(results.text, "html.parser") 
i = 0 
schools = [] 

for school in reqSoup: 
    x = reqSoup.find_all("a", {"class" : "school-name"}) 
    while i < len(x): 
     for name in x: 
      y = x[i].get_text() 
      i += 1 
      schools.append(y) 

with open('usnwr_schools.csv', 'wb') as f: 
    writer = csv.writer(f) 
     for y in schools: 
     writer.writerow([y]) 

私の問題は、全角ダッシュは、結果のCSVファイルでUTF-8として表示されているということです。私は、次のコードを使用しています。私はそれを修正するためにいくつかの異なるものを試しましたが、何も動作しない（それを取り除くためにattempting to use regexを含むだけでなく、数年前に.translate method that I found in a StackOverflow質問をしようとしているようです）。

私には何が欠けていますか？私は、csvの結果にダッシュを除いたテキストを含めるだけです。

私はPython 3.5を使用しています。これはPythonにとってかなり新しいものです。

Answer 1

ダッシュを除去するためだけASCII-コードポイントを使用する場合は

（最初のものはマイナスに全角ダッシュで、二番目は、エンダッシュマイナスにある）y.replace("—","-").replace("–","-")を試してみてください、あなたは

import string 
whitelist=string.printable+string.whitespace 
def clean(s): 
    return "".join(c for c in s if c in whitelist) 

と他のすべてを削除することができます

（これが唯一の純粋な英語のテキストのほとんどは、合理的な結果が得られます）

ところで

open('usnwr_schools.csv', 'w', newline='', encoding='utf-8') # or whatever encoding you like 

を使用してみてください

ためのPython 3にcsv.writerは、テキストを取ることは、Python 2で行ったように、バイナリファイルではなく（あなたがバイナリモード（"wb"）でそれを開いた）

Answer 2

Unicodeを受け入れることを学ぶ...世界はもうASCIIではありません。

あなたがWindows上にあり、.CSVをExcelまたはメモ帳で表示していると仮定すると、Python 3では次の行を使用します。この変更だけで（あなたの投稿のインデントを固定して） ASCII文字を正しく入力してください。メモ帳とExcelは、ファイルの先頭にUTF-8 BOM署名があり、utf-8-sigが提供しています。

with open('usnwr_schools.csv', 'w', newline='', encoding='utf-8-sig') as f: 

別のPythonスクリプトでファイルを読む場合は、必ず次のファイルを読んでください。 b'University of Michigan\xe2\x80\x94\xe2\x80\x8bAnn Arbor'を読んだあなたの例はバイナリモード'rb'で読まれました。

with open('usnwr_schools.csv', encoding='utf-8-sig') as f: 

Linuxの場合は、utf-8-sigの代わりにutf8を使用することができます。余談として

、あなたがあなたのループを置き換えることができます。

with open('usnwr_schools.csv', 'w', newline='', encoding='utf-8-sig') as f: 
    writer = csv.writer(f) 
    for school in reqSoup: 
     x = reqSoup.find_all("a", {"class" : "school-name"}) 
     for item in x: 
      y = item.get_text() 
      writer.writerow([y]) 

それを裏読み：

with open('usnwr_schools.csv',encoding='utf-8-sig') as f: 
    print(f.read()) 

出力：あなたはまだASCIIのみになりたい場合は

Massachusetts Institute of Technology 
Stanford University 
University of California—​Berkeley 
California Institute of Technology 
Carnegie Mellon University 
University of Michigan—​Ann Arbor 
Georgia Institute of Technology 
University of Illinois—​Urbana-​Champaign 
Purdue University—​West Lafayette 
University of Texas—​Austin (Cockrell) 
Texas A&M; University—​College Station (Look) 
Cornell University 
University of Southern California (Viterbi) 
Columbia University (Fu Foundation) 
University of California—​Los Angeles (Samueli) 
University of California—​San Diego (Jacobs) 
Princeton University 
Northwestern University (McCormick) 
University of Pennsylvania 
Johns Hopkins University (Whiting) 
Virginia Tech 
University of California—​Santa Barbara 
Harvard University 
University of Maryland—​College Park (Clark) 
University of Washington 
Massachusetts Institute of Technology 
Stanford University 
University of California—​Berkeley 
California Institute of Technology 
Carnegie Mellon University 
University of Michigan—​Ann Arbor 
Georgia Institute of Technology 
University of Illinois—​Urbana-​Champaign 
Purdue University—​West Lafayette 
University of Texas—​Austin (Cockrell) 
Texas A&M; University—​College Station (Look) 
Cornell University 
University of Southern California (Viterbi) 
Columbia University (Fu Foundation) 
University of California—​Los Angeles (Samueli) 
University of California—​San Diego (Jacobs) 
Princeton University 
Northwestern University (McCormick) 
University of Pennsylvania 
Johns Hopkins University (Whiting) 
Virginia Tech 
University of California—​Santa Barbara 
Harvard University 
University of Maryland—​College Park (Clark) 
University of Washington 
Massachusetts Institute of Technology 
Stanford University 
University of California—​Berkeley 
California Institute of Technology 
Carnegie Mellon University 
University of Michigan—​Ann Arbor 
Georgia Institute of Technology 
University of Illinois—​Urbana-​Champaign 
Purdue University—​West Lafayette 
University of Texas—​Austin (Cockrell) 
Texas A&M; University—​College Station (Look) 
Cornell University 
University of Southern California (Viterbi) 
Columbia University (Fu Foundation) 
University of California—​Los Angeles (Samueli) 
University of California—​San Diego (Jacobs) 
Princeton University 
Northwestern University (McCormick) 
University of Pennsylvania 
Johns Hopkins University (Whiting) 
Virginia Tech 
University of California—​Santa Barbara 
Harvard University 
University of Maryland—​College Park (Clark) 
University of Washington 
Massachusetts Institute of Technology 
Stanford University 
University of California—​Berkeley 
California Institute of Technology 
Carnegie Mellon University 
University of Michigan—​Ann Arbor 
Georgia Institute of Technology 
University of Illinois—​Urbana-​Champaign 
Purdue University—​West Lafayette 
University of Texas—​Austin (Cockrell) 
Texas A&M; University—​College Station (Look) 
Cornell University 
University of Southern California (Viterbi) 
Columbia University (Fu Foundation) 
University of California—​Los Angeles (Samueli) 
University of California—​San Diego (Jacobs) 
Princeton University 
Northwestern University (McCormick) 
University of Pennsylvania 
Johns Hopkins University (Whiting) 
Virginia Tech 
University of California—​Santa Barbara 
Harvard University 
University of Maryland—​College Park (Clark) 
University of Washington 
Massachusetts Institute of Technology 
Stanford University 
University of California—​Berkeley 
California Institute of Technology 
Carnegie Mellon University 
University of Michigan—​Ann Arbor 
Georgia Institute of Technology 
University of Illinois—​Urbana-​Champaign 
Purdue University—​West Lafayette 
University of Texas—​Austin (Cockrell) 
Texas A&M; University—​College Station (Look) 
Cornell University 
University of Southern California (Viterbi) 
Columbia University (Fu Foundation) 
University of California—​Los Angeles (Samueli) 
University of California—​San Diego (Jacobs) 
Princeton University 
Northwestern University (McCormick) 
University of Pennsylvania 
Johns Hopkins University (Whiting) 
Virginia Tech 
University of California—​Santa Barbara 
Harvard University 
University of Maryland—​College Park (Clark) 
University of Washington 

これでやります：

import requests 
import bs4 
import csv 

results = requests.get('http://grad-schools.usnews.rankingsandreviews.com/best-graduate-schools/top-engineering-schools/eng-rankings?int=a74509') 

replacements = {ord('\N{EN DASH}'):'-', 
       ord('\N{EM DASH}'):'-', 
       ord('\N{ZERO WIDTH SPACE}'):None} 

reqSoup = bs4.BeautifulSoup(results.text, "html.parser") 

with open('usnwr_schools.csv', 'w', newline='', encoding='ascii') as f: 
    writer = csv.writer(f) 
    for school in reqSoup: 
     x = reqSoup.find_all("a", {"class" : "school-name"}) 
     for item in x: 
      y = item.get_text() 
      writer.writerow([y.translate(replacements)]) 

with open('usnwr_schools.csv',encoding='ascii') as f: 
    print(f.read())