パンダデータフレームの前処理文字列データ

Question

私はユーザーレビューデータセットを持っています。このデータセットを読み込んでいますが、分類基準に適合させる前にユーザーのレビューを事前処理しています（停止語、句読点の削除、小文字への変換、挨拶の削除など）。エラーが発生しています。ここに私のコードは次のとおりです。

import pandas as pd 
    import numpy as np 
    df=pd.read_json("C:/Users/ABC/Downloads/Compressed/reviews_Musical_Instruments_5.json/Musical_Instruments_5.json",lines=True) 
    dataset=df.filter(['overall','reviewText'],axis=1) 
    def cleanText(text): 
     """ 
     removes punctuation, stopwords and returns lowercase text in a list 
     of single words 
     """ 
     text = (text.lower() for text in text) 

     from bs4 import BeautifulSoup 
     text = BeautifulSoup(text).get_text() 

     from nltk.tokenize import RegexpTokenizer 
     tokenizer = RegexpTokenizer(r'\w+') 
     text = tokenizer.tokenize(text) 

     from nltk.corpus import stopwords 
     clean = [word for word in text if word not in 
     stopwords.words('english')] 

     return clean 

    dataset['reviewText']=dataset['reviewText'].apply(cleanText) 
    dataset['reviewText'] 

私はこれらのエラー取得しています：

TypeError         Traceback (most recent call last) 
<ipython-input-68-f42f70ec46e5> in <module>() 
----> 1 dataset['reviewText']=dataset['reviewText'].apply(cleanText) 
     2 dataset['reviewText'] 

~\Anaconda3\lib\site-packages\pandas\core\series.py in apply(self, func, convert_dtype, args, **kwds) 
    2353    else: 
    2354     values = self.asobject 
-> 2355     mapped = lib.map_infer(values, f, convert=convert_dtype) 
    2356 
    2357   if len(mapped) and isinstance(mapped[0], Series): 

pandas/_libs/src\inference.pyx in pandas._libs.lib.map_infer() 

<ipython-input-64-5c6792de405c> in cleanText(text) 
    10  from nltk.tokenize import RegexpTokenizer 
    11  tokenizer = RegexpTokenizer(r'\w+') 
---> 12  text = tokenizer.tokenize(text) 
    13 
    14  from nltk.corpus import stopwords 

~\Anaconda3\lib\site-packages\nltk\tokenize\regexp.py in tokenize(self, text) 
    127   # If our regexp matches tokens, use re.findall: 
    128   else: 
--> 129    return self._regexp.findall(text) 
    130 
    131  def span_tokenize(self, text): 

TypeError: expected string or bytes-like object 

と

TypeError         Traceback (most recent call last) 
<ipython-input-70-f42f70ec46e5> in <module>() 
----> 1 dataset['reviewText']=dataset['reviewText'].apply(cleanText) 
     2 dataset['reviewText'] 

~\Anaconda3\lib\site-packages\pandas\core\series.py in apply(self, func, convert_dtype, args, **kwds) 
    2353    else: 
    2354     values = self.asobject 
-> 2355     mapped = lib.map_infer(values, f, convert=convert_dtype) 
    2356 
    2357   if len(mapped) and isinstance(mapped[0], Series): 

pandas/_libs/src\inference.pyx in pandas._libs.lib.map_infer() 

<ipython-input-69-5c6792de405c> in cleanText(text) 
    10  from nltk.tokenize import RegexpTokenizer 
    11  tokenizer = RegexpTokenizer(r'\w+') 
---> 12  text = tokenizer.tokenize(text) 
    13 
    14  from nltk.corpus import stopwords 

~\Anaconda3\lib\site-packages\nltk\tokenize\regexp.py in tokenize(self, text) 
    127   # If our regexp matches tokens, use re.findall: 
    128   else: 
--> 129    return self._regexp.findall(text) 
    130 
    131  def span_tokenize(self, text): 

TypeError: expected string or bytes-like object 

を私のデータについては、この機能で修正候補やデータクリーニングのための新しい機能を提案してください。 。ここで

は私のデータです：

overall reviewText 
0 5 Not much to write about here, but it does exac... 
1 5 The product does exactly as it should and is q... 
2 5 The primary job of this device is to block the... 
3 5 Nice windscreen protects my MXL mic and preven... 
4 5 This pop filter is great. It looks and perform... 
5 5 So good that I bought another one. Love the h... 
6 5 I have used monster cables for years, and with... 
7 3 I now use this cable to run from the output of... 
8 5 Perfect for my Epiphone Sheraton II. Monster ... 
9 5 Monster makes the best cables and a lifetime w... 
10 5 Monster makes a wide array of cables, includin... 
11 4 I got it to have it if I needed it. I have fou... 
12 3 If you are not use to using a large sustaining... 
13 5 I love it, I used this for my Yamaha ypt-230 a... 
14 5 I bought this to use in my home studio to cont... 
15 2 I bought this to use with my keyboard. I wasn'... 

Answer 1

プリント（DF）

overall reviewText 
0 5 Not much to write about here, but it does exac... 
1 5 The product does exactly as it should and is q... 
2 5 The primary job of this device is to block the... 
3 5 Nice windscreen protects my MXL mic and preven... 
4 5 This pop filter is great. It looks and perform... 
5 5 So good that I bought another one. Love the h... 
6 5 I have used monster cables for years, and with... 
7 3 I now use this cable to run from the output of... 
8 5 Perfect for my Epiphone Sheraton II. Monster ... 
9 5 Monster makes the best cables and a lifetime w... 
10 5 Monster makes a wide array of cables, includin... 
11 4 I got it to have it if I needed it. I have fou... 
12 3 If you are not use to using a large sustaining... 
13 5 I love it, I used this for my Yamaha ypt-230 a... 
14 5 I bought this to use in my home studio to cont... 
15 2 I bought this to use with my keyboard. I wasn'... 

句読点や数字

import re 
df.loc[:,"reviewText"] = df.reviewText.apply(lambda x : " ".join(re.findall('[\w]+',x))) 

を削除するには、小文字

df.loc[:,"reviewText"] = df.reviewText.apply(lambda x : str.lower(x)) 

に変換するには3210

ストップワードを削除するには、あなたがストップワードをインストールしたり、独自のストップワードリストを作成し、機能

from stop_words import get_stop_words 
stop_words = get_stop_words('en') 

def remove_stopWords(s): 
    '''For removing stop words 
    ''' 
    s = ' '.join(word for word in s.split() if word not in stop_words) 
    return s 

df.loc[:,"reviewText"] = df.reviewText.apply(lambda x: remove_stopWords(x))