pysparkで破損したレコードが表示された後保存されたjsonファイルからデータフレームを作成

Question

私はsparkフォルダ内のurljson.jsonというjsonファイルにurlからjsonデータを保存しました。そして ルート を示している私は

tesdf1.printSchema() 

怒鳴るフォーマットを実行した後に、この

を持っているこの

path="urljson.json/" 
testdf1=spark.read.json(path) 
testdf1.show() 

ことによってそれ 上のデータフレームを作成するには以下のコードを実行しました| - _corrupt_record：文字列（nullable = true）私は、任意の指導は非常に私は火花を使用しています 2.0

私のJSONデータは次のようになり理解されるであろう、私がそれ

result:[{"BldgID":"1006AVE ","BldgName":"100-6th Avenue SW (Oddfellows)   ","BldgCity":"Calgary    ","BldgState":"AB ","BldgZip":"T2G 2C4 ","BldgAddress1":"100-6th Avenue Southwest    ","BldgAddress2":"ZZZ None","BldgPhone":"4035439600  ","BldgLandlord":"1006AV","BldgLandlordName":"100-6 TH Avenue SW Inc.          ","BldgManager":"AVANDE","BldgManagerName":"Alyssa Van de Vorst   ","BldgManagerType":"Internal","BldgGLA":"34242","BldgEntityID":"1006AVE ","BldgInactive":"N","BldgPropType":"ZZZ None","BldgPropTypeDesc":"ZZZ None","BldgPropSubType":"ZZZ None","BldgPropSubTypeDesc":"ZZZ None","BldgRetailFlag":"N","BldgEntityType":"REIT      ","BldgCityName":"Calgary    ","BldgDistrictName":"Downtown   ","BldgRegionName":"Western Canada         ","BldgAccountantID":"KKAUN  ","BldgAccountantName":"Kendra Kaun     ","BldgAccountantMgrID":"LVALIANT ","BldgAccountantMgrName":"Lorretta Valiant      ","BldgFASBStartDate":"2012-10-24","BldgFASBStartDateStr":"2012-10-24"},{"BldgID":"1007AVE ","BldgName":"100-7th Avenue Southwest-Art Central ","BldgCity":"Calgary    ","BldgState":"AB ","BldgZip":"T2P 0W4 ","BldgAddress1":"100-7th Avenue Southwest    ","BldgAddress2":"ZZZ None","BldgPhone":"4035439600  ","BldgLandlord":"1007AV","BldgLandlordName":"100-7th Avenue SW (Art Central) Inc.      ","BldgManager":"LPATER","BldgManagerName":"Lyndsey Paterson    ","BldgManagerType":"Internal","BldgGLA":"27127","BldgEntityID":"1007AVE ","BldgInactive":"N","BldgPropType":"ZZZ None","BldgPropTypeDesc":"ZZZ None","BldgPropSubType":"ZZZ None","BldgPropSubTypeDesc":"ZZZ None","BldgRetailFlag":"N","BldgEntityType":"Property Under Dev't  ","BldgCityName":"Calgary    ","BldgDistrictName":"Downtown   ","BldgRegionName":"Western Canada         ","BldgAccountantID":"ABRITTON ","BldgAccountantName":"Angie Britton     ","BldgAccountantMgrID":"ZZZ None","BldgAccountantMgrName":"ZZZ None","BldgFASBStartDate":"2011-09-01","BldgFASBStartDateStr":"2011-09-01"},{"BldgID":"100LOMB ","BldgName":"100 Lombard Street      ","BldgCity":"Toronto    ","BldgState":"ON ","BldgZip":"M5C 1M3 ","BldgAddress1":"100 Lombard Street      ","BldgAddress2":"ZZZ None","BldgPhone":"4169779002  ","BldgLandlord":"100LOM","BldgLandlordName":"100 Lombard Street Inc.          ","BldgManager":"TCHALM","BldgManagerName":"Tiffany Chalmers    ","BldgManagerType":"Internal","BldgGLA":"43697.64","BldgEntityID":"100LOMB ","BldgInactive":"N","BldgPropType":"ZZZ None","BldgPropTypeDesc":"ZZZ None","BldgPropSubType":"ZZZ None","BldgPropSubTypeDesc":"ZZZ None","BldgRetailFlag":"N","BldgEntityType":"REIT      ","BldgCityName":"Toronto    ","BldgDistrictName":"Queen - Richmond ","BldgRegionName":"Central Canada         ","BldgAccountantID":"MALLORDE ","BldgAccountantName":"May Ann Allorde    ","BldgAccountantMgrID":"TTSANG ","BldgAccountantMgrName":"Tony Tsang        ","BldgFASBStartDate":"2005-11-01","BldgFASBStartDateStr":"2005-11-01"},{"BldgID":"10190104","BldgName":"10190-104th Street NW-The Metals Buildi ","BldgCity":"Edmonton   ","BldgState":"AB ","BldgZip":"T5J 1A7 ","BldgAddress1":"10190-104st Street SW     ","BldgAddress2":"ZZZ None","BldgPhone":"7804234400  ","BldgLandlord":"10190 ","BldgLandlordName":"10190-104 Street Inc.          ","BldgManager":"NEWWES","BldgManagerName":"New West Enterprise Property ","BldgManagerType":"Third ","BldgGLA":"20447.75","BldgEntityID":"10190104","BldgInactive":"N","BldgPropType":"ZZZ None","BldgPropTypeDesc":"ZZZ None","BldgPropSubType":"ZZZ None","BldgPropSubTypeDesc":"ZZZ None","BldgRetailFlag":"N","BldgEntityType":"REIT      ","BldgCityName":"Edmonton   ","BldgDistrictName":"Edmonton   ","BldgRegionName":"Western Canada         ","BldgAccountantID":"RYANG  ","BldgAccountantName":"Raymond Yang     ","BldgAccountantMgrID":"LVALIANT ","BldgAccountantMgrName":"Lorretta Valiant      ","BldgFASBStartDate":"2011-08-08","BldgFASBStartDateStr":"2011-08-08"}] 

Answer 1

チェックの一部を掲載している非常に大きいこの問題を解決する可能性がどのように

http://jsonlint.com/で提供したJSONの部分がエラー：not a valid JSONになりました。あなたのJSONの入力から、火花のように値スパークJSON入力を得られない場合があります

部分JSONからresult:の取り外しとhttp://jsonlint.com/のチェックインは、あなたのケースでも「結果」を取り除くことvalid JSON

注意しました

http://spark.apache.org/docs/latest/sql-programming-guide.html#json-datasets

JSONデータセット

：JSONの限られた種類をサポートしていますSpark SQLは自動的にJSONデータセットのスキーマを推論し、データセット[行]としてロードできます。この変換は、StringのRDDまたはJSONファイルでSparkSession.read.json（）を使用して行うことができます。

jsonファイルとして提供されるファイルは、標準的なJSONファイルではありません。各行には、独立した有効なJSONオブジェクトが含まれている必要があります。詳細については、改行で区切られたJSONとも呼ばれるJSON Linesテキスト形式を参照してください。その結果、通常複数行のJSONファイルが失敗することがよくあります。