Spark Scala CSV列名を小文字に変換する

Question

以下のコードを見て、列名を小文字に変更する方法を教えてください。私はwithColumnRenameを試しましたが、私はそれを各列に対して行い、すべての列名を入力しなければなりません。私はちょうど列でそれをしたいので、あまりにも多くの列名があるので、すべての列名について言及したくありません。

Scalaのバージョン：2.11 スパーク：アウト必要な2.2

import org.apache.spark.sql.SparkSession 
import org.apache.log4j.{Level, Logger} 
import com.datastax 


import org.apache.spark.SparkContext 
import org.apache.spark.SparkConf 
import com.datastax.spark.connector._ 
import org.apache.spark.sql._ 

object dataframeset { 

    def main(args: Array[String]): Unit = { 

    val conf = new SparkConf().setAppName("Sample1").setMaster("local[*]") 
    val sc = new SparkContext(conf) 
    sc.setLogLevel("ERROR") 
    val rdd1 = sc.cassandraTable("tdata", "map3") 
    Logger.getLogger("org").setLevel(Level.ERROR) 
    Logger.getLogger("akka").setLevel(Level.ERROR) 
    val spark1 = org.apache.spark.sql.SparkSession.builder().master("local").config("spark.cassandra.connection.host","127.0.0.1") 
     .appName("Spark SQL basic example").getOrCreate() 

    val df = spark1.read.format("csv").option("header","true").option("inferschema", "true").load("/Users/Desktop/del2.csv") 
    import spark1.implicits._ 
    println("\nTop Records are:") 
    df.show(1) 


    val dfprev1 = df.select(col = "sno", "year", "StateAbbr") 

    dfprev1.show(1) 
} 
} 

プット：

|sno|year|stateabbr| statedesc|cityname|geographiclevel 

All the Columns names should be in lower case. 

実際の出力：

Top Records are: 
+---+----+---------+-------------+--------+---------------+----------+----------+--------+--------------------+---------------+---------------+--------------------+----------+--------------------+---------------------+--------------------------+-------------------+---------------+-----------+----------+---------+--------+---------+-------------------+ 
|sno|year|StateAbbr| StateDesc|CityName|GeographicLevel|DataSource| category|UniqueID|    Measure|Data_Value_Unit|DataValueTypeID|  Data_Value_Type|Data_Value|Low_Confidence_Limit|High_Confidence_Limit|Data_Value_Footnote_Symbol|Data_Value_Footnote|PopulationCount|GeoLocation|categoryID|MeasureId|cityFIPS|TractFIPS|Short_Question_Text| 
+---+----+---------+-------------+--------+---------------+----------+----------+--------+--------------------+---------------+---------------+--------------------+----------+--------------------+---------------------+--------------------------+-------------------+---------------+-----------+----------+---------+--------+---------+-------------------+ 
| 1|2014|  US|United States| null|    US|  BRFSS|Prevention|  59|Current lack of h...|    %|  AgeAdjPrv|Age-adjusted prev...|  14.9|    14.6|     15.2|      null|    null|  308745538|  null| PREVENT| ACCESS2| null|  null| Health Insurance| 
+---+----+---------+-------------+--------+---------------+----------+----------+--------+--------------------+---------------+---------------+--------------------+----------+--------------------+---------------------+--------------------------+-------------------+---------------+-----------+----------+---------+--------+---------+-------------------+ 
only showing top 1 row 

+---+----+---------+ 
|sno|year|StateAbbr| 
+---+----+---------+ 
| 1|2014|  US| 
+---+----+---------+ 
only showing top 1 row 

Answer 1

だけtoDFを使用します。

df.toDF(df.columns map(_.toLowerCase): _*) 

Answer 2

他の方法では、FoldLeftメソッドを使用しています。

val myDFcolNames = myDF.columns.toList 
val rdoDenormDF = myDFcolNames.foldLeft(myDF)((myDF, c) => 
    myDF.withColumnRenamed(c.toString.split(",")(0), c.toString.toLowerCase()))