0
私は、Livyが0.3.0、SparkバージョンがYarnモデルの1.6.0であるScala ApIでLivy Clientアプリケーションを実行しています。Livy ERROR RSCClient：コンテキストへの接続に失敗しました
ここにメインコードがあります。
def main(args: Array[String]): Unit = {
var scalaClient: LivyScalaClient = null
val livyurl = "http://172.24.4.110:8998"
try {
scalaClient = new LivyClientBuilder(false).setURI(new URI(livyurl)).build().asScalaClient
// scalaClient.uploadFile(new File("logs/spath2.jar"))
val sFuture : ScalaJobHandle[_]= scalaClient.submit(jct => {
//Spark_Log()
val conf1 = jct.sc.getConf
conf1.setAppName("by livy ... ")
val sc = jct.sc
val sqlContext = jct.sqlctx
val dataset: DataFrame = sqlContext.createDataFrame(Seq(
(1, Vectors.dense(0.0, 0.0, 0.0)),
(2, Vectors.dense(0.1, 0.1, 0.1)),
(3, Vectors.dense(0.2, 0.2, 0.2)),
(4, Vectors.dense(9.0, 9.0, 9.0)),
(5, Vectors.dense(9.1, 9.1, 9.1)),
(6, Vectors.dense(9.2, 9.2, 9.2))
)).toDF("id", "features")
dataset.write.save("usr/root/livy")
})
}
finally {
if (scalaClient != null) {
scalaClient.stop(true)
scalaClient = null;
}
}
とリウィウスログに例外：
17/08/24 05:50:09 INFO InteractiveSession$: Creating LivyClient for sessionId: 30
17/08/24 05:50:09 WARN RSCConf: Your hostname, master-172-24-4-110.neucloud-bd.org, resolves to a loopback address, but we couldn't find any external IP address!
17/08/24 05:50:09 WARN RSCConf: Set livy.rsc.rpc.server.address if you need to bind to another address.
17/08/24 05:50:09 INFO InteractiveSessionManager: Registering new session 30
17/08/24 05:50:10 INFO InteractiveSession: Stopping InteractiveSession 30...
17/08/24 05:50:10 INFO RSCClient: Failing pending job 5efb67ea-1b76-4b80-a7b1-4f2527bf023b due to shutdown.
17/08/24 05:50:10 WARN ContextLauncher: Waiting thread interrupted, killing child process.
17/08/24 05:50:10 INFO InteractiveSession: Stopped InteractiveSession 30.
17/08/24 05:50:10 ERROR RSCClient: Failed to connect to context.
java.util.concurrent.CancellationException
17/08/24 05:50:10 WARN InteractiveSession: (Fail to get rsc uri,java.util.concurrent.ExecutionException: java.util.concurrent.CancellationException)
17/08/24 05:51:09 ERROR SparkYarnApp: Error whiling refreshing YARN state: java.lang.Exception: No YARN application is found with tag livy-session-30-lkjckxwd in 60 seconds. Please check your cluster status, it is may be very busy.しかし、それは、Java API