1. ホーム
  2. 質問と答え
  3. YARNを使用しないspark-submit

YARNを使用しないspark-submit

2016-11-07 12 views
3

私は5ノードのhadoopクラスタにYARNを設定しましたが、Sparkも5つのノードすべてに設定されています。私は期待通りにシェルを起動YARNを使用しないspark-submit

spark-shell --master yarn --num-executors 3

これを実行して、RMからリソースを取得するための糸を使用しているとき、私は火花-1.5.0-cdh5.5.0

を使用しています。だから、私はsparkがhadoop confファイルを期待通りに使用していると推測しています。私が行うとき、これは糸上で動作しているときに必要とされていないと信じているマスターを、スパークに接続しようとしている

spark-submit word_count.py --master yarn-cluster --num-executors 3

火花提出します。

16/11/08 00:18:31 INFO util.Utils: Successfully started service 'HTTP file server' on port 47990. 
16/11/08 00:18:31 INFO spark.SparkEnv: Registering OutputCommitCoordinator 
16/11/08 00:18:41 INFO server.Server: jetty-8.y.z-SNAPSHOT 
16/11/08 00:18:41 INFO server.AbstractConnector: Started [email protected]:4040 
16/11/08 00:18:41 INFO util.Utils: Successfully started service 'SparkUI' on port 4040. 
16/11/08 00:18:41 INFO ui.SparkUI: Started SparkUI at http://10.0.0.4:4040 
16/11/08 00:18:41 INFO util.Utils: Copying /home/rshaik26/word_count.py to /tmp/spark-0a5348f8-5ba8-4906-89af-7499054b554e/userFiles-287b5d13-123a-4bd6-9fe3-489af2a502a1/word_count.py 
16/11/08 00:18:41 INFO spark.SparkContext: Added file file:/home/rshaik26/word_count.py at http://10.0.0.4:47990/files/word_count.py with timestamp 1478544521986 
16/11/08 00:18:42 WARN metrics.MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set. 
16/11/08 00:18:42 INFO client.AppClient$ClientEndpoint: Connecting to master spark://ubuntuhdp2:7077... 
16/11/08 00:18:42 WARN client.AppClient$ClientEndpoint: Could not connect to ubuntuhdp2:7077: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[email protected]:7077] 
16/11/08 00:18:42 ERROR akka.ErrorMonitor: AssociationError [akka.tcp://[email protected]:53411] -> [akka.tcp://[email protected]:7077]: Error [Association failed with [akka.tcp://[email protected]:7077]] [ 
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[email protected]:7077] 
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: ubuntuhdp2/10.0.0.4:7077 
] 
akka.event.Logging$Error$NoCause$ 
16/11/08 00:18:42 WARN client.AppClient$ClientEndpoint: Could not connect to ubuntuhdp2:7077: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[email protected]:7077] 
16/11/08 00:18:42 ERROR akka.ErrorMonitor: AssociationError [akka.tcp://[email protected]:53411] -> [akka.tcp://[email protected]:7077]: Error [Association failed with [akka.tcp://[email protected]:7077]] [ 
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[email protected]:7077] 
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: ubuntuhdp2/10.0.0.4:7077 
] 
akka.event.Logging$Error$NoCause$ 
16/11/08 00:18:42 WARN client.AppClient$ClientEndpoint: Could not connect to ubuntuhdp2:7077: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[email protected]:7077] 
16/11/08 00:18:42 ERROR akka.ErrorMonitor: AssociationError [akka.tcp://[email protected]:53411] -> [akka.tcp://[email protected]:7077]: Error [Association failed with [akka.tcp://[email protected]:7077]] [ 
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[email protected]:7077] 
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: ubuntuhdp2/10.0.0.4:7077 
] 
akka.event.Logging$Error$NoCause$ 
16/11/08 00:18:42 WARN client.AppClient$ClientEndpoint: Could not connect to ubuntuhdp2:7077: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[email protected]:7077] 
16/11/08 00:18:42 ERROR akka.ErrorMonitor: AssociationError [akka.tcp://[email protected]:53411] -> [akka.tcp://[email protected]:7077]: Error [Association failed with [akka.tcp://[email protected]:7077]] [ 
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[email protected]:7077] 
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: ubuntuhdp2/10.0.0.4:7077 
] 
akka.event.Logging$Error$NoCause$ 
16/11/08 00:18:42 WARN client.AppClient$ClientEndpoint: Failed to connect to master ubuntuhdp2:7077 
akka.actor.ActorNotFound: Actor not found for: ActorSelection[Actor[akka.tcp://[email protected]:7077/]/user/Master] 
    at akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:66) 
    at akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:64) 
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) 
    at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67) 
    at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82) 
    at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) 
    at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) 
    at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) 
    at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58) 
    at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74) 
    at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110) 
    at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73) 
    at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) 
    at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248) 
    at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:269) 
    at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:512) 
    at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:545) 
    at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:535) 
    at akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:91) 
    at akka.actor.ActorRef.tell(ActorRef.scala:125) 
    at akka.dispatch.Mailboxes$$anon$1$$anon$2.enqueue(Mailboxes.scala:44) 
    at akka.dispatch.QueueBasedMessageQueue$class.cleanUp(Mailbox.scala:438) 
    at akka.dispatch.UnboundedDequeBasedMailbox$MessageQueue.cleanUp(Mailbox.scala:650) 
    at akka.dispatch.Mailbox.cleanUp(Mailbox.scala:309) 
    at akka.dispatch.MessageDispatcher.unregister(AbstractDispatcher.scala:204) 
    at akka.dispatch.MessageDispatcher.detach(AbstractDispatcher.scala:140) 
    at akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:203) 
    at akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:163) 
    at akka.actor.ActorCell.terminate(ActorCell.scala:338) 
    at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:431) 
    at akka.actor.ActorCell.systemInvoke(ActorCell.scala:447) 
    at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:262) 
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:240) 
    at akka.dispatch.Mailbox.run(Mailbox.scala:219) 
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) 
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) 
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) 
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) 
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 
16/11/08 00:19:02 INFO client.AppClient$ClientEndpoint: Connecting to master spark://ubuntuhdp2:7077... 
16/11/08 00:19:02 WARN client.AppClient$ClientEndpoint: Could not connect to ubuntuhdp2:7077: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[email protected]:7077] 
16/11/08 00:19:02 ERROR akka.ErrorMonitor: AssociationError [akka.tcp://[email protected]:53411] -> [akka.tcp://[email protected]:7077]: Error [Association failed with [akka.tcp://[email protected]:7077]] [ 
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[email protected]:7077] 
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: ubuntuhdp2/10.0.0.4:7077 
] 
akka.event.Logging$Error$NoCause$ 
16/11/08 00:19:02 WARN client.AppClient$ClientEndpoint: Could not connect to ubuntuhdp2:7077: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[email protected]:7077] 
16/11/08 00:19:02 ERROR akka.ErrorMonitor: AssociationError [akka.tcp://[email protected]:53411] -> [akka.tcp://[email protected]:7077]: Error [Association failed with [akka.tcp://[email protected]:7077]] [ 
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[email protected]:7077] 
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: ubuntuhdp2/10.0.0.4:7077 
] 
akka.event.Logging$Error$NoCause$ 
16/11/08 00:19:02 WARN client.AppClient$ClientEndpoint: Could not connect to ubuntuhdp2:7077: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[email protected]:7077] 
16/11/08 00:19:02 ERROR akka.ErrorMonitor: AssociationError [akka.tcp://sp[email protected]:53411] -> [akka.tcp://[email protected]:7077]: Error [Association failed with [akka.tcp://[email protected]:7077]] [ 
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[email protected]:7077] 
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: ubuntuhdp2/10.0.0.4:7077 
] 
akka.event.Logging$Error$NoCause$ 
16/11/08 00:19:02 WARN client.AppClient$ClientEndpoint: Could not connect to ubuntuhdp2:7077: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[email protected]:7077] 
16/11/08 00:19:02 ERROR akka.ErrorMonitor: AssociationError [akka.tcp://[email protected]:53411] -> [akka.tcp://[email protected]:7077]: Error [Association failed with [akka.tcp://[email protected]:7077]] [ 
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[email protected]:7077] 
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: ubuntuhdp2/10.0.0.4:7077 
] 
akka.event.Logging$Error$NoCause$ 
16/11/08 00:19:02 WARN client.AppClient$ClientEndpoint: Failed to connect to master ubuntuhdp2:7077 
akka.actor.ActorNotFound: Actor not found for: ActorSelection[Actor[akka.tcp://[email protected]:7077/]/user/Master] 
    at akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:66) 
    at akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:64) 
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) 
    at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67) 
    at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82) 
    at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) 
    at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) 
    at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) 
    at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58) 
    at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74) 
    at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110) 
    at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73) 
    at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) 
    at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248) 
    at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:269) 
    at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:512) 
    at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:545) 
    at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:535) 
    at akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:91) 
    at akka.actor.ActorRef.tell(ActorRef.scala:125) 
    at akka.dispatch.Mailboxes$$anon$1$$anon$2.enqueue(Mailboxes.scala:44) 
    at akka.dispatch.QueueBasedMessageQueue$class.cleanUp(Mailbox.scala:438) 
    at akka.dispatch.UnboundedDequeBasedMailbox$MessageQueue.cleanUp(Mailbox.scala:650) 
    at akka.dispatch.Mailbox.cleanUp(Mailbox.scala:309) 
    at akka.dispatch.MessageDispatcher.unregister(AbstractDispatcher.scala:204) 
    at akka.dispatch.MessageDispatcher.detach(AbstractDispatcher.scala:140) 
    at akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:203) 
    at akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:163) 
    at akka.actor.ActorCell.terminate(ActorCell.scala:338) 
    at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:431) 
    at akka.actor.ActorCell.systemInvoke(ActorCell.scala:447) 
    at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:262) 
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:240) 
    at akka.dispatch.Mailbox.run(Mailbox.scala:219) 
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) 
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) 
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) 
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) 
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

私を助けてください任意の設定ミスがある場合は、ここで

がspark-env.shある

SPARK_JAVA_OPTS=-Dspark.driver.port=53411 
HADOOP_CONF_DIR=/usr/lib/hadoop-2.6.0-cdh5.5.0/etc/hadoop/ 
SPARK_MASTER_IP=ubuntuhdp2 
SPARK_DIST_CLASSPATH=$(hadoop classpath):/usr/lib/hadoop-2.6.0-cdh5.5.0/share/hadoop/tools/lib/*

火花-defaults.confに

spark.master      spark://ubuntuhdp2:7077 
# spark.eventLog.enabled   true 
# spark.eventLog.dir    hdfs://namenode:8021/directory 
    spark.serializer     org.apache.spark.serializer.KryoSerializer 
# spark.driver.memory    5g 
# spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"

出典

2016-11-07 zerocool

+0

'火花defaults.conf'で' spark.master'は（火花submit' 'で' --master'フラグで置き換えられることはありません、なぜあなたが求めていますあなたは 'spark-defaults.conf'に' spark.master'を持っているからです）?? – VladoDemcak

+0

@VladoDemcakこんにちは、はい！シェルモードが糸とどのように機能するのか？ – zerocool

+0

'spark-defaults.conf'ファイルに' spark.master = yarn.client'が必要だと思います。クラスタモードで実行する場合は、 'spark.master = yarn.cluster'を実行します。 – simple

答えて

1

：以下のエラー私は信じている：

あなたは spark launching 「私はword_count.pyを提出したいと私の[application-arguments]は--master yarn-cluster --num-executors 3ている」ので、彼はデフォルトのマスターを選んだ。に言っ

は、次の試してみてください。

spark-submit --master yarn-cluster --num-executors 3 word_count.py

そして、はい、これは糸に火花を提出しなければならないし、あなたが-clusterを定義するとき、アプリケーションが（「糸リソースノード」上）クラスタ上のどこかで実行されます。

出典

2016-11-07 19:27:44 VladoDemcak

0

私はこれを私のシステムで試しました。手がかりは、このログ声明た：

numExecutorsがデフォルトに設定されている

：私が得たエグゼキュータのNUMを増加させたとき、2

をさえ：2

maxExecutorsがデフォルトに設定されています同じ。

ので修正は簡単です：

spark-submit --master yarn-cluster --num-executors 3 word_count.py

出典

2016-11-07 19:33:28 Venkat

関連する問題

 関連する問題