2
私はちょうど私のSpark仕事が亡くなった理由を少しgrokkingしているので、私はこの記事の一番下にトレースバックを入れて、洞察力：memoryOverheadが超過していたので私のノードが死んでいたと言うことができる限り。 awscliからこれを設定して、この問題に遭遇しないようにするにはどうすればよいですか？EMRのためにAWSCLIからYARN memoryOverheadを設定する方法
16/05/17 20:20:46 WARN TaskSetManager: Lost task 97.0 in stage 3.0 (TID 9937, ip-172-31-14-59.us-west-2.compute.internal): ExecutorLostFailure (executor 9 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 WARN TaskSetManager: Lost task 60.0 in stage 3.0 (TID 9900, ip-172-31-14-59.us-west-2.compute.internal): ExecutorLostFailure (executor 9 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 WARN TaskSetManager: Lost task 134.0 in stage 3.0 (TID 9974, ip-172-31-14-59.us-west-2.compute.internal): ExecutorLostFailure (executor 9 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 WARN TaskSetManager: Lost task 23.0 in stage 3.0 (TID 9863, ip-172-31-14-59.us-west-2.compute.internal): ExecutorLostFailure (executor 9 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 INFO YarnClientSchedulerBackend: Asked to remove non-existent executor 9
16/05/17 20:20:46 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 ERROR YarnScheduler: Lost executor 15 on ip-172-31-14-46.us-west-2.compute.internal: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 WARN TaskSetManager: Lost task 88.0 in stage 3.0 (TID 9928, ip-172-31-14-46.us-west-2.compute.internal): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 WARN TaskSetManager: Lost task 51.0 in stage 3.0 (TID 9891, ip-172-31-14-46.us-west-2.compute.internal): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 WARN TaskSetManager: Lost task 125.0 in stage 3.0 (TID 9965, ip-172-31-14-46.us-west-2.compute.internal): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 WARN TaskSetManager: Lost task 14.0 in stage 3.0 (TID 9854, ip-172-31-14-46.us-west-2.compute.internal): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 INFO YarnClientSchedulerBackend: Asked to remove non-existent executor 15
16/05/17 20:20:46 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 ERROR YarnScheduler: Lost executor 14 on ip-172-31-14-61.us-west-2.compute.internal: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 WARN TaskSetManager: Lost task 85.0 in stage 3.0 (TID 9925, ip-172-31-14-61.us-west-2.compute.internal): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 WARN TaskSetManager: Lost task 48.0 in stage 3.0 (TID 9888, ip-172-31-14-61.us-west-2.compute.internal): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 WARN TaskSetManager: Lost task 122.0 in stage 3.0 (TID 9962, ip-172-31-14-61.us-west-2.compute.internal): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 WARN TaskSetManager: Lost task 11.0 in stage 3.0 (TID 9851, ip-172-31-14-61.us-west-2.compute.internal): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
1
どのようにAmazon EMRに仕事を提出していますか？ 'spark.yarn.executor.memoryOverhead'を渡すことは、サブミットするジョブの一部です。awscliを通して変更することはできません。 –
spark-submitとpythonファイル – flybonzai
@PawełJurczenko私はspark-shellからも実行しました。 – flybonzai