スパークランダムフォレストトレーニングStackOverflowエラー

Question

モデルのトレーニングを実行していて、maxDepthを12以上に増やすとStackOverflowエラーが発生します。 すべてが5,10,11で正しく動作します。

私はspark 2.0.2を使用しています（今後数週間はアップグレードできません）。 私は3M以上のデータ、200個の特徴、2500本の木を持っています。最大の深さを増やして精度を向上させたいと思います。この問題を克服する方法はありますか？

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 92 in stage 789.0 failed 4 times, most recent failure: Lost task 92.3 in stage 789.0 (TID 66903, 10.0.0.11): java.lang.StackOverflowError 
at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:3069) 
at java.io.ObjectInputStream.readHandle(ObjectInputStream.java:1643) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1515) 
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) 
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) 
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) 
at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:479) 
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:498) 
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2122) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) 
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) 
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) 
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) 
at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:479) 
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:498) 
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2122) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) 
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) 
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) 
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) 
at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:479) 
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:498) 
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2122) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) 
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) 
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) 
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) 
at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:479) 
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:498) 

Answer 1

は火花提出オプションを指定するようになっています

--conf spark.executor.extraJavaOptions=-Xss8m --conf spark.driver.extraJavaOptions=-Xss8m 

は仕事をしていません。

これはまた、その詳細については、各ノードのspark-env.shのconf /に設定することができます。 http://spark.apache.org/docs/latest/configuration.html