Support Questions
Find answers, ask questions, and share your expertise

Pyspark Yarn prelaunch Error

Pyspark Yarn prelaunch Error

Explorer

I got the prelaunch error as the following. Any suggestion on this issue will be appreciated. Thank you.


Container: container_e120_1553895037044_45137_01_000001 on strplpa62gp04.fg.rbc.com_45454_1554344301376

LogAggregationType: AGGREGATED

=======================================================================================================

LogType:prelaunch.err

LogLastModifiedTime:Wed Apr 03 22:18:21 -0400 2019

LogLength:1187

LogContents:

/disk/disk2/hadoop/yarn/local/usercache/PU190SRV/appcache/application_1553895037044_45137/container_e120_1553895037044_45137_01_000001/launch_container.sh: line 31: $PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:/etc/hadoop/conf:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:/usr/hdp/current/ext/hadoop/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure:/etc/hadoop/conf:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:/opt/wandisco/fusion/client/lib/*: bad substitution


End of LogType:prelaunch.err

******************************************************************************


Container: container_e120_1553895037044_45137_01_000001 on strplpa62gp04.fg.rbc.com_45454_1554344301376

LogAggregationType: AGGREGATED

=======================================================================================================

LogType:prelaunch.out

LogLastModifiedTime:Wed Apr 03 22:18:21 -0400 2019

LogLength:25

LogContents:

Setting up env variables


End of LogType:prelaunch.out

******************************************************************************


Container: container_e120_1553895037044_45137_02_000001 on strplpa62hb03.fg.rbc.com_45454_1554344301712

LogAggregationType: AGGREGATED

=======================================================================================================

LogType:prelaunch.err

LogLastModifiedTime:Wed Apr 03 22:18:21 -0400 2019

LogLength:1187

LogContents:

/disk/disk5/hadoop/yarn/local/usercache/PU190SRV/appcache/application_1553895037044_45137/container_e120_1553895037044_45137_02_000001/launch_container.sh: line 31: $PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:/etc/hadoop/conf:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:/usr/hdp/current/ext/hadoop/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure:/etc/hadoop/conf:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:/opt/wandisco/fusion/client/lib/*: bad substitution


End of LogType:prelaunch.err

******************************************************************************


Container: container_e120_1553895037044_45137_02_000001 on strplpa62hb03.fg.rbc.com_45454_1554344301712

LogAggregationType: AGGREGATED

=======================================================================================================

LogType:prelaunch.out

LogLastModifiedTime:Wed Apr 03 22:18:21 -0400 2019

LogLength:25

LogContents:

Setting up env variables


End of LogType:prelaunch.out