Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Pyspark Yarn prelaunch Error

Highlighted

Pyspark Yarn prelaunch Error

New Contributor

I got the prelaunch error as the following. Any suggestion on this issue will be appreciated. Thank you.


Container: container_e120_1553895037044_45137_01_000001 on strplpa62gp04.fg.rbc.com_45454_1554344301376

LogAggregationType: AGGREGATED

=======================================================================================================

LogType:prelaunch.err

LogLastModifiedTime:Wed Apr 03 22:18:21 -0400 2019

LogLength:1187

LogContents:

/disk/disk2/hadoop/yarn/local/usercache/PU190SRV/appcache/application_1553895037044_45137/container_e120_1553895037044_45137_01_000001/launch_container.sh: line 31: $PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:/etc/hadoop/conf:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:/usr/hdp/current/ext/hadoop/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure:/etc/hadoop/conf:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:/opt/wandisco/fusion/client/lib/*: bad substitution


End of LogType:prelaunch.err

******************************************************************************


Container: container_e120_1553895037044_45137_01_000001 on strplpa62gp04.fg.rbc.com_45454_1554344301376

LogAggregationType: AGGREGATED

=======================================================================================================

LogType:prelaunch.out

LogLastModifiedTime:Wed Apr 03 22:18:21 -0400 2019

LogLength:25

LogContents:

Setting up env variables


End of LogType:prelaunch.out

******************************************************************************


Container: container_e120_1553895037044_45137_02_000001 on strplpa62hb03.fg.rbc.com_45454_1554344301712

LogAggregationType: AGGREGATED

=======================================================================================================

LogType:prelaunch.err

LogLastModifiedTime:Wed Apr 03 22:18:21 -0400 2019

LogLength:1187

LogContents:

/disk/disk5/hadoop/yarn/local/usercache/PU190SRV/appcache/application_1553895037044_45137/container_e120_1553895037044_45137_02_000001/launch_container.sh: line 31: $PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:/etc/hadoop/conf:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:/usr/hdp/current/ext/hadoop/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure:/etc/hadoop/conf:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:/opt/wandisco/fusion/client/lib/*: bad substitution


End of LogType:prelaunch.err

******************************************************************************


Container: container_e120_1553895037044_45137_02_000001 on strplpa62hb03.fg.rbc.com_45454_1554344301712

LogAggregationType: AGGREGATED

=======================================================================================================

LogType:prelaunch.out

LogLastModifiedTime:Wed Apr 03 22:18:21 -0400 2019

LogLength:25

LogContents:

Setting up env variables


End of LogType:prelaunch.out