Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Spark History Server fails to start with new installation

avatar
Contributor
+ BASIC_GC_TUNING_ARGS=
+ case $JAVA_MAJOR in
+ BASIC_GC_TUNING_ARGS=' '
+ CSD_GC_ARGS=' '
+ CSD_JAVA_OPTS+='  '
++ replace_pid -XX:+HeapDumpOnOutOfMemoryError '-XX:HeapDumpPath=/tmp/spark_on_yarn_spar40365358-SPARK_YARN_HISTORY_SERVER-42af0f75a56c8c9b8b467a684_pid{{PID}}.hprof' -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh
++ echo -XX:+HeapDumpOnOutOfMemoryError '-XX:HeapDumpPath=/tmp/spark_on_yarn_spar40365358-SPARK_YARN_HISTORY_SERVER-42af0f75a56c8c9b8b467a684_pid{{PID}}.hprof' -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh
++ sed 's#{{PID}}#29430#g'
+ export 'CSD_JAVA_OPTS=-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/spark_on_yarn_spar40365358-SPARK_YARN_HISTORY_SERVER-42af0f75a56c8c9b8b467a684_pid29430.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh'
+ CSD_JAVA_OPTS='-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/spark_on_yarn_spar40365358-SPARK_YARN_HISTORY_SERVER-42af0f75a56c8c9b8b467a684_pid29430.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh'
+ echo 'Using -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/spark_on_yarn_spar40365358-SPARK_YARN_HISTORY_SERVER-42af0f75a56c8c9b8b467a684_pid29430.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh as CSD_JAVA_OPTS'
+ source_parcel_environment
+ '[' '!' -z /opt/cloudera/parcels/CDH-7.1.4-1.cdh7.1.4.p0.6300266/meta/cdh_env.sh ']'
+ OLD_IFS=' 	
'
+ IFS=:
+ SCRIPT_ARRAY=($SCM_DEFINES_SCRIPTS)
+ DIRNAME_ARRAY=($PARCEL_DIRNAMES)
4 REPLIES 4

avatar
Master Collaborator

@PrernaU From the logs, I see below error

 

Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=spark, access=EXECUTE, inode="/user":mapred:supergroup:drwxrwx---

 

Assign required permission for spark user on /user directory

avatar
Explorer

@PrernaU 
There are 2 issues here the home directory for spark  doesn't exist and a permission issue  switch to the superuser  i.e hdfs or whatever and run the below command

 

$ hdfs dfs -mkdir /user/spark

Then change the ownership 

$ hdfs dfs -chown spark:spark /user/spark

Make sure the permissions are correct for spark 

 You should now be able to restart successfully spark 

 

Hope that helps

 

avatar
Contributor

Permisson seems to be okay

 

[root@cdp1pub ~]# hdfs dfs -ls /user/
Found 4 items
drwx------ - hdfs supergroup 0 2021-03-02 00:42 /user/hdfs
drwxrwx--- - mapred supergroup 0 2021-03-01 09:10 /user/history
drwxr-x--x - spark spark 0 2021-03-10 05:26 /user/spark
drwxr-xr-x - yarn supergroup 0 2021-03-02 00:37 /user/yarn
[root@cdp1pub ~]# hdfs dfs -ls /user/spark
Found 2 items
drwxrwxrwt - spark spark 0 2021-03-10 05:26 /user/spark/applicationHistory
drwxrwxrwt - spark spark 0 2021-03-10 05:26 /user/spark/driverLogs
[root@cdp1pub ~]#

Tried to restart history server it still fails

###################

Logs

Wed Mar 10 20:57:31 EST 2021
JAVA_HOME=/usr/java/jdk1.8.0_232-cloudera
Using -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/spark_on_yarn_spar40365358-SPARK_YARN_HISTORY_SERVER-42af0f75a56c8c9b8b467a684_pid7765.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh as CSD_JAVA_OPTS
Using /var/run/cloudera-scm-agent/process/1546336662-spark_on_yarn-SPARK_YARN_HISTORY_SERVER as conf dir
Using scripts/control.sh as process script
CONF_DIR=/var/run/cloudera-scm-agent/process/1546336662-spark_on_yarn-SPARK_YARN_HISTORY_SERVER
CMF_CONF_DIR=
Wed Mar 10 20:57:31 EST 2021: Running Spark CSD control script...
Wed Mar 10 20:57:31 EST 2021: Detected CDH_VERSION of [7]
Wed Mar 10 20:57:31 EST 2021: Starting Spark History Server
Running [/opt/cloudera/parcels/CDH-7.1.4-1.cdh7.1.4.p0.6300266/lib/spark/bin/spark-class org.apache.spark.deploy.history.HistoryServer --properties-file /var/run/cloudera-scm-agent/process/1546336662-spark_on_yarn-SPARK_YARN_HISTORY_SERVER/spark-conf/spark-history-server.conf]
Wed Mar 10 20:57:38 EST 2021
JAVA_HOME=/usr/java/jdk1.8.0_232-cloudera
Using -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/spark_on_yarn_spar40365358-SPARK_YARN_HISTORY_SERVER-42af0f75a56c8c9b8b467a684_pid8080.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh as CSD_JAVA_OPTS
Using /var/run/cloudera-scm-agent/process/1546336662-spark_on_yarn-SPARK_YARN_HISTORY_SERVER as conf dir
Using scripts/control.sh as process script
CONF_DIR=/var/run/cloudera-scm-agent/process/1546336662-spark_on_yarn-SPARK_YARN_HISTORY_SERVER
CMF_CONF_DIR=
Wed Mar 10 20:57:39 EST 2021: Running Spark CSD control script...
Wed Mar 10 20:57:39 EST 2021: Detected CDH_VERSION of [7]
Wed Mar 10 20:57:39 EST 2021: Starting Spark History Server
Running [/opt/cloudera/parcels/CDH-7.1.4-1.cdh7.1.4.p0.6300266/lib/spark/bin/spark-class org.apache.spark.deploy.history.HistoryServer --properties-file /var/run/cloudera-scm-agent/process/1546336662-spark_on_yarn-SPARK_YARN_HISTORY_SERVER/spark-conf/spark-history-server.conf]
Wed Mar 10 20:57:48 EST 2021
JAVA_HOME=/usr/java/jdk1.8.0_232-cloudera
Using -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/spark_on_yarn_spar40365358-SPARK_YARN_HISTORY_SERVER-42af0f75a56c8c9b8b467a684_pid8375.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh as CSD_JAVA_OPTS
Using /var/run/cloudera-scm-agent/process/1546336662-spark_on_yarn-SPARK_YARN_HISTORY_SERVER as conf dir
Using scripts/control.sh as process script
CONF_DIR=/var/run/cloudera-scm-agent/process/1546336662-spark_on_yarn-SPARK_YARN_HISTORY_SERVER
CMF_CONF_DIR=
Wed Mar 10 20:57:48 EST 2021: Running Spark CSD control script...
Wed Mar 10 20:57:48 EST 2021: Detected CDH_VERSION of [7]
Wed Mar 10 20:57:48 EST 2021: Starting Spark History Server
Running [/opt/cloudera/parcels/CDH-7.1.4-1.cdh7.1.4.p0.6300266/lib/spark/bin/spark-class org.apache.spark.deploy.history.HistoryServer --properties-file /var/run/cloudera-scm-agent/process/1546336662-spark_on_yarn-SPARK_YARN_HISTORY_SERVER/spark-conf/spark-history-server.conf]
Wed Mar 10 20:57:57 EST 2021
JAVA_HOME=/usr/java/jdk1.8.0_232-cloudera
Using -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/spark_on_yarn_spar40365358-SPARK_YARN_HISTORY_SERVER-42af0f75a56c8c9b8b467a684_pid8757.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh as CSD_JAVA_OPTS
Using /var/run/cloudera-scm-agent/process/1546336662-spark_on_yarn-SPARK_YARN_HISTORY_SERVER as conf dir
Using scripts/control.sh as process script
CONF_DIR=/var/run/cloudera-scm-agent/process/1546336662-spark_on_yarn-SPARK_YARN_HISTORY_SERVER
CMF_CONF_DIR=
Wed Mar 10 20:57:57 EST 2021: Running Spark CSD control script...
Wed Mar 10 20:57:57 EST 2021: Detected CDH_VERSION of [7]
Wed Mar 10 20:57:57 EST 2021: Starting Spark History Server
Running [/opt/cloudera/parcels/CDH-7.1.4-1.cdh7.1.4.p0.6300266/lib/spark/bin/spark-class org.apache.spark.deploy.history.HistoryServer --properties-file /var/run/cloudera-scm-agent/process/1546336662-spark_on_yarn-SPARK_YARN_HISTORY_SERVER/spark-conf/spark-history-server.conf]

avatar
Contributor

I modified the permission to 777 for /user

After which I am able to install Spark and able to install history server.

sudo -u hdfs hadoop fs -chmod 777 /user