Spark History Server fails to start with new installation

[#1]

+ BASIC_GC_TUNING_ARGS=
+ case $JAVA_MAJOR in
+ BASIC_GC_TUNING_ARGS=' '
+ CSD_GC_ARGS=' '
+ CSD_JAVA_OPTS+='  '
++ replace_pid -XX:+HeapDumpOnOutOfMemoryError '-XX:HeapDumpPath=/tmp/spark_on_yarn_spar40365358-SPARK_YARN_HISTORY_SERVER-42af0f75a56c8c9b8b467a684_pid{{PID}}.hprof' -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh
++ echo -XX:+HeapDumpOnOutOfMemoryError '-XX:HeapDumpPath=/tmp/spark_on_yarn_spar40365358-SPARK_YARN_HISTORY_SERVER-42af0f75a56c8c9b8b467a684_pid{{PID}}.hprof' -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh
++ sed 's#{{PID}}#29430#g'
+ export 'CSD_JAVA_OPTS=-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/spark_on_yarn_spar40365358-SPARK_YARN_HISTORY_SERVER-42af0f75a56c8c9b8b467a684_pid29430.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh'
+ CSD_JAVA_OPTS='-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/spark_on_yarn_spar40365358-SPARK_YARN_HISTORY_SERVER-42af0f75a56c8c9b8b467a684_pid29430.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh'
+ echo 'Using -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/spark_on_yarn_spar40365358-SPARK_YARN_HISTORY_SERVER-42af0f75a56c8c9b8b467a684_pid29430.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh as CSD_JAVA_OPTS'
+ source_parcel_environment
+ '[' '!' -z /opt/cloudera/parcels/CDH-7.1.4-1.cdh7.1.4.p0.6300266/meta/cdh_env.sh ']'
+ OLD_IFS=' 	
'
+ IFS=:
+ SCRIPT_ARRAY=($SCM_DEFINES_SCRIPTS)
+ DIRNAME_ARRAY=($PARCEL_DIRNAMES)

[#2]

@PrernaU From the logs, I see below error

Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=spark, access=EXECUTE, inode="/user":mapred:supergroup:drwxrwx---

Assign required permission for spark user on /user directory

[#3]

@PrernaU 
There are 2 issues here the home directory for spark  doesn't exist and a permission issue  switch to the superuser  i.e hdfs or whatever and run the below command

$ hdfs dfs -mkdir /user/spark

Then change the ownership 

$ hdfs dfs -chown spark:spark /user/spark

Make sure the permissions are correct for spark 

 You should now be able to restart successfully spark 

Hope that helps

[#4]

Permisson seems to be okay

[root@cdp1pub ~]# hdfs dfs -ls /user/
Found 4 items
drwx------ - hdfs supergroup 0 2021-03-02 00:42 /user/hdfs
drwxrwx--- - mapred supergroup 0 2021-03-01 09:10 /user/history
drwxr-x--x - spark spark 0 2021-03-10 05:26 /user/spark
drwxr-xr-x - yarn supergroup 0 2021-03-02 00:37 /user/yarn
[root@cdp1pub ~]# hdfs dfs -ls /user/spark
Found 2 items
drwxrwxrwt - spark spark 0 2021-03-10 05:26 /user/spark/applicationHistory
drwxrwxrwt - spark spark 0 2021-03-10 05:26 /user/spark/driverLogs
[root@cdp1pub ~]#

Tried to restart history server it still fails

###################

Logs

Wed Mar 10 20:57:31 EST 2021
JAVA_HOME=/usr/java/jdk1.8.0_232-cloudera
Using -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/spark_on_yarn_spar40365358-SPARK_YARN_HISTORY_SERVER-42af0f75a56c8c9b8b467a684_pid7765.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh as CSD_JAVA_OPTS
Using /var/run/cloudera-scm-agent/process/1546336662-spark_on_yarn-SPARK_YARN_HISTORY_SERVER as conf dir
Using scripts/control.sh as process script
CONF_DIR=/var/run/cloudera-scm-agent/process/1546336662-spark_on_yarn-SPARK_YARN_HISTORY_SERVER
CMF_CONF_DIR=
Wed Mar 10 20:57:31 EST 2021: Running Spark CSD control script...
Wed Mar 10 20:57:31 EST 2021: Detected CDH_VERSION of [7]
Wed Mar 10 20:57:31 EST 2021: Starting Spark History Server
Running [/opt/cloudera/parcels/CDH-7.1.4-1.cdh7.1.4.p0.6300266/lib/spark/bin/spark-class org.apache.spark.deploy.history.HistoryServer --properties-file /var/run/cloudera-scm-agent/process/1546336662-spark_on_yarn-SPARK_YARN_HISTORY_SERVER/spark-conf/spark-history-server.conf]
Wed Mar 10 20:57:38 EST 2021
JAVA_HOME=/usr/java/jdk1.8.0_232-cloudera
Using -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/spark_on_yarn_spar40365358-SPARK_YARN_HISTORY_SERVER-42af0f75a56c8c9b8b467a684_pid8080.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh as CSD_JAVA_OPTS
Using /var/run/cloudera-scm-agent/process/1546336662-spark_on_yarn-SPARK_YARN_HISTORY_SERVER as conf dir
Using scripts/control.sh as process script
CONF_DIR=/var/run/cloudera-scm-agent/process/1546336662-spark_on_yarn-SPARK_YARN_HISTORY_SERVER
CMF_CONF_DIR=
Wed Mar 10 20:57:39 EST 2021: Running Spark CSD control script...
Wed Mar 10 20:57:39 EST 2021: Detected CDH_VERSION of [7]
Wed Mar 10 20:57:39 EST 2021: Starting Spark History Server
Running [/opt/cloudera/parcels/CDH-7.1.4-1.cdh7.1.4.p0.6300266/lib/spark/bin/spark-class org.apache.spark.deploy.history.HistoryServer --properties-file /var/run/cloudera-scm-agent/process/1546336662-spark_on_yarn-SPARK_YARN_HISTORY_SERVER/spark-conf/spark-history-server.conf]
Wed Mar 10 20:57:48 EST 2021
JAVA_HOME=/usr/java/jdk1.8.0_232-cloudera
Using -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/spark_on_yarn_spar40365358-SPARK_YARN_HISTORY_SERVER-42af0f75a56c8c9b8b467a684_pid8375.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh as CSD_JAVA_OPTS
Using /var/run/cloudera-scm-agent/process/1546336662-spark_on_yarn-SPARK_YARN_HISTORY_SERVER as conf dir
Using scripts/control.sh as process script
CONF_DIR=/var/run/cloudera-scm-agent/process/1546336662-spark_on_yarn-SPARK_YARN_HISTORY_SERVER
CMF_CONF_DIR=
Wed Mar 10 20:57:48 EST 2021: Running Spark CSD control script...
Wed Mar 10 20:57:48 EST 2021: Detected CDH_VERSION of [7]
Wed Mar 10 20:57:48 EST 2021: Starting Spark History Server
Running [/opt/cloudera/parcels/CDH-7.1.4-1.cdh7.1.4.p0.6300266/lib/spark/bin/spark-class org.apache.spark.deploy.history.HistoryServer --properties-file /var/run/cloudera-scm-agent/process/1546336662-spark_on_yarn-SPARK_YARN_HISTORY_SERVER/spark-conf/spark-history-server.conf]
Wed Mar 10 20:57:57 EST 2021
JAVA_HOME=/usr/java/jdk1.8.0_232-cloudera
Using -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/spark_on_yarn_spar40365358-SPARK_YARN_HISTORY_SERVER-42af0f75a56c8c9b8b467a684_pid8757.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh as CSD_JAVA_OPTS
Using /var/run/cloudera-scm-agent/process/1546336662-spark_on_yarn-SPARK_YARN_HISTORY_SERVER as conf dir
Using scripts/control.sh as process script
CONF_DIR=/var/run/cloudera-scm-agent/process/1546336662-spark_on_yarn-SPARK_YARN_HISTORY_SERVER
CMF_CONF_DIR=
Wed Mar 10 20:57:57 EST 2021: Running Spark CSD control script...
Wed Mar 10 20:57:57 EST 2021: Detected CDH_VERSION of [7]
Wed Mar 10 20:57:57 EST 2021: Starting Spark History Server
Running [/opt/cloudera/parcels/CDH-7.1.4-1.cdh7.1.4.p0.6300266/lib/spark/bin/spark-class org.apache.spark.deploy.history.HistoryServer --properties-file /var/run/cloudera-scm-agent/process/1546336662-spark_on_yarn-SPARK_YARN_HISTORY_SERVER/spark-conf/spark-history-server.conf]

[#5]

I modified the permission to 777 for /user

After which I am able to install Spark and able to install history server.

sudo -u hdfs hadoop fs -chmod 777 /user