question Spark History Server failed to start in Archives of Support Questions (Read Only)

Spark History Server failed to start

vikash145 — Fri, 16 Sep 2022 11:56:07 GMT

Hi,

I am using CDH 5.12 . I have installed using path A. Spark History server is not getting started. All, I am getting below logs continiously and history server is not starting.

2017-07-15 13:52:47,000 INFO org.apache.spark.SecurityManager: Changing view acls to: cloudera-scm
2017-07-15 13:52:47,001 INFO org.apache.spark.SecurityManager: Changing modify acls to: cloudera-scm
2017-07-15 13:52:47,001 INFO org.apache.spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(cloudera-scm); users with modify permissions: Set(cloudera-scm)
2017-07-15 13:52:47,054 INFO org.apache.spark.deploy.history.FsHistoryProvider: History server ui acls disabled; users with admin permissions:
2017-07-15 13:52:47,060 WARN org.apache.spark.SparkConf: The configuration key 'spark.history.fs.update.interval.seconds' has been deprecated as of Spark 1.4 and and may be removed in the future. Please use the new key 'spark.history.fs.update.interval' instead.
2017-07-15 13:52:58,260 INFO org.apache.spark.deploy.history.HistoryServer: Started daemon with process name: 2561@compute-15
2017-07-15 13:52:58,273 INFO org.apache.spark.deploy.history.HistoryServer: Registered signal handlers for [TERM, HUP, INT]
2017-07-15 13:52:58,312 WARN org.apache.spark.SparkConf: The configuration key 'spark.history.fs.update.interval.seconds' has been deprecated as of Spark 1.4 and and may be removed in the future. Please use the new key 'spark.history.fs.update.interval' instead.

Is there something wrong in configuration ? How can this be solved ?

Thanks.

Re: Spark History Server failed to start

saranvisa — Sun, 16 Jul 2017 16:35:52 GMT

@vikash145

According to this log, your user id doesn't have permission to start the server, login as root and try again

additionally you can check the 'service <service name> status>' and try to start it

also check the config by running the below command and on it

chkconfig --list

Re: Spark History Server failed to start

vikash145 — Sun, 16 Jul 2017 17:09:34 GMT

@saranvisa

The user is cloudera-scm. All the other services are running fine. Are u suggesting apart from sudoers passwordless permission, I should set any other permission ? Please let me know , what and where this permission need to be set.

Thanks

Re: Spark History Server failed to start

saranvisa — Sun, 16 Jul 2017 18:08:27 GMT

@vikash145

when do you get this error? when you try to launch spark shell (or) when you start the history servicer (or) in a different situation?

how did you start all other services? using the same user

also i don't see any keyword called error in your log, it says only warning

Re: Spark History Server failed to start

vikash145 — Mon, 17 Jul 2017 17:52:30 GMT

@mbigelow

Any pointer on this ?

Re: Spark History Server failed to start

mbigelow — Mon, 17 Jul 2017 18:02:30 GMT

There isn't anything of use in the logs you posted. Are you able to post the full log?

Also check the stderr log, either in CM or in the process running directory, as I think this is where it posted launch errors.

Re: Spark History Server failed to start

vikash145 — Mon, 17 Jul 2017 18:06:18 GMT

Once this HIVE service is launched. I will post the log.

Re: Spark History Server failed to start

vikash145 — Tue, 18 Jul 2017 04:50:35 GMT

/user/spark directory was missing. I created it manually and it worked fine.