unable to start spark2 history server in HDP 2.6

[#1]

Please see the error messsage below and help to start the Spark2 history server.

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/SPARK2/2.0.0/package/scripts/job_history_server.py", line 103, in <module>
    JobHistoryServer().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/SPARK2/2.0.0/package/scripts/job_history_server.py", line 56, in start
    spark_service('jobhistoryserver', upgrade_type=upgrade_type, action='start')
  File "/var/lib/ambari-agent/cache/common-services/SPARK2/2.0.0/package/scripts/spark_service.py", line 65, in spark_service
    make_tarfile(tmp_archive_file, source_dir)
  File "/var/lib/ambari-agent/cache/common-services/SPARK2/2.0.0/package/scripts/spark_service.py", line 44, in make_tarfile
    os.chmod(parent_dir, 0711) 

OSError: [Errno 1] Operation not permitted: '/tmp/spark2'

[#2]

Could you share more details? It looks like you are not running as a privileged user, if you're getting an 'operation not permitted' error, but it is hard to know without context.

• Are any of your other services starting?
• Have you ever been able to start the Spark2 History Server? Is this a recent issue?
• What does your cluster look like? # nodes, specs for each node?
• Any other issues you've noticed, other details you can provide?

[#3]

@Jay Kumar SenSharma Can I have your advise on my post please..

[#4 · Accepted solution]

@Saravana V

As we do not have the complete stacktrace of the error hence i am assuming that the last line of the error is :

  File "/var/lib/ambari-agent/cache/common-services/SPARK2/2.0.0/package/scripts/spark_service.py", line 44, in make_tarfile    
os.chmod(parent_dir, 0711) 

Which indicates that when ambari is trying to start the Spark2 history server then at the time when it is failing.. it is trying to change the directory "/tmp/spark2" permission to 0711 , So i will suggest you to first check if that directory exist on the Spark2 History server host? Please try the following two approaches to see if it works:

1. Try Creating it on your own like following:

# mkdir /tmp/spark2
# chmod -R 711 /tmp/spark2
# ls -ld /tmp/spark2
drwx--x--x. 2 root root 43 Mar 28 23:15 /tmp/spark2

1. If the directory already exist there then try to remove it and then start the component.

.

Please let us know if you are running the ambari agent on that host as "Non Root" user by any chance?

If yes then have you followed the steps mentioned in https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.1.5/bk_ambari-security/content/how_to_configure...

[#5]

@Jay Kumar SenSharma Thanks for heads up.

Ambari agents are running as Non root (hadoop) user

the directory was created when we tried to srart the spark2 service and the owner was assigned as spark user. Now I've changed the owner as hadoop user and the service came up.

thanks a lot 🙂