Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

spark history server - Permission denied

avatar

we start the spark history as the following /usr/hdp/2.6.0.3-8/spark2/sbin/start-history-server.sh

from the log spark-root-org.apache.spark.deploy.history.HistoryServer-1-master01

we get many lines like that Caused by:

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=root, access=READ, inode="/spark2-history/application_1497173286109_0003_2.inprogress":hdfs:hadoop:-rwxrwx---

we try

hdfs dfs -chown spark /spark2-history

but not help not sure if we need to see the folder ?

# ls /spark2-history/ ls: cannot access /spark2-history/:

No such file or directory

please advice what is solution in order to strat the spark history

Michael-Bronson
1 ACCEPTED SOLUTION

avatar
Master Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
4 REPLIES 4

avatar
Super Collaborator

What user are you starting spark-history-server.sh as?

Do a su spark, before launching shell script. I think you're starting as root user, so it's saying root user doesn't have access to that folder.

Since you've given spark ownership, it should be able to access.

If you must start as root, then give root access to that directory.

avatar

ok I run it , and now we get - WARN AbstractLifeCycle: FAILED ServerConnector@14a54ef6{HTTP/1.1}{0.0.0.0:18081}: java.net.BindException: Address already in use java.net.BindException: Address already in use

Michael-Bronson

avatar
Cloudera Employee

Hi,

 

Did you checked what process is running on this port and tried killing the process if required?  if the issue got solved?

 

Thanks

AKR

avatar
Master Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login