Created 11-01-2017 04:18 AM
we start the spark history as the following /usr/hdp/2.6.0.3-8/spark2/sbin/start-history-server.sh
from the log spark-root-org.apache.spark.deploy.history.HistoryServer-1-master01
we get many lines like that Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=root, access=READ, inode="/spark2-history/application_1497173286109_0003_2.inprogress":hdfs:hadoop:-rwxrwx---
we try
hdfs dfs -chown spark /spark2-history
but not help not sure if we need to see the folder ?
# ls /spark2-history/ ls: cannot access /spark2-history/:
No such file or directory
please advice what is solution in order to strat the spark history
Created 11-01-2017 05:07 AM
Please check which process is using that port which is causing the conflict (If needed the kill that process and then try again)
# netstat -tnlpa | grep 18081 # kill -9 $PID_OF_ABOVE_PROCESS
.
Created 11-01-2017 04:29 AM
What user are you starting spark-history-server.sh as?
Do a su spark, before launching shell script. I think you're starting as root user, so it's saying root user doesn't have access to that folder.
Since you've given spark ownership, it should be able to access.
If you must start as root, then give root access to that directory.
Created 11-01-2017 04:39 AM
ok I run it , and now we get - WARN AbstractLifeCycle: FAILED ServerConnector@14a54ef6{HTTP/1.1}{0.0.0.0:18081}: java.net.BindException: Address already in use java.net.BindException: Address already in use
Created 10-11-2019 04:45 AM
Hi,
Did you checked what process is running on this port and tried killing the process if required? if the issue got solved?
Thanks
AKR
Created 11-01-2017 05:07 AM
Please check which process is using that port which is causing the conflict (If needed the kill that process and then try again)
# netstat -tnlpa | grep 18081 # kill -9 $PID_OF_ABOVE_PROCESS
.