question Re: Hands-on Spark Tutorial: Permission Denied and no such file or directory in Archives of Support Questions (Read Only)

Hands-on Spark Tutorial: Permission Denied and no such file or directory

snasir — Mon, 19 Aug 2019 12:27:32 GMT

I was able to download and bring up the sandbox. I also changed the password root user. I am also able navigate to http://127.0.0.1:8088/ and see the hadoop cluster. Now I am trying to go through following tutorial:

http://hortonworks.com/blog/hands-on-tour-of-apach...

but running in to permissions issues or message like no such file or directory.

I was able to run this command and download data:

wget <a href="http://en.wikipedia.org/wiki/Hortonworks">http://en.wikipedia.org/wiki/Hortonworks</a>

but when I run the next step:

hadoop fs -put ~/Hortonworks /user/guest/Hortonworks

I get following message:

When I try to run hadoop fs -ls , I get the message the no such file or directory exists:

I then tried to create the /home/root directory but that failed too:

Any thoughts on what may be happening here?

Thanks

Re: Hands-on Spark Tutorial: Permission Denied and no such file or directory

nsabharwal — Wed, 23 Dec 2015 10:04:53 GMT

@shahab nasir

Do this

sudo su - hdfs

hdfs dfs -mkdir /user/root

hdfs dfs -chown root:hdfs /user/root

exit

and run hdfs commands

Re: Hands-on Spark Tutorial: Permission Denied and no such file or directory

snasir — Wed, 23 Dec 2015 10:33:52 GMT

thanks Neeraj. That worked. Appreciate it.

Re: Hands-on Spark Tutorial: Permission Denied and no such file or directory

amcbarnett — Fri, 01 Jan 2016 05:02:39 GMT

Specifically for /user/guest/Hortonworks do this:

sudo su -hdfs
hadoop fs -chmod -R 777 /user/guest
exit

Re: Hands-on Spark Tutorial: Permission Denied and no such file or directory

rhryniewicz — Thu, 03 Mar 2016 10:35:21 GMT

In the Sandbox, the /user/guest does not actually exist. You can create /user/guest/ using the following command:

hadoop fs -mkdir /user/guest/

Alternatively, you can use /tmp directory:

hadoop fs -put ~/Hortonworks /tmp

The tutorial has been updated to use /tmp directory to simplify the flow.

Re: Hands-on Spark Tutorial: Permission Denied and no such file or directory

sdutta — Thu, 03 Mar 2016 12:59:19 GMT

The tutorial needs to be fixed.

Re: Hands-on Spark Tutorial: Permission Denied and no such file or directory

sdutta — Thu, 03 Mar 2016 13:02:43 GMT

@shahab nasir

Best is to use the ambari-qa user. This is special user with super powers 🙂

su ambari-qa

Overall, please understand the Hadoop security model. Spend some time to understand the user permissions. This is mostly like unix. The service accounts hdfs, yarn etc are service accounts that are part of hadoop group.

Spend some time on the Hadoop HDFS section. This will help your understanding better.