Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

DEVPH Folder in Self paced learning VM

avatar
Contributor

Hi there:

I am an existing student of self paced learning. In one of the modules titled 'HDP Developer: Apache Pig and Hive Rev 6', I need to use a VM from the following link: https://s3.amazonaws.com/vm-us-east/data_analyst/HDP2.3-Pig-Hive-Rev6.zip./.

There is a folder titled DEVPH under /root. I can easily navigate to the folder using the command prompt, as shown below:

9403-p1.png

However I am struggling to navigate to DEVPH using the UI. I am trying to edit a file under DEVPH using GEDIT. However I can't see the folder under root, through the UI, as shown below:

9404-p1.png

Any help will be much appreciated.

Thanks and regards,

Anindya

1 ACCEPTED SOLUTION

avatar
Super Guru

@Anindya Chattopadhyay

You are right. Let me explain other way -

1. You have your local system which has ubuntu installed.

2. On Ubuntu you are running Sandbox [for which you ssh to Login]

3. Sandbox has also operating system [generally centos/ubuntu - depends upon which sandbox you downloaded].

4. And you have HDFS filesystem which is in Sandbox which is Hadoop Filesystem.

5. All hadoop config files are stored on Sandbox Local filesystem [ie. core-site.xml,hdfs-site.xml.etc..]

You can do ssh to sandbox and then access the hadoop config files using [vi or cat command].

But you cannot browse it from the UI as you shown in screenshot above, since the UI/browser which you are loading is of your Local Ubuntu machine and not of Sandbox.

if you need to do the same for Sandbox then you need to install GUI for Sandbox [As sandbox does not comes default with GUI]

Hope that makes clear now.

View solution in original post

13 REPLIES 13

avatar
Contributor

Hi :

Any help with this one?

I just realised we can't really access the HDFS files the way we acces File System files. Normally we need to copy the file to the local File System, edit it and put it back to HDFS.

So I am not sure what is meant by the following steps in the Self paced learning lab:

9438-ss.png

Thanks and regards,

Anindya

avatar
Super Guru

@Anindya Chattopadhyay

In the above image just put the value infront of "Location" as /root/devph/labs/Lab3.3

and enter. This will take you to the path.

avatar
Super Guru

So it will look like -

Location: /root/devph/labs/Lab3.3

avatar
Contributor

@Sagar Shimpi, don't think so. That's how you access the local file system, not HDFS !! Look at the following screenshot.

9548-capture.png

avatar
Super Guru
@Anindya Chattopadhyay

If this is HDFS path then you need to have NFS gateway in place. Check https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_hdfs_nfs_gateway/bk_hdfs_nfs_gateway-201...

From above logs the path seems to be on local path and not HDFS.

avatar
Contributor

@Sagar Shimpi, thanks for that PDF link. However I am just wondering how will I access files like core_site.xml, hdfs_site.xml etc.? Any help will be much appreciated.

Regards,

Anindya

avatar
Super Guru

@Anindya Chattopadhyay core_site.xml, hdfs_site.xml etc are on local OS filesystem and you should be able to browse as per above screenshot you displayed.

Else go to terminal and do

$cat /etc/hadoop/conf/core-site.xml

You should have basic linux command knowledge.

Please accept the answer to close this thread if you feel initial question was answered. Will be happy to help you if any new question raised.

avatar
Contributor

@Sagar Shimpi, I am sort of across the linux commands (not an expert TBH). But I think I am understanding something really wrong here. My bad as I am very new to this HDP platform. My realization is core_site.xml, hdfs_site.xml etc. are not part of local file system. I can only see them after I run the SSH SANDBOX command through the terminal! See the following 2 screenshots: Screenshot 1 (without running the SSH SANDBOX command and I can't see any hadoop folder under etc)

9641-c1.png

Screenshot 2 (Running the SSH SANDBOX command and I can see hadoop folder under etc)

9642-c2.png

This makes me think those files are still part of HDFS and not local file system. So I still face the same situation?

Regards,

Anindya

avatar
Super Guru

@Anindya Chattopadhyay

You are right. Let me explain other way -

1. You have your local system which has ubuntu installed.

2. On Ubuntu you are running Sandbox [for which you ssh to Login]

3. Sandbox has also operating system [generally centos/ubuntu - depends upon which sandbox you downloaded].

4. And you have HDFS filesystem which is in Sandbox which is Hadoop Filesystem.

5. All hadoop config files are stored on Sandbox Local filesystem [ie. core-site.xml,hdfs-site.xml.etc..]

You can do ssh to sandbox and then access the hadoop config files using [vi or cat command].

But you cannot browse it from the UI as you shown in screenshot above, since the UI/browser which you are loading is of your Local Ubuntu machine and not of Sandbox.

if you need to do the same for Sandbox then you need to install GUI for Sandbox [As sandbox does not comes default with GUI]

Hope that makes clear now.