Support Questions

Find answers, ask questions, and share your expertise

DEVPH Folder in Self paced learning VM

Hi there:

I am an existing student of self paced learning. In one of the modules titled 'HDP Developer: Apache Pig and Hive Rev 6', I need to use a VM from the following link: https://s3.amazonaws.com/vm-us-east/data_analyst/HDP2.3-Pig-Hive-Rev6.zip./.

There is a folder titled DEVPH under /root. I can easily navigate to the folder using the command prompt, as shown below:

9403-p1.png

However I am struggling to navigate to DEVPH using the UI. I am trying to edit a file under DEVPH using GEDIT. However I can't see the folder under root, through the UI, as shown below:

9404-p1.png

Any help will be much appreciated.

Thanks and regards,

Anindya

1 ACCEPTED SOLUTION

@Anindya Chattopadhyay

You are right. Let me explain other way -

1. You have your local system which has ubuntu installed.

2. On Ubuntu you are running Sandbox [for which you ssh to Login]

3. Sandbox has also operating system [generally centos/ubuntu - depends upon which sandbox you downloaded].

4. And you have HDFS filesystem which is in Sandbox which is Hadoop Filesystem.

5. All hadoop config files are stored on Sandbox Local filesystem [ie. core-site.xml,hdfs-site.xml.etc..]

You can do ssh to sandbox and then access the hadoop config files using [vi or cat command].

But you cannot browse it from the UI as you shown in screenshot above, since the UI/browser which you are loading is of your Local Ubuntu machine and not of Sandbox.

if you need to do the same for Sandbox then you need to install GUI for Sandbox [As sandbox does not comes default with GUI]

Hope that makes clear now.

View solution in original post

13 REPLIES 13

Hi :

Any help with this one?

I just realised we can't really access the HDFS files the way we acces File System files. Normally we need to copy the file to the local File System, edit it and put it back to HDFS.

So I am not sure what is meant by the following steps in the Self paced learning lab:

9438-ss.png

Thanks and regards,

Anindya

@Anindya Chattopadhyay

In the above image just put the value infront of "Location" as /root/devph/labs/Lab3.3

and enter. This will take you to the path.

So it will look like -

Location: /root/devph/labs/Lab3.3

@Sagar Shimpi, don't think so. That's how you access the local file system, not HDFS !! Look at the following screenshot.

9548-capture.png

@Anindya Chattopadhyay

If this is HDFS path then you need to have NFS gateway in place. Check https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_hdfs_nfs_gateway/bk_hdfs_nfs_gateway-201...

From above logs the path seems to be on local path and not HDFS.

@Sagar Shimpi, thanks for that PDF link. However I am just wondering how will I access files like core_site.xml, hdfs_site.xml etc.? Any help will be much appreciated.

Regards,

Anindya

@Anindya Chattopadhyay core_site.xml, hdfs_site.xml etc are on local OS filesystem and you should be able to browse as per above screenshot you displayed.

Else go to terminal and do

$cat /etc/hadoop/conf/core-site.xml

You should have basic linux command knowledge.

Please accept the answer to close this thread if you feel initial question was answered. Will be happy to help you if any new question raised.

@Sagar Shimpi, I am sort of across the linux commands (not an expert TBH). But I think I am understanding something really wrong here. My bad as I am very new to this HDP platform. My realization is core_site.xml, hdfs_site.xml etc. are not part of local file system. I can only see them after I run the SSH SANDBOX command through the terminal! See the following 2 screenshots: Screenshot 1 (without running the SSH SANDBOX command and I can't see any hadoop folder under etc)

9641-c1.png

Screenshot 2 (Running the SSH SANDBOX command and I can see hadoop folder under etc)

9642-c2.png

This makes me think those files are still part of HDFS and not local file system. So I still face the same situation?

Regards,

Anindya

@Anindya Chattopadhyay

You are right. Let me explain other way -

1. You have your local system which has ubuntu installed.

2. On Ubuntu you are running Sandbox [for which you ssh to Login]

3. Sandbox has also operating system [generally centos/ubuntu - depends upon which sandbox you downloaded].

4. And you have HDFS filesystem which is in Sandbox which is Hadoop Filesystem.

5. All hadoop config files are stored on Sandbox Local filesystem [ie. core-site.xml,hdfs-site.xml.etc..]

You can do ssh to sandbox and then access the hadoop config files using [vi or cat command].

But you cannot browse it from the UI as you shown in screenshot above, since the UI/browser which you are loading is of your Local Ubuntu machine and not of Sandbox.

if you need to do the same for Sandbox then you need to install GUI for Sandbox [As sandbox does not comes default with GUI]

Hope that makes clear now.

Hi @Sagar Shimpi

Good to know I have got the same understanding.

Anyways I will give it a go with the Vi /CAT editor and let you know.

Cheers!

See my answer as well as the reply I gave you from our Training group's service ticket system.

@Sagar Shimpi explains this well, but there is another option. You basically can just "map" the inner VM filesystem so it can be "seen" by the outer machine. Instructions for this are in the course lab guide (step 7 of lab 1). For benefit of all, I stripped out just those pages and am attaching the steps as jpegs.

12139-part-a.png

12140-part-b.png

12141-part-c.png

12142-part-d.png

12143-part-e.png

Thank you @Lester Martin

That helped!

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.