DEVPH Folder in Self paced learning VM

[#1]

Hi there:

I am an existing student of self paced learning. In one of the modules titled 'HDP Developer: Apache Pig and Hive Rev 6', I need to use a VM from the following link: https://s3.amazonaws.com/vm-us-east/data_analyst/HDP2.3-Pig-Hive-Rev6.zip./.

There is a folder titled DEVPH under /root. I can easily navigate to the folder using the command prompt, as shown below:

However I am struggling to navigate to DEVPH using the UI. I am trying to edit a file under DEVPH using GEDIT. However I can't see the folder under root, through the UI, as shown below:

Any help will be much appreciated.

Thanks and regards,

Anindya

[#2]

Hi :

Any help with this one?

I just realised we can't really access the HDFS files the way we acces File System files. Normally we need to copy the file to the local File System, edit it and put it back to HDFS.

So I am not sure what is meant by the following steps in the Self paced learning lab:

Thanks and regards,

Anindya

[#3]

@Anindya Chattopadhyay

In the above image just put the value infront of "Location" as /root/devph/labs/Lab3.3

and enter. This will take you to the path.

[#4]

So it will look like -

Location: /root/devph/labs/Lab3.3

[#5]

@Sagar Shimpi, don't think so. That's how you access the local file system, not HDFS !! Look at the following screenshot.

[#6]

@Anindya Chattopadhyay

If this is HDFS path then you need to have NFS gateway in place. Check https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_hdfs_nfs_gateway/bk_hdfs_nfs_gateway-201...

From above logs the path seems to be on local path and not HDFS.

[#7]

@Sagar Shimpi, thanks for that PDF link. However I am just wondering how will I access files like core_site.xml, hdfs_site.xml etc.? Any help will be much appreciated.

Regards,

Anindya

[#8]

@Anindya Chattopadhyay core_site.xml, hdfs_site.xml etc are on local OS filesystem and you should be able to browse as per above screenshot you displayed.

Else go to terminal and do

$cat /etc/hadoop/conf/core-site.xml

You should have basic linux command knowledge.

Please accept the answer to close this thread if you feel initial question was answered. Will be happy to help you if any new question raised.

[#9]

@Sagar Shimpi, I am sort of across the linux commands (not an expert TBH). But I think I am understanding something really wrong here. My bad as I am very new to this HDP platform. My realization is core_site.xml, hdfs_site.xml etc. are not part of local file system. I can only see them after I run the SSH SANDBOX command through the terminal! See the following 2 screenshots: Screenshot 1 (without running the SSH SANDBOX command and I can't see any hadoop folder under etc)

Screenshot 2 (Running the SSH SANDBOX command and I can see hadoop folder under etc)

This makes me think those files are still part of HDFS and not local file system. So I still face the same situation?

Regards,

Anindya

[#10 · Accepted solution]

@Anindya Chattopadhyay

You are right. Let me explain other way -

1. You have your local system which has ubuntu installed.

2. On Ubuntu you are running Sandbox [for which you ssh to Login]

3. Sandbox has also operating system [generally centos/ubuntu - depends upon which sandbox you downloaded].

4. And you have HDFS filesystem which is in Sandbox which is Hadoop Filesystem.

5. All hadoop config files are stored on Sandbox Local filesystem [ie. core-site.xml,hdfs-site.xml.etc..]

You can do ssh to sandbox and then access the hadoop config files using [vi or cat command].

But you cannot browse it from the UI as you shown in screenshot above, since the UI/browser which you are loading is of your Local Ubuntu machine and not of Sandbox.

if you need to do the same for Sandbox then you need to install GUI for Sandbox [As sandbox does not comes default with GUI]

Hope that makes clear now.