HDP file location for Nifi

[#1]

Where can i see the files on the sandbox which i put using the PutHDFS processor. I do understand they may not be directly visible in the file system (by directly logging into the terminal on the sandbox)as they are on the DataNode but how can i see the files that are present on the datanode. The ambari file view displays the files on the DataNode i guess.I can also navigate to view the files on the data node on sandbox.hortonworks.com:50070/explorer.html#/

I have also added GNOME GUI to the sandbox. The files that I see in "Computer" are different from the ones i see when i do a "dir" in the terminal. Which is a bit confusing.

Lastly i put a file using PutHDFS in a folder a directory called NiFi. where can i see this? i dont see it on the data node either.

01 image: shows files when connected to the sandbox via Putty

02 image: Shows the Computer files in the Sandbox

03 image: Shows the files by navigating to HDFS.Quicklinks.namenodeui.utilities.browse file system (same as FileView in Ambari)

(i guess this shows the name node and not the data node)

04 image:PutHDFS configuration which puts the file to Nifi directory. A getHDFS also fetched the file.

Lastly how do i get a file from my local PC in Nifi now which is running inside the sandbox.

[#2 · Accepted solution]

Your first screen shot shows the local file system, not hdfs. To see the files in hdfs from the command line you have to run the command

hdfs dfs -ls path

If you do not start path with / the path will be relative to the user directory of the user you are logged in as. In your case it will be /user/root in hdfs.

Similarly from the NiFi side if you do not start the path in PutHDFS with / it will put to hdfs under the nifi user (I think it will be /user/nifi/Nifi).

It is a best practice to specify hdfs paths explicitly (ie starting with /)

You can use the explorer ui to navigate the hdfs file system and you can also use the Ambari files view. (Just log into Ambari and go to views in upper right, then Files View)

See the following links for more:

http://www.dummies.com/programming/big-data/hadoop/hadoop-distributed-file-system-shell-commands/

https://hadoop.apache.org/docs/r2.6.3/hadoop-project-dist/hadoop-common/FileSystemShell.html