question Re: Not able find directory created under HDFS in Support Questions

Not able find directory created under HDFS

RaoCary — Fri, 16 Sep 2022 09:46:50 GMT

Hi Friends, I am trying to understand the PIG scripting.

I have created directory called pig_demo and am not able to find when i do hadoop fs -ls /

Appericiate your timely response on this.

details are given bleow

1) [cloudera@quickstart ~]$ pwd
/home/cloudera
[cloudera@quickstart ~]$ hadoop fs -ls /
Found 5 items
drwxr-xr-x   - hbase supergroup          0 2015-10-29 13:01 /hbase
drwxr-xr-x   - solr solr                0 2015-06-09 08:55 /solr
drwxrwxrwx   - hdfs supergroup          0 2015-10-09 06:24 /tmp
drwxr-xr-x   - hdfs supergroup          0 2015-06-09 08:56 /user
drwxr-xr-x   - hdfs supergroup          0 2015-06-09 08:54 /var

2) [cloudera@quickstart ~]$ hadoop fs -mkdir pig_demo
mkdir: `pig_demo': File exists
[cloudera@quickstart ~]$ hadoop fs -ls /
Found 5 items
drwxr-xr-x   - hbase supergroup          0 2015-10-29 13:01 /hbase
drwxr-xr-x   - solr solr                0 2015-06-09 08:55 /solr
drwxrwxrwx   - hdfs supergroup          0 2015-10-09 06:24 /tmp
drwxr-xr-x   - hdfs supergroup          0 2015-06-09 08:56 /user
drwxr-xr-x   - hdfs supergroup          0 2015-06-09 08:54 /var

Re: Not able find directory created under HDFS

Somu — Thu, 29 Oct 2015 21:02:03 GMT

Have you cheked in /user directory?

Re: Not able find directory created under HDFS

Sean — Thu, 29 Oct 2015 21:02:27 GMT

Check in /user/cloudera. Unless you're the 'hdfs' user, HDFS treats
/user/[username] as your home directory. When your paths don't start with
/, they go in there. If they DO start with /, it doesn't resolve it
relative to your home directory. If you just type 'hadoop fs -ls', it'll
look in your home directory and you should see it.

Re: Not able find directory created under HDFS

RaoCary — Thu, 29 Oct 2015 21:13:19 GMT

I am not able to find under usr

[cloudera@quickstart usr]$ ls
bin games jars lib libexec sbin src
etc include java lib64 local share tmp

[cloudera@quickstart usr]$

[cloudera@quickstart /]$ hadoop fs -ls /
Found 5 items
drwxr-xr-x   - hbase supergroup          0 2015-10-29 13:01 /hbase
drwxr-xr-x   - solr solr                0 2015-06-09 08:55 /solr
drwxrwxrwx   - hdfs supergroup          0 2015-10-09 06:24 /tmp
drwxr-xr-x   - hdfs supergroup          0 2015-06-09 08:56 /user
drwxr-xr-x   - hdfs supergroup          0 2015-06-09 08:54 /var

Re: Not able find directory created under HDFS

Sue — Thu, 29 Oct 2015 21:25:53 GMT

I think what the earlier responders were suggesting is that you type this at the Linux command prompt:

hdfs dfs -ls /user/cloudera

You should see your new directory, pig-demo, listed as a subdirectory of /user/cloudera.

Can you try this?

(Note: you can use 'hadoop fs -ls /user/cloudera' too, but 'hadoop fs' is being deprecated so best to switch to using 'hdfs dfs ...')

Re: Not able find directory created under HDFS

Sean — Thu, 29 Oct 2015 21:41:29 GMT

Also note that when you just run 'ls' or 'pwd', that's referring to your local / native filesystem. What directory you are currently in in your shell has no effect on what directory it shows you when you say 'hdfs dfs -ls'. Now, there's a /usr directory on the local / native filesystem, but that's a complete separate idea from the /user directory in HDFS.

Re: Not able find directory created under HDFS

RaoCary — Thu, 29 Oct 2015 22:50:07 GMT

Sean , Thank you very much for providing quick resolution