Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Not able find directory created under HDFS

avatar
Explorer

Hi Friends, I am trying to understand the PIG scripting.

I have created directory called pig_demo and am not able to find when i do hadoop fs -ls /

Appericiate your timely response on this.

 

details are given bleow

1) [cloudera@quickstart ~]$ pwd
/home/cloudera
[cloudera@quickstart ~]$ hadoop fs -ls /
Found 5 items
drwxr-xr-x   - hbase supergroup          0 2015-10-29 13:01 /hbase
drwxr-xr-x   - solr  solr                0 2015-06-09 08:55 /solr
drwxrwxrwx   - hdfs  supergroup          0 2015-10-09 06:24 /tmp
drwxr-xr-x   - hdfs  supergroup          0 2015-06-09 08:56 /user
drwxr-xr-x   - hdfs  supergroup          0 2015-06-09 08:54 /var

 

2) [cloudera@quickstart ~]$ hadoop fs -mkdir pig_demo
mkdir: `pig_demo': File exists
[cloudera@quickstart ~]$ hadoop fs -ls /
Found 5 items
drwxr-xr-x   - hbase supergroup          0 2015-10-29 13:01 /hbase
drwxr-xr-x   - solr  solr                0 2015-06-09 08:55 /solr
drwxrwxrwx   - hdfs  supergroup          0 2015-10-09 06:24 /tmp
drwxr-xr-x   - hdfs  supergroup          0 2015-06-09 08:56 /user
drwxr-xr-x   - hdfs  supergroup          0 2015-06-09 08:54 /var

1 ACCEPTED SOLUTION

avatar
Guru

Also note that when you just run 'ls' or 'pwd', that's referring to your local / native filesystem. What directory you are currently in in your shell has no effect on what directory it shows you when you say 'hdfs dfs -ls'. Now, there's a /usr directory on the local / native filesystem, but that's a complete separate idea from the /user directory in HDFS.

View solution in original post

6 REPLIES 6

avatar
Rising Star

Have you cheked in /user directory?

avatar
Guru

Check in /user/cloudera. Unless you're the 'hdfs' user, HDFS treats
/user/[username] as your home directory. When your paths don't start with
/, they go in there. If they DO start with /, it doesn't resolve it
relative to your home directory. If you just type 'hadoop fs -ls', it'll
look in your home directory and you should see it.

avatar
Explorer

I am not able to find under usr

[cloudera@quickstart usr]$ ls
bin  games    jars  lib    libexec  sbin   src
etc  include  java  lib64  local    share  tmp

[cloudera@quickstart usr]$

 

[cloudera@quickstart /]$ hadoop fs -ls /
Found 5 items
drwxr-xr-x   - hbase supergroup          0 2015-10-29 13:01 /hbase
drwxr-xr-x   - solr  solr                0 2015-06-09 08:55 /solr
drwxrwxrwx   - hdfs  supergroup          0 2015-10-09 06:24 /tmp
drwxr-xr-x   - hdfs  supergroup          0 2015-06-09 08:56 /user
drwxr-xr-x   - hdfs  supergroup          0 2015-06-09 08:54 /var

avatar
Expert Contributor

I think what the earlier responders were suggesting is that you type this at the Linux command prompt:

 

hdfs dfs -ls /user/cloudera

 

You should see your new directory, pig-demo, listed as a subdirectory of /user/cloudera.

Can you try this?

 

(Note: you can use 'hadoop fs -ls /user/cloudera' too, but 'hadoop fs' is being deprecated so best to switch to using 'hdfs dfs ...')

 

avatar
Guru

Also note that when you just run 'ls' or 'pwd', that's referring to your local / native filesystem. What directory you are currently in in your shell has no effect on what directory it shows you when you say 'hdfs dfs -ls'. Now, there's a /usr directory on the local / native filesystem, but that's a complete separate idea from the /user directory in HDFS.

avatar
Explorer

Sean , Thank you very much for providing quick resolution