Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Why hdfs://mycluster/ different from /

avatar
Expert Contributor

We have HDInsight cluster setup in Azure.

When I do hadoop fs -ls / it shows me

drwxr-xr-x   - root    supergroup          0 2016-06-17 20:56 /HdiNotebooks
drwxr-xr-x   - root    supergroup          0 2016-06-17 21:00 /HdiSamples
drwxr-xr-x   - hdfs    supergroup          0 2016-06-17 20:48 /ams
drwxr-xr-x   - hdfs    supergroup          0 2016-06-17 20:48 /amshbase
drwxrwxrwx   - yarn    hadoop              0 2016-06-17 20:48 /app-logs
drwxr-xr-x   - yarn    hadoop              0 2016-06-17 20:48 /atshistory
drwxr-xr-x   - sshuser supergroup          0 2016-06-21 18:38 /data
drwxr-xr-x   - root    supergroup          0 2016-06-17 20:59 /example
drwxr-xr-x   - hdfs    supergroup          0 2016-06-17 20:48 /hdp
drwxr-xr-x   - hdfs    supergroup          0 2016-06-17 20:48 /hive
drwxr-xr-x   - mapred  supergroup          0 2016-06-17 20:48 /mapred
drwx------   - sshuser supergroup          0 2016-06-20 14:22 /mapreducestaging
drwxrwxrwx   - mapred  hadoop              0 2016-06-17 20:48 /mr-history
drwxr-xr-x   - sshuser supergroup          0 2016-06-20 19:20 /sqoop
drwxrwxrwx   - hdfs    supergroup          0 2016-06-17 20:48 /tmp
drwxr-xr-x   - hdfs    supergroup          0 2016-06-17 20:48 /user


But hadoop fs -ls hdfs://mycluster/

shows following result.

root@hn0-haspar:~# hadoop fs -ls hdfs://mycluster/
Found 3 items
drwxr-xr-x   - root hdfs          0 2016-06-21 18:48 hdfs://mycluster/data
drwx-wx-wx   - root hdfs          0 2016-06-17 20:57 hdfs://mycluster/tmp
drwx------   - root hdfs          0 2016-06-22 17:24 hdfs://mycluster/user


Dont know where this different dir coming.

Cluster has HA configuration.

1 ACCEPTED SOLUTION

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
4 REPLIES 4

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Expert Contributor

Thanks @Chris Nauroth for explanation. At present We have Namenode HA and we are putting data from Flume into this cluster. We are configuring hdfs://mycluster/flume as destination in Flume sink.

Whats is the correct way to put data into default HDFS storage (WASB) from Flume and make it accessible from hadoop fs -ls / ?

Appreciate help in this.

avatar
@roy p, if you want to route the data from Flume to WASB instead of HDFS, then I expect you can achieve that by changing the "hdfs:" URI to a "wasb:" URI. The full WASB URI will have an authority component that references an Azure Storage account and a container within that account. You can get the WASB URI by looking at configuration property fs.defaultFS in core-site.xml. If that doesn't work, then I recommend creating a new question specifically asking how to configure Flume to write to a file system different from HDFS. Please also apply the "flume" tag to the question. That will help get attention from Flume experts.

avatar
Expert Contributor

Thanks, it worked.