Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

how to check the dfsadmin report in wasb space

avatar

I've a hortonworks cluster @azure cloud and I need to check the dfsadmin report in blob space.

when I'm trying with hdfs dfsadmin -report this command is giving the message "blob.core.windows.net is not an HDFS file system"

How do i get the dfsadmin report in wasb

1 ACCEPTED SOLUTION

avatar
Master Mentor

@Praveen Atmakuri

The dfsadmin is run like below

Get a report of status from each name node to show that the name nodes are in Safe Mode or not.

$ hdfs dfsadmin -D 'fs.default.name=hdfs://mycluster/' -safemode get 

A report that shows the details of HDFS state:

$ hdfs dfsadmin -D 'fs.default.name=hdfs://mycluster/' -report 

Get HDFS out of safe mode

$ hdfs dfsadmin -D 'fs.default.name=hdfs://mycluster/' -safemode leave

What's is your HDInsight cluster name

View solution in original post

8 REPLIES 8

avatar
Master Mentor

@Praveen Atmakuri

This is the format of the hdfsadmin command in Azure, where mycluster is your cluster name

hdfs dfsadmin -D "fs.default.name=hdfs://mycluster/" -report

Please revert

avatar

@Geoffrey Shelton Okot....Getting the same error.

"blob.core.windows.net is not an HDFS file system"

avatar
Master Mentor

@Praveen Atmakuri

Can you give the output of this command

hadoop fs -ls wasb://yourcontainer@youraccount.blob.core.windows.net/

The following is a list of configurations that should be modified to configure WASB: I assume you have done the below

fs.defaultFS

wasb://<containername>@<accountname>.blob.core.windows.net 

fs.AbstractFileSystem.wasb.impl

org.apache.hadoop.fs.azure.Wasb 

fs.azure.account.key. blob.core.windows.net

storage_access_key 

Please revert

avatar

fidn the output of the command: hadoop fs -ls wasb://mycontainer@youraccount.blob.core.windows.net/
18/04/02 08:31:26 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
18/04/02 08:31:26 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
18/04/02 08:31:26 INFO impl.MetricsSystemImpl: azure-file-system metrics system started
Found 3 items
drwxr-xr-x - paplcloudadmin supergroup 0 2017-12-29 09:53 wasb://containername@account.blob.core.windows.net/user/paplcloudadmin/.hiveJars
drwx------ - paplcloudadmin supergroup 0 2018-01-11 13:18 wasb://containername@account.blob.core.windows.net/user/paplcloudadmin/.staging
drwxr-xr-x - paplcloudadmin supergroup 0 2018-01-11 13:28 wasb://containername@account.blob.core.windows.net/user/paplcloudadmin/prod.db
18/04/02 08:31:26 INFO impl.MetricsSystemImpl: Stopping azure-file-system metrics system...
18/04/02 08:31:26 INFO impl.MetricsSystemImpl: azure-file-system metrics system stopped.
18/04/02 08:31:26 INFO impl.MetricsSystemImpl: azure-file-system metrics system shutdown complete.
The following is a list of configurations that should be modified to configure WASB: I assume you have done the below Correct, All configurations has been done properly and verified again...Result all are same.

avatar
Master Mentor

@Praveen Atmakuri

So I can see you have only the user paplcloudamin home directory set

Can you run the following

$ hdfs dfs -D "fs.default.name=hdfs://mycluster/" -mkdir /test

Check if the directory was created, you should be able to see test directory

$ hdfs dfs -D "fs.default.name=hdfs://mycluster/" -ls /

This command should run successfully,can you just copy and paste this command replacing my cluster with your clustername

$ hdfs dfsadmin -D "fs.default.name=hdfs://mycluster/" -report

Please revert

avatar

$ hdfs dfs -D "fs.default.name=wasb://containername@storage_name.blob.core.windows.net//"-mkdir /test

Done

$ hdfs dfs -D "fs.default.name=wasb://containername@storage_name.blob.core.windows.net/"-ls /

Result:
drwxr-xr-x - paplcloudadmin supergroup 0 2018-04-02 09:33 /test

hdfs dfsadmin -D "fs.default.name=wasb://containername@storage_name.blob.core.windows.net/" -report

18/04/02 09:40:56 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
18/04/02 09:40:56 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
18/04/02 09:40:56 INFO impl.MetricsSystemImpl: azure-file-system metrics system started
report: FileSystem wasb://containername@storage_name.blob.core.windows.net

is not an HDFS file system
Usage: hdfs dfsadmin [-report] [-live] [-dead] [-decommissioning]
18/04/02 09:40:56 INFO impl.MetricsSystemImpl: Stopping azure-file-system metrics system...
18/04/02 09:40:56 INFO impl.MetricsSystemImpl: azure-file-system metrics system stopped.
18/04/02 09:40:56 INFO impl.MetricsSystemImpl: azure-file-system metrics system shutdown complete.

avatar
Master Mentor

@Praveen Atmakuri

The dfsadmin is run like below

Get a report of status from each name node to show that the name nodes are in Safe Mode or not.

$ hdfs dfsadmin -D 'fs.default.name=hdfs://mycluster/' -safemode get 

A report that shows the details of HDFS state:

$ hdfs dfsadmin -D 'fs.default.name=hdfs://mycluster/' -report 

Get HDFS out of safe mode

$ hdfs dfsadmin -D 'fs.default.name=hdfs://mycluster/' -safemode leave

What's is your HDInsight cluster name

avatar
Expert Contributor

I'm getting the same error. This is the response I received form Cloudera support "Only the dfs commands such as ls/put/mv etc works on wasb using the wasb connector. Admin commands such as dfsadmin as well fsck works only with native hadoop/hdfs implementation"