question Re: how to check the dfsadmin report in wasb space in Support Questions

how to check the dfsadmin report in wasb space

praveen_atmakur — Fri, 01 Nov 2019 12:54:54 GMT

I've a hortonworks cluster @azure cloud and I need to check the dfsadmin report in blob space.

when I'm trying with hdfs dfsadmin -report this command is giving the message "blob.core.windows.net is not an HDFS file system"

How do i get the dfsadmin report in wasb

Re: how to check the dfsadmin report in wasb space

Shelton — Wed, 28 Mar 2018 17:43:40 GMT

@Praveen Atmakuri

This is the format of the hdfsadmin command in Azure, where mycluster is your cluster name

hdfs dfsadmin -D "fs.default.name=hdfs://mycluster/" -report

Please revert

Re: how to check the dfsadmin report in wasb space

praveen_atmakur — Mon, 02 Apr 2018 14:12:00 GMT

@Geoffrey Shelton Okot....Getting the same error.

"blob.core.windows.net is not an HDFS file system"

Re: how to check the dfsadmin report in wasb space

Shelton — Mon, 02 Apr 2018 15:21:25 GMT

@Praveen Atmakuri

Can you give the output of this command

hadoop fs -ls wasb://yourcontainer@youraccount.blob.core.windows.net/

The following is a list of configurations that should be modified to configure WASB: I assume you have done the below

fs.defaultFS

wasb://<containername>@<accountname>.blob.core.windows.net

fs.AbstractFileSystem.wasb.impl

org.apache.hadoop.fs.azure.Wasb

fs.azure.account.key. blob.core.windows.net

storage_access_key

Please revert

Re: how to check the dfsadmin report in wasb space

praveen_atmakur — Mon, 02 Apr 2018 15:40:45 GMT

fidn the output of the command: hadoop fs -ls wasb://mycontainer@youraccount.blob.core.windows.net/
18/04/02 08:31:26 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
18/04/02 08:31:26 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
18/04/02 08:31:26 INFO impl.MetricsSystemImpl: azure-file-system metrics system started
Found 3 items
drwxr-xr-x - paplcloudadmin supergroup 0 2017-12-29 09:53 wasb://containername@account.blob.core.windows.net/user/paplcloudadmin/.hiveJars
drwx------ - paplcloudadmin supergroup 0 2018-01-11 13:18 wasb://containername@account.blob.core.windows.net/user/paplcloudadmin/.staging
drwxr-xr-x - paplcloudadmin supergroup 0 2018-01-11 13:28 wasb://containername@account.blob.core.windows.net/user/paplcloudadmin/prod.db
18/04/02 08:31:26 INFO impl.MetricsSystemImpl: Stopping azure-file-system metrics system...
18/04/02 08:31:26 INFO impl.MetricsSystemImpl: azure-file-system metrics system stopped.
18/04/02 08:31:26 INFO impl.MetricsSystemImpl: azure-file-system metrics system shutdown complete.
The following is a list of configurations that should be modified to configure WASB: I assume you have done the below Correct, All configurations has been done properly and verified again...Result all are same.

Re: how to check the dfsadmin report in wasb space

Shelton — Mon, 02 Apr 2018 16:30:24 GMT

@Praveen Atmakuri

So I can see you have only the user paplcloudamin home directory set

Can you run the following

$ hdfs dfs -D "fs.default.name=hdfs://mycluster/" -mkdir /test

Check if the directory was created, you should be able to see test directory

$ hdfs dfs -D "fs.default.name=hdfs://mycluster/" -ls /

This command should run successfully,can you just copy and paste this command replacing my cluster with your clustername

$ hdfs dfsadmin -D "fs.default.name=hdfs://mycluster/" -report

Please revert

Re: how to check the dfsadmin report in wasb space

praveen_atmakur — Mon, 02 Apr 2018 16:50:50 GMT

$ hdfs dfs -D "fs.default.name=wasb://containername@storage_name.blob.core.windows.net//"-mkdir /test

Done

$ hdfs dfs -D "fs.default.name=wasb://containername@storage_name.blob.core.windows.net/"-ls /

Result:
drwxr-xr-x - paplcloudadmin supergroup 0 2018-04-02 09:33 /test

hdfs dfsadmin -D "fs.default.name=wasb://containername@storage_name.blob.core.windows.net/" -report

18/04/02 09:40:56 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
18/04/02 09:40:56 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
18/04/02 09:40:56 INFO impl.MetricsSystemImpl: azure-file-system metrics system started
report: FileSystem wasb://containername@storage_name.blob.core.windows.net

is not an HDFS file system
Usage: hdfs dfsadmin [-report] [-live] [-dead] [-decommissioning]
18/04/02 09:40:56 INFO impl.MetricsSystemImpl: Stopping azure-file-system metrics system...
18/04/02 09:40:56 INFO impl.MetricsSystemImpl: azure-file-system metrics system stopped.
18/04/02 09:40:56 INFO impl.MetricsSystemImpl: azure-file-system metrics system shutdown complete.

Re: how to check the dfsadmin report in wasb space

Shelton — Mon, 02 Apr 2018 17:53:03 GMT

@Praveen Atmakuri

The dfsadmin is run like below

Get a report of status from each name node to show that the name nodes are in Safe Mode or not.

$ hdfs dfsadmin -D 'fs.default.name=hdfs://mycluster/' -safemode get

A report that shows the details of HDFS state:

$ hdfs dfsadmin -D 'fs.default.name=hdfs://mycluster/' -report

Get HDFS out of safe mode

$ hdfs dfsadmin -D 'fs.default.name=hdfs://mycluster/' -safemode leave

What's is your HDInsight cluster name

Re: how to check the dfsadmin report in wasb space

pd47 — Mon, 28 Oct 2019 19:19:52 GMT

I'm getting the same error. This is the response I received form Cloudera support "Only the dfs commands such as ls/put/mv etc works on wasb using the wasb connector. Admin commands such as dfsadmin as well fsck works only with native hadoop/hdfs implementation"