Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Value of dfs.blocksize on ambari and BlockCapacity from namenodes REST response is different

Highlighted

Value of dfs.blocksize on ambari and BlockCapacity from namenodes REST response is different

New Contributor

BlockCapacity given in the REST reponse of Namenode is different than what we have set in the Ambari under HDFS dfs.blocksize

On Rest I am getting the value for BlockCapacity is 2097152 which equivalent to 2mb although the value of blocksize in Ambari is 128mb by default Fetching the blockcapacity from from following REST url https://<namenode-ip>:50470/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem

Does BlockCapacity from namenodes rest api and dfs.blocksize from Ambari are the same or different?

 

img1.pngimg2.png

3 REPLIES 3
Highlighted

Re: Value of dfs.blocksize on ambari and BlockCapacity from namenodes REST response is different

Expert Contributor

@MAS The BlockCapacity metric is different from Blocksize. Per the metrics page [https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Metrics.html], block capacity denotes "Current number of block capacity" its unit is not bytes/mb it is number of blocks

Highlighted

Re: Value of dfs.blocksize on ambari and BlockCapacity from namenodes REST response is different

New Contributor

@venkatsambath If it is the value of blocks, then how 2097152 blocks can be fit in 100gb space cluster each of 128 mb of block? Acutally i'm not getting how this block capacity value will be calculated, while calculating this metric total memory is considered or not?

I have single node hadoop cluster as well as multinode test envirnonment in both the value for block capacity is same 2097152.

Where should i found the block size in rest response? Is there any other URI for this metric? Initially it was assumed that the block capacity is same block size but over the time this value got changed.

Re: Value of dfs.blocksize on ambari and BlockCapacity from namenodes REST response is different

Expert Contributor

The capacity is not determined based on your cluster size rather estimated based on NN memory. Refer the code here for the computation details https://github.com/cloudera/hadoop-common/blob/134a29f22676e9a5226a07de6c6b9f34934a4625/hadoop-commo...

 

BlockSize details are not available in rest response of NN. But you may use RestAPI on ClouderaManager to get the configs. Example is in link: https://docs.cloudera.com/documentation/enterprise/5-14-x/topics/cm_intro_api.html#xd_583c10bfdbd326...

 

For Ambari check in link https://github.com/apache/ambari/blob/trunk/ambari-server/docs/api/v1/index.md

Don't have an account?
Coming from Hortonworks? Activate your account here