Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

[Block Report Frequency] dfs.blockreport.intervalMsec 21600000 does not seem to control the actual

Highlighted

[Block Report Frequency] dfs.blockreport.intervalMsec 21600000 does not seem to control the actual

New Contributor

Hi Folks,

 

I am experimenting around with the 

dfs.blockreport.intervalMsec21600000

 

Param and want to learn more about it.

One confusion I have is in our cluster, we are using the default value which is 6 hours.

But when I check the hdfs log (logs/hbase-hadoop/ hadoop-datanode-hdaas-dnds1-3-prd.eng.sfdc.net.log) on one of the data node, I observe the following record:

 

2019-02-23 02:31:20,779 INFO  [net/10.231.0.10:8020] datanode.DataNode - Successfully sent block report 0x7fd496451f3c49a6,  containing 4 storage report(s), of which we sent 4. The reports had 84393 total blocks and used 1 RPC(s). This took 12 msec to generate and 328 msecs for RPC and NN processing. Got back one command: FinalizeCommand/5.

 

And I assume this should be the event that dn is sending the block report to nn.

Then I observe that the frequency between each of these event is definitely not 6 hours but much more frequent than that (around 1-2 hours).

 

I know there is another param "dfs.blockreport.incremental.intervalMsec" which controls the incemental block report, but our hadoop version does not support this yet.

 

Are there anyting else impacting the block report send frequency?

Or is the incremental block report sent by default?

 

Thanks