03-15-2019 12:47 PM
I am experimenting around with the
Param and want to learn more about it.
One confusion I have is in our cluster, we are using the default value which is 6 hours.
But when I check the hdfs log (logs/hbase-hadoop/ hadoop-datanode-hdaas-dnds1-3-prd.eng.sfdc.net.log) on one of the data node, I observe the following record:
2019-02-23 02:31:20,779 INFO [net/10.231.0.10:8020] datanode.DataNode - Successfully sent block report 0x7fd496451f3c49a6, containing 4 storage report(s), of which we sent 4. The reports had 84393 total blocks and used 1 RPC(s). This took 12 msec to generate and 328 msecs for RPC and NN processing. Got back one command: FinalizeCommand/5.
And I assume this should be the event that dn is sending the block report to nn.
Then I observe that the frequency between each of these event is definitely not 6 hours but much more frequent than that (around 1-2 hours).
I know there is another param "dfs.blockreport.incremental.intervalMsec" which controls the incemental block report, but our hadoop version does not support this yet.
Are there anyting else impacting the block report send frequency?
Or is the incremental block report sent by default?