Cloudera Community
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
uxadmin
user rank
New Contributor

Re: Block Count threshold configuration

by New Contributor in Support Questions
‎03-05-2021 04:40 AM
‎03-05-2021 04:40 AM
Hello @PabitraDas , that's the information I needed thanks a lot! br ... View more

Re: Block Count threshold configuration

by New Contributor in Support Questions
‎03-05-2021 02:36 AM
‎03-05-2021 02:36 AM
Hello @PabitraDas , thanks for the clarification but may I ask you one more thing. The block count is a DN issue but also an NN issue as it has to keep the file info in memory right? So therefore we may not exceed a certain limit otherwise the cluster environment will suffer performance issues as far as I understand the context right. So my question is: what would be a feasible formula to set this threshold value and which  values would influence the value number (like a number of NN disks ? Memory ? ) br    ... View more

Block Count threshold configuration

by New Contributor in Support Questions
‎03-04-2021 01:53 AM
‎03-04-2021 01:53 AM
Hello, I'm somewhat confused about the hdfs block count threshold configuration. As a rule of thumb, I read about setting  the threshold like  1 GB Java Heap Size of NameNode ~= 1M Blocks so with our setting of 10GB it would mean  the threshold would be 10M Blocks but then I read the replication factor ( x3 standard ) isn't included ? So does this mean I have to set the threshold to 3,3M Blocks if I set the Java Heap Size of NameNode to 10GB ? thx Martin ... View more
Contact Me
Online
Offline
Last Visited
‎09-13-2021 08:09 AM
Community Statistics
Member Since ‎08-17-2020 04:03 AM
Last Visited ‎09-13-2021 08:09 AM
Posts 3