Member since
08-17-2020
3
Posts
0
Kudos Received
0
Solutions
03-05-2021
04:40 AM
Hello @PabitraDas , that's the information I needed thanks a lot! br
... View more
03-05-2021
02:36 AM
Hello @PabitraDas , thanks for the clarification but may I ask you one more thing. The block count is a DN issue but also an NN issue as it has to keep the file info in memory right? So therefore we may not exceed a certain limit otherwise the cluster environment will suffer performance issues as far as I understand the context right. So my question is: what would be a feasible formula to set this threshold value and which values would influence the value number (like a number of NN disks ? Memory ? ) br
... View more
03-04-2021
01:53 AM
Hello, I'm somewhat confused about the hdfs block count threshold configuration. As a rule of thumb, I read about setting the threshold like 1 GB Java Heap Size of NameNode ~= 1M Blocks so with our setting of 10GB it would mean the threshold would be 10M Blocks but then I read the replication factor ( x3 standard ) isn't included ? So does this mean I have to set the threshold to 3,3M Blocks if I set the Java Heap Size of NameNode to 10GB ? thx Martin
... View more
Labels:
- Labels:
-
Cloudera Data Platform (CDP)
-
HDFS