Member since
01-09-2019
29
Posts
3
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1448 | 06-03-2020 12:34 AM | |
1098 | 06-03-2020 12:24 AM | |
927 | 05-28-2020 02:37 AM |
01-07-2021
07:40 AM
Dear @janvanbesien , Were you able to migrate to a supported version of JDK 11, please? Kind regards, Julius
... View more
01-05-2021
05:36 AM
Hello Boenu, Were you able to resolve this issue, please? We upgraded our NiFi deployment today and are facing the same issue (tested in Firefox and Chrome). Kind regards, Julius EDIT: found the related JIRA issues, seems like there is no fix for it, just a workaround 😞 https://issues.apache.org/jira/browse/NIFI-7849 https://issues.apache.org/jira/browse/NIFI-7870
... View more
11-26-2020
02:49 AM
Hi SherKhan, Based on the screenshot, you have two conditions configured for the warning trigger. The first one will trigger if the free space falls below 5GB, the second one will trigger once the disk utilization is higher than 50%. The disk belonging to master.ndctech.net has 55.53% of it's capacity used, and so the warning is triggered. If you consider this to be a false alarm, you should raise the warning threshold from 50% to 70% perhaps. Kind regards, Gyuszi
... View more
09-30-2020
03:51 AM
@rbloughThank you for the continued support. 2) The command is being run as the hdfs user. 1) The detailed output showed that there are 603,723 blocks in total. Looking at the HDFS UI, the Datanodes report having 586,426 blocks each. 3) hdfs fsck / -openforwrite says that there are 506,549 blocks in total. The discrepancy in block count seems to be there still. Below are the summaries of the different fsck outputs. hdfs fsck / -files -blocks -locations -includeSnapshots Status: HEALTHY Number of data-nodes: 3 Number of racks: 1 Total dirs: 64389 Total symlinks: 0 Replicated Blocks: Total size: 330079817503 B (Total open files size: 235302 B) Total files: 625308 (Files currently being written: 129) Total blocks (validated): 603723 (avg. block size 546740 B) (Total open file blocks (not validated): 122) Minimally replicated blocks: 603723 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 3.0 Missing blocks: 0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Blocks queued for replication: 0 Erasure Coded Block Groups: Total size: 0 B Total files: 0 Total block groups (validated): 0 Minimally erasure-coded block groups: 0 Over-erasure-coded block groups: 0 Under-erasure-coded block groups: 0 Unsatisfactory placement block groups: 0 Average block group size: 0.0 Missing block groups: 0 Corrupt block groups: 0 Missing internal blocks: 0 Blocks queued for replication: 0 FSCK ended at Wed Sep 30 12:23:06 CEST 2020 in 23305 milliseconds hdfs fsck / -openforwrite Status: HEALTHY Number of data-nodes: 3 Number of racks: 1 Total dirs: 63922 Total symlinks: 0 Replicated Blocks: Total size: 329765860325 B Total files: 528144 Total blocks (validated): 506549 (avg. block size 651004 B) Minimally replicated blocks: 506427 (99.975914 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 2.9992774 Missing blocks: 0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Blocks queued for replication: 0 Erasure Coded Block Groups: Total size: 0 B Total files: 0 Total block groups (validated): 0 Minimally erasure-coded block groups: 0 Over-erasure-coded block groups: 0 Under-erasure-coded block groups: 0 Unsatisfactory placement block groups: 0 Average block group size: 0.0 Missing block groups: 0 Corrupt block groups: 0 Missing internal blocks: 0 Blocks queued for replication: 0 FSCK ended at Wed Sep 30 12:28:06 CEST 2020 in 11227 milliseconds
... View more
09-22-2020
11:23 PM
Hi @Madhur , Thank you for your prompt response. Unfortunately I do not have a subscription with Cloudera, I am unable to access the Knowledge Base. So far we have managed to get by with the free version of CDH 6.3 and the help of the community on and off these forums 🙂 Kind regards, Gyuszi Kovacs
... View more
09-22-2020
11:17 PM
@rblough- thank you very much for your prompt reply. 1) I got the block numbers from the HDFS status page in Cloudera Manager. Based on your question I checked the numbers on the NameNode UI. For the DEV environment all three DataNodes are online, showing 1,613,019 blocks on each (CM shows 1,613,104). For the PROD environment the NameNodeUI shows 477,464 blocks on each of the three DataNodes. 2) Yes, all of the DataNodes are showing as online. dfsadmin -report confirms this, so does the NameNodeUI and CM. Coincidentally, the report did not include the total block count, just the number of missing or under replicated blocks - everything sits at zero. 3) The replication factor is set to 3 in both environments. Kind regards, Gyuszi Kovacs
... View more
09-22-2020
07:01 AM
Greetings, I would like to clear up my understanding of how the block count is measured for the cluster. A bit of background information - we started receiving high block count warnings on Cloudera Manager (6.3), this lead to some investigating and cleanup. Currently I am trying to lower the block count on our DEV environment, but I am a bit confused. On Cloudera Manager, when I navigate to the HDFS service, and look at the Health Tests I see that there are "...1,608,301 total blocks in the cluster." However, when I run: sudo -u hdfs hdfs fsck / -files -blocks -locations the summary at the end states that: Replicated Blocks: Total size: 53194573887 B (Total open files size: 1557592 B) Total files: 569244 (Files currently being written: 202) Total blocks (validated): 553524 (avg. block size 96101 B) (Total open file blocks (not validated): 193) Minimally replicated blocks: 553524 (100.0 %) Now this is fine, from what I understand, the fsck command shows the block count without the replication factor of 3, so the numbers add up roughly. However, when I perform the same comparison on our PROD environment, I get the following. On Cloudera Manager I see that there are "... 449,966 total blocks in the cluster.", yet the fsck command returns: Replicated Blocks: Total size: 317645298827 B (Total open files size: 2529389 B) Total files: 389375 (Files currently being written: 142) Total blocks (validated): 368223 (avg. block size 862643 B) (Total open file blocks (not validated): 130) Minimally replicated blocks: 368223 (100.0 %) Could someone please explain the discrepancy between the numbers in this case? Thank you, kind regards, Gyuszi
... View more
Labels:
- Labels:
-
Cloudera Manager
-
HDFS
08-11-2020
08:31 AM
1 Kudo
More than a year later and there is still no open source version available, from what I can tell. Am I missing something? :S @Cloudera - know that Covid shook the world, but an updated roadmap would be appreciated. Cheers!
... View more
06-03-2020
12:34 AM
Hi @Maria_pl , Does the send-mail processor produce any errors when the message fails to send? Also, please check the NiFi logs (nifi-app.log should be the one that might give you a hint as to what is happening in the background, but while there you might want to check the rest of the NiFi logs). I see that you are using the latest version of NiFi. We are currently on version 1.10.0 and PutEmail works fine. Kind regards, Julius
... View more
06-03-2020
12:24 AM
Hello, This topic is described at length in the official documentation - https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_ig_ports.html Kind regards, Julius
... View more