Hi Prash,   By default CM will throw an alert on CM->HDFS ->status page when the namenode is in Safe Mode.    NameNode Safemode Health Test - Enables the health test that the NameNode is not in safemode https://docs.cloudera.com/documentation/enterprise/6/properties/6.1/topics/cm_props_cdh5150_hdfs.html You can configure email Alert, using the below  https://docs.cloudera.com/documentation/enterprise/latest/topics/cm_ag_email.html   Let me know if that helps.   Thanks Vipin ... View more

The connectivity between the DN and NN has to be gauged. Check whether DN is able to resolve the NN hostname (forward & reverse ). ... View more

The error means maximum number of folders or file in one directory has reached its limit and thats when we see the error.    " The directory item limit of /tmp/logs/Appln/logs is exceeded: limit=1048576 items=1048576"   There's a parameters which we can invoke in NameNode safety valves in order to increase this limit   ->  dfs.namenode.fs-limits.max-directory-items determines the maximum number of folders or files (not recursive) in one directory. The value range of this parameter is 1 to 6400000, and the default value is 1048576. -> After setting the above parameter, restart the NameNode to make changes come into effect and verify with   hdfs-site.xml file whether changes are imbibed.     ... View more

Hi @dsht6955 , Can you try running the hdfs oiv command with -t flag as show below and see if that helps:   hdfs oiv -p Delimited -delimiter delimiterString -t temporaryDir -i fsimage -o output.xml   Thanks Vipin ... View more

It seems NN refuses to connect to DN, it could be DN is not there in include list file on NN or maybe DNS issue (check /etc/hosts).  Can you run the below command and see if the error goes off- hdfs dfsadmin -refreshNodes   Further, Check this link - https://stackoverflow.com/questions/17252955/getting-the-following-error-datanode-denied-communication-with-namenode-while/29598059#29598059 ... View more

Hi Vijaya,   Under-replicated blocks are the blocks which does not meet the target replication factor. HDFS has self healing mechanism wherein it will create  new replicas of under-replicated blocks until they meet the target replication.  if multi node cluster then you can verify if any datanodes are down  Or if it is one node cluster then please ensure replication factor is set to 1.    Thanks. ... View more

-> I dont think there's a way as of now to check the overall capacity of kudu. However, You can check the individual kudu tserver size on disk by using the linux command on the host "du -sh" /<kudu-path>/   Further you can check the individual table size on disk by using CM charts.   How much space a Kudu table is using on disk: 1. Go to the Kudu service and navigate to the Charts Library tab. 2. On the left-hand side menu, click Tables to display the list of tables currently stored in Kudu. 3. Click on a table name to view the default dashboard for that table. The Total Tablet Size On Disk Across Kudu Replicas chart displays the total size of the table on disk using a time-series chart. Hovering with your mouse over the line on the chart opens a small pop-up window that displays information about that data point. Click the data stream within the chart to display a larger pop-up window that includes additional information for the table at the point in time where the mouse was clicked.   Link- https://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_kudu_service.html#charts_library Hope this helps. ... View more

Hi,   When the NameNode flushes the edits to Journal Nodes it maintains the quorum of 20 seconds. The reason you are seeing this Error message is because it took >20 sec for NN to send the edits. This could be because of various reasons i.e NN GC or JVM pause, whether JN is sharing the disks with other roles, network communication issues , slow group lookups  etc.    Checking the NameNode logs just before the FATAL message would be a good starting point. Check for Warning messages just before the FATAL error message on NameNode logs. ... View more

Addition to what Ben has said, You may want to check if the Journal Node is sharing the disk with any other roles.  Journal Node shouldnt be sharing its disk with any other roles (not even at OS level roles) As this will effect the JN response time under high load and cause slow sync problem.  ... View more

It seems like the DataNode webserver is not responding to CM agent. This could be because of various reasons, from my experience i would suggest you to check for "JVM" Pauses or " Slow BlockReceiver" in the DataNode logs as those are primarliy the reason for slow response from the DN to CM agent.  Datanode logs which you have provided  give some  hint about the slow block receiver but just to be sure check again filter the above mentioned keywords in datanode logs.    Whats the frequency of this alert on CM? is it just a one time alert or persistent ?   You can check your Datanode heap utilization by using the chart. Here's how you can do it CM-> chart builder-> select jvm_max_memory_mb, jvm_heap_used_mb where entityName="<DN-instance"       ... View more

Hi Arun, This error is very generic timeout error. It would be better if you could share the exact problem and share thread dumps. This timeout error is very common and you will be getting it when you are not able to communicate to a service. ... View more