Hi @sipocootap2    Answered it here - https://community.cloudera.com/t5/Support-Questions/How-can-I-get-fsimage-with-curl-command/m-p/314859/highlight/false#M226223   cheers, ... View more

Hi @sipocootap2 ,   AFAIK "/getimage" is deprecated in CDH and we suggest this not to be used. Instead you can use the command " hdfs dfsadmin -fetchImage <dir>" to download & save the latest fsimage.   Based on research, in earlier versions of CDH, the getImage method was available after which a need was realized to provide a proper command/utility to download the FSimage and as a result of which "hdfs dfsadmin -fetchImage" was born. Once that was put in place, the getImage was removed.   Does that answers your questions ? If yes, feel free to mark this post as "accept as solution"   Regards,     ... View more

Hi @ROACH    Ideally we recommend 1gb heap per 1 million blocks.   Also , How much memory you actually need depends on your workload, especially on the number of files, directories, and blocks generated in each namespace. Type of hardware VM or bare metal etc also taken into account. https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/admin_nn_memory_config.html   Also have a look at examples of estimating namenode heap memory https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/hdfs-overview/topics/hdfs-examples-namenode-heap-memory.html   If write intensive operations or snapshots operations are being performed on Cluster oftenly then 6-9 gb sounds fine. I would suggest to do grep for GC in Namenode logs and if you see long Pauses says for more than 3-5 seconds then its a good starting point to increase the heap size.   Does that answer your question. do let us know.   regards,       ... View more

Hi @Chetankumar ,   I think we answered in this thread :- https://community.cloudera.com/t5/Support-Questions/How-to-move-block-from-one-mount-point-to-other-and-remove/td-p/314861   If that answers all your questions, feel free to mark the post "accepted solution"    Regards,   ... View more

Hi @rubysimmons63    Also, Falcon is explained in detail in contrast with Atlas here, do check it out for better understanding :- https://community.cloudera.com/t5/Support-Questions/What-is-the-difference-between-Apache-atlas-and-Apache/m-p/122450   https://www.cloudera.com/products/open-source/apache-hadoop/apache-falcon.html   Regards,   ... View more

Hi @ryu ,   We can trigger manual GC in datanode JVM application AFAIK.  The best way to deal with long GC pauses is to allocate right amount of heap memory.  We recommend the formula of - 1gb heap per 1 million blocks. You can get the number of block count from NameNode Webui -> datanode (through ambari or CM). Increase the heap and that should fix your issue.  Do check for "No GC detected" in Datanode logs, if you see those then it could be hardware problem triggering GC.   Does that answers your questions. Let me know   Regards, Vipin ... View more

Hi @Seeker90 ,   The ERROR message that you see  is because you are running ZK in standalone mode. This is more of a warning than ERROR. Invalid configuration, only one server specified (ignoring) Further i see the ZK started properly, However while reading snapshots it throws Exception.  Probable causes of Canary test failure &  ZooKeeper Quorum:   1. Max Client Connections is set too low -  2. Long fsyncs (disk writes) - 3. Insufficient heap (long GCs) -  Try below :-  1. Increasing ZK heap size (maybe undersized heap or if size of snapshots is huge, increasing heap would be good starting point ) 2. Increase maximum number of connection to 300 3. grep for "fsync" in ZK logs. Check if ZK disk is independent.    Does that answers your questions. Do let us know.   Regards,     ... View more

Hi @rocky_tian    Although it appears to be https://issues.apache.org/jira/browse/KUDU-2412  but since you are on el7. Not sure if this jira is valid for your case.   Ensure all the prerequisites  kudu libraries are installed as mentioned in the doc :-  https://kudu.apache.org/docs/installation.html#_install_on_rhel_or_centos_hosts   Related external link -  https://stackoverflow.com/questions/52526013/how-to-read-from-kudu-to-python    Regards, Vipin ... View more

Hi @rOckChew    Its consensus error. we would need to validate if all masters are voting to leader.   In a multi master Kudu environment, if a master is restarted or goes offline for a few minutes, it can occasionally have trouble joining the cluster on startup. For example, if this happens in case of three kudu masters, and one of the other two masters is stopped or dies during this time, then the overall Kudu cluster is down because the majority of the masters are not running. This issue is resolved by the   KUDU-2748   upstream JIRA.   https://my.cloudera.com/knowledge/TSB-2020-442-Kudu-Masters-unable-to-join-back-after-a?id=304920   Run   sudo -u kudu kudu cluster ksck <master1,master2,master3>   to find no ‘*’ on one of the masters’ consensus.    >>A quick workaround is to restart all the kudu masters together. >> Rewrite kudu master consensus   Thanks Vipin   ... View more

Hi @JeromeAlbin    Looks like https://issues.apache.org/jira/browse/IMPALA-9486   The Error pop up  because you are connecting to Impala anonymously (no user, no password). You can specify a user (even if it's not declared in Kudu), then it should work Please read the page 12 of the following document: https://docs.cloudera.com/documentation/other/connectors/impala-jdbc/2-6-15/Cloudera-JDBC-Driver-for-Impala-Install-Guide.pdf Using User Name ----------------------- This authentication mechanism requires a user name but does not require a password. The user name labels the session, facilitating database tracking.   Does that answer your question ? if yes, then feel free to mark this post "accept as solution"   Regards, vipin ... View more

Hi @Smashedcat32  To give some background on the ZooKeeper Canary, the ServiceMonitor will regularly check the health of the ZooKeeper Service by   1. connecting to the ZooKeeper quorum and locate the leader 2. create a znode 3. read the znode 4 deleting the znode.   If any of these steps fail the ServiceMonitor will report the ZOOKEEPER_CANARY_HEALTH has become bad.   In the health reported above, the reason was "Canary test failed to establish a connection or a client session to the ZooKeeper service", which means it failed on step 1.   The problem could lie in three locations:   1. The ZooKeeper Quorum - Fsync, low GC , Low max client connections 2. The Service Monitor - false reports 3. Network connectivity between the Service Monitor and the ZooKeepers   Now coming to your query regarding canary test commands, i dont think we have it available in docs. You can use the commands from ZK guide to test Example - To verify if the ZK instance is leader echo stat | nc ZOOKEEPER_IP ZOOKEEPER PORT | grep Mode  http://www.corejavaguru.com/bigdata/zookeeper/cli https://zookeeper.apache.org/doc/r3.3.3/zookeeperStarted.html#sc_ConnectingToZooKeeper   ... View more

The disk space occupied by a deleted row is only reclaimable via compaction and  given you have deleted some data and if the space is not reclaimed then probably you are hitting the bug  https://issues.apache.org/jira/browse/KUDU-1625   The jira stands unresolved. However if the goal is to delete the data and reclaim disk space, then you can drop partition (if range partition) in order to reclaim space.    Tombstone tablets have all their data removed from disk and don't consume significant resources. These tablet are necessary for correct operation of kudu. See - https://docs.cloudera.com/runtime/7.1.0/troubleshooting-kudu/topics/kudu-tombstoned-or-stopped-tablet-replicas.html   ... View more

Ideally if you have dropped the table then the data should get deleted immediately. The metrics in CM may take some time to reflect, we can verify from backend if the table is actually deleted.    Verify if the table still exist in kudu FS. You can verify this by using kudu ksck command with -tables flags :- kudu cluster ksck <master_addresses> -tables=<tables>   Note if the table created through impala use "impala::db.tablename"   If you see the table in ksck then run below command to delete the table from kudu:- kudu table delete <master_addresses> <table_name>    ... View more

@mike_bronson7  Adding to @GangWar :-   To your question -  dose this action could also affected the data itself on the data-nodes machines ?   No it doesnt affect data on datanode directly. This is metadata operation on namenode which when need to be run  when  NameNode fails to progress through the edits or fsimage  then the NameNode may need to be started with -recover option.    Since the metadata has reference to the blocks on datanode, hence this is a critical operation and may incur data loss. ... View more

Adding to @smdas    This is one of the kudu limitations :- " There is no way to run compaction manually, but dropping the table will reclaim the space immediately."    You can verify the size from CM graphs:- Go to the Kudu service and navigate to the  Charts Library  tab. On the left-hand side menu, click  Tables  to display the list of tables currently stored in Kudu. Click on a table name to view the default dashboard for that table. The  Total Tablet Size On Disk Across Kudu Replicas  chart displays the total size of the table on disk using a time-series chart. Hovering with your mouse over the line on the chart opens a small pop-up window that displays information about that data point. Click the data stream within the chart to display a larger pop-up window that includes additional information for the table at the point in time where the mouse was clicked.   reference :- http://apache.github.io/kudu/docs/known_issues.html#_other_usage_limitations ... View more

Hi Prash,   By default CM will throw an alert on CM->HDFS ->status page when the namenode is in Safe Mode.    NameNode Safemode Health Test - Enables the health test that the NameNode is not in safemode https://docs.cloudera.com/documentation/enterprise/6/properties/6.1/topics/cm_props_cdh5150_hdfs.html You can configure email Alert, using the below  https://docs.cloudera.com/documentation/enterprise/latest/topics/cm_ag_email.html   Let me know if that helps.   Thanks Vipin ... View more

The connectivity between the DN and NN has to be gauged. Check whether DN is able to resolve the NN hostname (forward & reverse ). ... View more

The error means maximum number of folders or file in one directory has reached its limit and thats when we see the error.    " The directory item limit of /tmp/logs/Appln/logs is exceeded: limit=1048576 items=1048576"   There's a parameters which we can invoke in NameNode safety valves in order to increase this limit   ->  dfs.namenode.fs-limits.max-directory-items determines the maximum number of folders or files (not recursive) in one directory. The value range of this parameter is 1 to 6400000, and the default value is 1048576. -> After setting the above parameter, restart the NameNode to make changes come into effect and verify with   hdfs-site.xml file whether changes are imbibed.     ... View more

Hi @dsht6955 , Can you try running the hdfs oiv command with -t flag as show below and see if that helps:   hdfs oiv -p Delimited -delimiter delimiterString -t temporaryDir -i fsimage -o output.xml   Thanks Vipin ... View more

It seems NN refuses to connect to DN, it could be DN is not there in include list file on NN or maybe DNS issue (check /etc/hosts).  Can you run the below command and see if the error goes off- hdfs dfsadmin -refreshNodes   Further, Check this link - https://stackoverflow.com/questions/17252955/getting-the-following-error-datanode-denied-communication-with-namenode-while/29598059#29598059 ... View more

Hi Vijaya,   Under-replicated blocks are the blocks which does not meet the target replication factor. HDFS has self healing mechanism wherein it will create  new replicas of under-replicated blocks until they meet the target replication.  if multi node cluster then you can verify if any datanodes are down  Or if it is one node cluster then please ensure replication factor is set to 1.    Thanks. ... View more

-> I dont think there's a way as of now to check the overall capacity of kudu. However, You can check the individual kudu tserver size on disk by using the linux command on the host "du -sh" /<kudu-path>/   Further you can check the individual table size on disk by using CM charts.   How much space a Kudu table is using on disk: 1. Go to the Kudu service and navigate to the Charts Library tab. 2. On the left-hand side menu, click Tables to display the list of tables currently stored in Kudu. 3. Click on a table name to view the default dashboard for that table. The Total Tablet Size On Disk Across Kudu Replicas chart displays the total size of the table on disk using a time-series chart. Hovering with your mouse over the line on the chart opens a small pop-up window that displays information about that data point. Click the data stream within the chart to display a larger pop-up window that includes additional information for the table at the point in time where the mouse was clicked.   Link- https://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_kudu_service.html#charts_library Hope this helps. ... View more

Hi,   When the NameNode flushes the edits to Journal Nodes it maintains the quorum of 20 seconds. The reason you are seeing this Error message is because it took >20 sec for NN to send the edits. This could be because of various reasons i.e NN GC or JVM pause, whether JN is sharing the disks with other roles, network communication issues , slow group lookups  etc.    Checking the NameNode logs just before the FATAL message would be a good starting point. Check for Warning messages just before the FATAL error message on NameNode logs. ... View more

Addition to what Ben has said, You may want to check if the Journal Node is sharing the disk with any other roles.  Journal Node shouldnt be sharing its disk with any other roles (not even at OS level roles) As this will effect the JN response time under high load and cause slow sync problem.  ... View more

It seems like the DataNode webserver is not responding to CM agent. This could be because of various reasons, from my experience i would suggest you to check for "JVM" Pauses or " Slow BlockReceiver" in the DataNode logs as those are primarliy the reason for slow response from the DN to CM agent.  Datanode logs which you have provided  give some  hint about the slow block receiver but just to be sure check again filter the above mentioned keywords in datanode logs.    Whats the frequency of this alert on CM? is it just a one time alert or persistent ?   You can check your Datanode heap utilization by using the chart. Here's how you can do it CM-> chart builder-> select jvm_max_memory_mb, jvm_heap_used_mb where entityName="<DN-instance"       ... View more