Could be similar behavior like this jira? https://issues.apache.org/jira/browse/HDFS-14800 Please try to tune the below properties from hbase: <property> <name>hbase.lease.recovery.timeout</name> <value>900000</value> <description>How long we wait on dfs lease recovery in total before giving up.</description> </property> <property> <name>hbase.lease.recovery.dfs.timeout</name> <value>64000</value> <description>How long between dfs recover lease invocations. Should be larger than the sum of the time it takes for the namenode to issue a block recovery command as part of datanode; dfs.heartbeat.interval and the time it takes for the primary datanode, performing block recovery to timeout on a dead datanode; usually dfs.client.socket-timeout. See the end of HBASE-8389 for more.</description> </property> ... View more

so it's datanode not namenode, can we get the full error stack of this: org.apache.hadoop.net.ConnectTimeoutException: Call From ics168027095.ics-eu-2.example.com/192.168.27.95 to ics168226185.ics-eu-2.example.com:8010 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=ics168226185.ics-eu-2.example.com/192.168.226.185:8010]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout also the full rack topology and the role allocation details. ... View more

@sagarshimpi , I agree with the first part of your analysis that the master was waiting for 16 minutes to recover a pv2 log.  But looking at this error: org.apache.hadoop.net.ConnectTimeoutException: Call From ics168027095.ics-eu-2.example.com/192.168.27.95 to ics168226185.ics-eu-2.example.com:8010 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=ics168226185.ics-eu-2.example.com/192.168.226.185:8010]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout It is not the datanode call to another datanode, but it should be the datanode call to the NameNode port, 8010 can be the custom port but you can check the configuration what is this port, I assume it is NameNode port. So I would look at NameNode logs and FailoverController logs to understand if the active NameNode was successfully failover to another NameNode at that time and how long it finished the failover. Can we know the full topology of the allocation of the roles in your zone1 and zone2, we would like to know how you allocate NN, DN, JN, ZKFC, HMasters, RS, ZK. ... View more

Some steps to help narrow down the issue, if possible please attach outputs or the answer to the below items: 1. In order to know if the issue is from CM or HDFS, can you please check if the Standby NN's process is existing or not by running: ps -ef|grep -i namenode 2. Please check do we have any ERROR/WARN in the latest Standby NN, Are there any GC pause issues detected in the Standby NN's log? Attaching the errors may help us know the issue better. 3. Please check the status of cloudera-scm-agent by running the below commands, make sure the agent is Active: systemctl status cloudera-scm-agent 4. How about the other services on this host, they are all good but only SNN has this issue? 5. Please try to open NN webUI and SNN webUI from the browser, if SNN is up and running, the webUI should be good: The default webUI port is 9870 http://NN_ip:9870/dfshealth.html#tab-overview 6. Please check if cpu utilization and memory utilization are sufficient in this SNN host. 7. When and how did this issue happen, did it happen after the restarting? ... View more

@Sindhu6 Please refer to this phoenix doc for java example and url syntax: https://phoenix.apache.org/faq.html#What_is_the_Phoenix_JDBC_URL_syntax An example URL is  jdbc:phoenix:thin:url=http://hostname:8765;serialization=PROTOBUF;authentication=SPNEGO; principal=hbase/hostname@EXAMPLE; keytab=/var/run/cloudera-scm-agent/process/xx-hbase-REGIONSERVER/hbase.keytab (due to system will hide long text, devide into multiple lines, but you need write url into single line.) Jar is /opt/cloudera/parcels/CDH-7.1.x/jars/phoenix-queryserver-client-xxx.jar Driver Class is "org.apache.phoenix.queryserver.client.Driver" You should replace keytab, principle, jar with your own and start to test single java class instead of complex projects. ... View more

Please check if your principal and keytab are set correctly. Here is another example: thick client: https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/phoenix_using.html thin client: https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/phoenix_thin_client_pqs.html     ... View more

Hi @Sindhu6, Please make sure phoenix and hbase are functional by accessing phoenix-sqlline and creating a test table then select data from it prior to using jdbc. Then please refer to the below doc for phoenix jdbc usage in cdp: https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/phoenix-access-data/topics/phoenix-orchestrating-sql.html ... View more

Hi @bavisetti, w riting to phoenix tables from HBase is not supported, please consider writing Phoenix tables from Phoenix only. ... View more

Please refer to this doc https://blog.cloudera.com/apache-hbase-region-splitting-and-merging/ for split policy.   So far based on your statement I cannot conclude it is due to salt bucket or split policy, we need more evidence from logs.   So we would suggest you raise a Cloudera support case. We need to collect some necessary information and logs to investigate.   Please make sure the above questions are answered, in addition, we also need to collect: hbase: - echo "scan 'namespace:tablename'" > /tmp/scan_meta.txt - echo "describe 'namespace:tablename'" > /tmp/desc_table.txt - echo "list_regions 'namespace:tablename'">/tmp/list_regions.txt phoenix-sqlline: - select * from system.catalog; - !tables - select * from namespace.table; - Your client code of using phoenix driver and the output reflects the issue " when I am querying data through Phoenix driver rowkey value is getting truncated (only the first letter) and other columns are good. " ... View more