Hi Team,   Spark job is hang or struck due to below error. can any one please help here..   22/12/05 22:29:55 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x5e29988e0x0, quorum=localhost:2181, baseZNode=/hbase 22/12/05 22:29:55 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) 22/12/05 22:29:55 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1126) ....   22/12/05 22:30:12 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 22/12/05 22:30:12 ERROR zookeeper.RecoverableZooKeeper: ZooKeeper exists failed after 4 attempts 22/12/05 22:30:12 WARN zookeeper.ZKUtil: hconnection-0x5e29988e0x0, quorum=localhost:2181, baseZNode=/hbase Unable to set watcher on znode (/hbase/hbaseid) org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid ... View more

Hi Team,   We have 3 DN's and Data Directories mount point size is around 1TB on each Node , total Data Directories size was 3TB. I deleted 500GB data from HDFS but space got release from only DN3 Data Directory not from other DN's.   Please advise..   ... View more

Hi Team,   Can you please explain me how to interpret  'Cluster CPU' graph from CM.     ... View more

Hi Team,   We have total 46 nodes cluster. 9(2 Master & 7 slave )  nodes on Baremetal and 37 nodes on VCP(VM's).    Can we set any limit/quota on CDH side by restrict using maximum bandwidth from N/W switches?? ... View more

Hi Team,   We have 2 Hive Metastore server services configured on cluster in node1 and node2. In node1 Hive Metastore getting alert with below message.   "Hive Metastore Canary"  The Hive Metastore canary failed to create a database.   I am seeing below errors in servicemonitor log file.   2022-10-12 22:40:06,657 WARN com.cloudera.cmf.cdh6client.hive.MetastoreClientImpl: (2 skipped) Could not drop hive database: cloudera_manager_metastore_canary_test_db_hive_HIVEMETASTORE_0137966f79e5f15b3b5d4dec61b7592e com.cloudera.cdh6client.hive.shaded.org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out at com.cloudera.cdh6client.hive.shaded.org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) at com.cloudera.cdh6client.hive.shaded.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at com.cloudera.cdh6client.hive.shaded.org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at com.cloudera.cdh6client.hive.shaded.org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at com.cloudera.cdh6client.hive.shaded.org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at com.cloudera.cdh6client.hive.shaded.org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:770) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:757) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropDatabase(HiveMetaStoreClient.java:940) at com.cloudera.cmf.cdh6client.hive.MetastoreClientImpl.dropDatabase(MetastoreClientImpl.java:163) at com.cloudera.cmon.firehose.polling.hive.HiveMetastoreCanary.cleanUpFromPreviousRuns(HiveMetastoreCanary.java:484) at com.cloudera.cmon.firehose.polling.hive.HiveMetastoreCanary.doWorkWithClientConfig(HiveMetastoreCanary.java:175) at com.cloudera.cmon.firehose.polling.hive.HiveMetastoreCanary.doWorkWithClientConfig(HiveMetastoreCanary.java:52) at com.cloudera.cmon.firehose.polling.AbstractCdhWorkUsingClientConfigs.doWork(AbstractCdhWorkUsingClientConfigs.java:45) at com.cloudera.cmon.firehose.polling.CdhTask$InstrumentedWork.doWork(CdhTask.java:230) at com.cloudera.cmf.cdhclient.util.ImpersonatingTaskWrapper.runTask(ImpersonatingTaskWrapper.java:72) at com.cloudera.cmf.cdhclient.util.ImpersonatingTaskWrapper.access$000(ImpersonatingTaskWrapper.java:21) at com.cloudera.cmf.cdhclient.util.ImpersonatingTaskWrapper$1.run(ImpersonatingTaskWrapper.java:107) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) at com.cloudera.cmf.cdh6client.security.UserGroupInformationImpl.doAs(UserGroupInformationImpl.java:42) at com.cloudera.cmf.cdhclient.util.ImpersonatingTaskWrapper.doWork(ImpersonatingTaskWrapper.java:104) at com.cloudera.cmf.cdhclient.CdhExecutor$1.call(CdhExecutor.java:125) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at com.cloudera.cdh6client.hive.shaded.org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ... 27 more   Please help me on this issue,   ... View more

Hi ,   We upgraded OS of one of the server in cluster from RHEL 7.5 to 7.9. After that we are unable to start cloudera services from webui, getting below errors in agent log file. can anyone help me to get rid of this issue.   AttributeError: 'NoneType' object has no attribute 'get' [30/Sep/2022 19:37:38 +0000] 1048 MainThread agent WARNING Long HB processing time: 7.00362992287 [30/Sep/2022 19:38:01 +0000] 1048 DnsResolutionMonitor throttling_logger INFO DnsTest not running. Java not located. [30/Sep/2022 19:38:37 +0000] 1048 MonitorDaemon-Reporter firehoses INFO Creating a connection to the ACTIVITYMONITOR. [30/Sep/2022 19:38:37 +0000] 1048 MonitorDaemon-Reporter firehoses INFO Creating a connection to the SERVICEMONITOR. [30/Sep/2022 19:38:37 +0000] 1048 MonitorDaemon-Reporter firehoses INFO Creating a connection to the HOSTMONITOR. [30/Sep/2022 19:38:37 +0000] 1048 MonitorDaemon-Reporter throttling_logger ERROR Error sending messages to firehose: mgmt-HOSTMONITOR-d85e01b86fca15e35281bae2797b3c77 Traceback (most recent call last): File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/firehose.py", line 121, in _send self._port) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 469, in __init__ self.conn.connect() File "/usr/lib64/python2.7/httplib.py", line 837, in connect self.timeout, self.source_address) File "/usr/lib64/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 111] Connection refused [30/Sep/2022 19:47:09 +0000] 1048 MainThread heartbeat_tracker INFO HB stats (seconds): num:46 LIFE_MIN:0.00 min:0.00 mean:0.02 max:0.26 LIFE_MAX:0.01 ~ ... View more