Receiving Namenode High Availability Status transient alerts by saying that unknow host standby and unable to determine the active namenode. In logs before the error it is saying that "unable to extract JSON from JMX Response" from the script. The issue is automatically getting solved. we have defualt settings grace alert up to 5 seconds. Please help what is the exact reason for this. No more information in namenode logs just seeing these error in ambari-alerts and agent files. ... View more

How will check the previous date alerts like NN Heap usage,Host agent disk alerts etc. we need to deploy some script. ... View more

Queries are getting operation timed out because scan is happening on Index table (which don’t have K0 column) K0 column is not part of MESSAGE_ARCHIVE_2_IDX_DEVICEID_PROCESSED_TIMESTAMP table but scan is happening on index table causing the operation timed out. ... View more

After the RM, Hbase HA configurations & cluster Restart the jobs were not starting in the master machine and entry automatically went wrong in the /etc/hosts and after some time I ran the jobs in the HA node and started working after moving the master services to HA. Now port also not listening to the IP address in the master.capture.png Please help why this issue is happened and how to solve. ... View more

We are planning to replace the faulty cache battery on the master server.Currently, in our set up HA is configured but except for Resource manager HA & Hbase HA. Also planning to configure the RM & Hbase HA in the HA node during this window.what is the best way of doing the shutdown and startup of the master server. Attaching the master and HA server components.Please guide what is the best way to do this activity. master-server-components.pngha-node.png ... View more

After Restarting also phoenix server is getting stopped .After restart it will work for some seconds and again getting stopped.Please help. logs:- INFO org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x15ca2ca52ba3d42 2018-02-25 11:15:54,847 FATAL org.apache.phoenix.queryserver.server.Main: Unrecoverable service error. Shutting down. java.lang.RuntimeException: java.sql.SQLException: ERROR 2006 (INT08): Incompatible jars detected between client and server. Ensure that phoenix.jar is put on the classpath of HBase in every region server: SYSTEM.CATALOG is disabled. at org.apache.phoenix.queryserver.server.PhoenixMetaFactoryImpl.create(PhoenixMetaFactoryImpl.java:73) at org.apache.phoenix.queryserver.server.Main.run(Main.java:203) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.phoenix.queryserver.server.Main.main(Main.java:226) Caused by: java.sql.SQLException: ERROR 2006 (INT08): Incompatible jars detected between client and server. Ensure that phoenix.jar is put on the classpath of HBase in every region server: SYSTEM.CATALOG is disabled. at org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:386) at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145) at org.apache.phoenix.query.ConnectionQueryServicesImpl.checkClientServerCompatibility(ConnectionQueryServicesImpl.java:990) at org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:869) ... View more

capture.pngcapture2.pngHcat client, Mapreduce2 client, Hive Client & Yarn client not able to restart successfully. ... View more

Active Hbase master goes down & and failover happened by making standby Hbase master as up.But after some time all the region servers goes down one by one and standby Hbase master also goes down and finally whole HBase cluster goes offline.We have started all the services make up. Active Hbase Master LOGS : - 2018-01-17 16:22:29,895 ERROR [master/post-om2.vodafone.flytxt.com/10.88.8.79:16000] master.ActiveMasterManager: master:16000-0x35ee3bbff600001, quorum=post-om2.vodafone.flytxt.com:2181,post-om1.vodafone.flytxt.com:2181,post-os1.vodafone.flytxt.com:2181, baseZNode=/hbase Error deleting our own master address node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:359) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:745) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:148) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:267) at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1145) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1071) at java.lang.Thread.run(Thread.java:744) 2018-01-17 16:22:29,895 INFO [master/post-om2.vodafone.flytxt.com/10.88.8.79:16000] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x15ee3bbf9910003 2018-01-17 16:22:29,897 INFO [master/post-om2.vodafone.flytxt.com/10.88.8.79:16000] zookeeper.ZooKeeper: Session: 0x15ee3bbf9910003 closed 2018-01-17 16:22:29,897 INFO [post-om2:16000.activeMasterManager-EventThread] zookeeper.ClientCnxn: EventThread shut down 2018-01-17 16:22:29,899 INFO [master/post-om2.vodafone.flytxt.com/10.88.8.79:16000] flush.MasterFlushTableProcedureManager: stop: server shutting down. 2018-01-17 16:22:29,899 INFO [master/post-om2.vodafone.flytxt.com/10.88.8.79:16000] ipc.RpcServer: Stopping server on 16000 2018-01-17 16:22:29,900 INFO [RpcServer.listener,port=16000] ipc.RpcServer: RpcServer.listener,port=16000: stopping Active /old Standby HBase Master logs : - 2018-01-17 16:23:06,025 INFO [post-om1:16000.activeMasterManager] master.ActiveMasterManager: Registered Active Master=post-om1.vodafone.flytxt.com,16000,1507059524710 2018-01-17 16:24:46,275 INFO [post-om1:16000.activeMasterManager] master.AssignmentManager: Joined the cluster in 119ms, failover=true Post-os10 goes down logs in the post-om1: 2018-01-17 18:45:35,364 ERROR [PriorityRpcServer.handler=11,queue=1,port=16000] master.MasterRpcServices: Region server post-os10.vodafone.flytxt.com,16020,1507059587301 reported a fatal error: ABORTING region server post-os10.vodafone.flytxt.com,16020,1507059587301: IOE in log roller Cause: java.io.IOException: cannot get log writer at org.apache.hadoop.hbase.wal.DefaultWALProvider.createWriter(DefaultWALProvider.java:365) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.createWriterInstance(FSHLog.java:746) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:711) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:137) at java.lang.Thread.run(Thread.java:744) Caused by: java.io.FileNotFoundException: Parent directory doesn't exist: /apps/hbase/data/WALs/post-os10.vodafone.flytxt.com,16020,1507059587301 2018-01-17 18:54:21,800 ERROR [PriorityRpcServer.handler=11,queue=1,port=16000] master.MasterRpcServices: Region server post-os5.vodafone.flytxt.com,16020,1513481333569 reported a fatal error: ABORTING region server post-os5.vodafone.flytxt.com,16020,1513481333569: IOE in log roller Cause: java.io.IOException: cannot get log writer Sample Region server logs : - 2018-01-17 18:52:10,311 INFO [regionserver/post-os5.vodafone.flytxt.com/10.88.8.76:16020-SendThread(post-om1.vodafone.flytxt.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server post-om1.vodafone.flytxt.com/10.88.8.71:2181. Will not attempt to authenticate using SASL (unknown error) 2018-01-17 18:52:10,311 INFO [regionserver/post-os5.vodafone.flytxt.com/10.88.8.76:16020-SendThread(post-om1.vodafone.flytxt.com:2181)] zookeeper.ClientCnxn: Socket connection established to post-om1.vodafone.flytxt.com/10.88.8.71:2181, initiating session 2018-01-17 18:52:10,313 INFO [regionserver/post-os5.vodafone.flytxt.com/10.88.8.76:16020-SendThread(post-om1.vodafone.flytxt.com:2181)] zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x15ee3bbf9933b39 has expired, closing socket connection 2018-01-17 18:52:10,313 WARN [regionserver/post-os5.vodafone.flytxt.com/10.88.8.76:16020-EventThread] client.ConnectionManager$HConnectionImplementation: This client just lost it's session with ZooKeeper, closing it. It will be recreated next time someone needs it org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:606) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:517) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) 2018-01-17 18:52:10,313 INFO [regionserver/post-os5.vodafone.flytxt.com/10.88.8.76:16020-EventThread] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x15ee3bbf9933b39 2018-01-17 18:52:10,313 INFO [regionserver/post-os5.vodafone.flytxt.com/10.88.8.76:16020-EventThread] zookeeper.ClientCnxn: EventThread shut down 2018-01-17 18:52:10,335 INFO [main-SendThread(post-om2.vodafone.flytxt.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server post-om2.vodafone.flytxt.com/10.88.8.79:2181. Will not attempt to authenticate using SASL (unknown error) ... View more

In Hortonworks how will separate jobs, memory for HBase and yarn?How we will differentiate both operations from an admin point of view. ... View more

Any documents like Hadoop cluster planning mode like pro with the important ecosystems & services ... View more

In master machines ambari server and ambari agent needed with ambari agent on data nodes.But once active namenode fails and standby acts as a active then how ambari server will work in standby.How ambari-server and agent will communicate ... View more

Hmaster:- 2017-10-19 04:19:45,616 ERROR [PriorityRpcServer.handler=8,queue=0,port=16000] master.MasterRpcServices: Region server cbc-slv1.vodafone.flytxt.com,16020,1505158335244 reported a fatal error: 2017-10-19 04:19:46,571 ERROR [PriorityRpcServer.handler=19,queue=1,port=16000] master.MasterRpcServices: Region server cbc-slv4.vodafone.flytxt.com,16020,1505158341501 reported a fatal error: HRegion :- cbc-slv1 4:18:19,755 INFO [B.defaultRpcServer.handler=9,queue=3,port=16020] shortcircuit.ShortCircuitCache: ShortCircuitCache(0x30040ff9): could not load 1218930687_BP-1248144674-10.88.8.135-1467124509456 due to InvalidToken exception. org.apache.hadoop.security.token.SecretManager$InvalidToken: access control error while attempting to set up short-circuit access to /hbase/data/default/SUBSCRIBER/6285afac5334bd9afd3527c7f43f9d30/T/050298e63ea648dd93093e04343a8e75Block token with block_token_identifier (expiryDate=1508362433325, keyId=1595020424, userId=hbase, blockPoolId=BP-1248144674-10.88.8.135-1467124509456, blockId=1218930687, access modes=[READ]) is expired. at org.apache.hadoop.hdfs.BlockReaderFactory.requestFileDescriptors(BlockReaderFactory.java:591)at org.apache.hadoop.hdfs.BlockReaderFactory.requestFileDescriptors(BlockReaderFactory.java:591) at org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:490)at org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:490) at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:782) at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:716) 2017-10-19 04:19:43,803 INFO [Thread-7] regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@420249a9 2017-10-19 04:19:43,803 INFO [Thread-7] regionserver.HRegionServer: STOPPED: Shutdown hook 2017-10-19 04:19:43,803 INFO [regionserver/cbc-slv1.vodafone.flytxt.com/10.88.8.137:16020] regionserver.SplitLogWorker: Sending interrupt to stop the worker thread 2017-10-19 04:19:43,806 INFO [regionserver/cbc-slv1.vodafone.flytxt.com/10.88.8.137:16020] regionserver.HRegionServer: Stopping infoServer 2017-10-19 04:19:43,814 INFO [SplitLogWorker-cbc-slv1:16020] regionserver.SplitLogWorker: SplitLogWorker interrupted. Exiting. 2017-10-19 04:19:43,814 INFO [SplitLogWorker-cbc-slv1:16020] regionserver.SplitLogWorker: SplitLogWorker cbc-slv1.vodafone.flytxt.com,16020,1505158335244 exiting 2017-10-19 04:19:43,819 INFO [regionserver/cbc-slv1.vodafone.flytxt.com/10.88.8.137:16020] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:16030 2017-10-19 04:19:43,920 INFO [regionserver/cbc-slv1.vodafone.flytxt.com/10.88.8.137:16020] regionserver.HeapMemoryManager: Stoping HeapMemoryTuner chore. 2017-10-19 04:19:43,920 INFO [regionserver/cbc-slv1.vodafone.flytxt.com/10.88.8.137:16020] snapshot.RegionServerSnapshotManager: Stopping RegionServerSnapshotManager graceful ... View more