Hey Pradheep ShanI am also facing the similar issue, could you find out any ways to resolve it. Thanks, Prashant Verma ... View more

Regions in Transition Region State RIT time (ms) e1b79d7f51636c99dabc71ecec3e706c hmcnsprod:HM_CNS_STD_IOT_INDEX,498633670,1542558189093.e1b79d7f51636c99dabc71ecec3e706c. state=PENDING_CLOSE, ts=Wed Nov 21 07:33:04 IST 2018 (716612s ago), server=chmcisprbddn10.chm.intra,16020,1542735119121 Total number of Regions in Transition for more than 60000 milliseconds Total number of Regions in Transition ... View more

hbase(main):001:0> get 'hmcnsprod:HM_CNS_STD_IOT_INDEX','498633670' COLUMN CELL ERROR: org.apache.hadoop.hbase.NotServingRegionException: Region hmcnsprod:HM_CNS_STD_IOT_INDEX,498633670,1542558189093.e1b79d7f51636c99dabc71ecec3e706c. is not online on chmcisprbddn07.chm.intra,16020,1543476489995 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3062) at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1007) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:1984) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32381) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2141) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167) Regions in Transition Region State RIT time (ms) e1b79d7f51636c99dabc71ecec3e706c hmcnsprod:HM_CNS_STD_IOT_INDEX,498633670,1542558189093.e1b79d7f51636c99dabc71ecec3e706c. state=PENDING_CLOSE, ts=Wed Nov 21 07:33:04 IST 2018 (716612s ago), server=chmcisprbddn10.chm.intra,16020,1542735119121 Total number of Regions in Transition for more than 60000 milliseconds Total number of Regions in Transition ... View more

Need to add hbase-site.xml file in hadoop bin also, that solved my problem...thanks... ... View more

@Felix thanks for replying, yes the Phoenix client jar is correctly installed and also added in spark submit command. Thanks, Prashant Verma ... View more

Dump of Hbase Master.. 2018-08-03 12:59:58,968 ERROR [MASTER_SERVER_OPERATIONS-CHMCISPRBDSN01:16000-0] executor.EventHandler: Caught throwable while processing event M_SERVER_SHUTDOWN java.io.IOException: failed log splitting for chmcisprbddn01.chm.intra,16020,1503542011010, will retry at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:378) at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:222) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: error or interrupted while splitting logs in [hdfs://prodcluster/apps/hbase/data/WALs/chmcisprbddn01.chm.intra,16020,1503542011010-splitting] Task = installed = 1 done = 0 error = 1 at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:290) at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:429) at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:402) at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:319) at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:215) ... 4 more 2018-08-03 12:59:58,969 WARN [MASTER_SERVER_OPERATIONS-CHMCISPRBDSN01:16000-0] hbase.HBaseConfiguration: Config option "hbase.regionserver.lease.period" is deprecated. Instead, use "hbase.client.scanner.timeout.period" 2018-08-03 12:59:58,971 INFO [MASTER_SERVER_OPERATIONS-CHMCISPRBDSN01:16000-2] handler.ServerShutdownHandler: Splitting logs for chmcisprbddn02.chm.intra,16020,1503542193852 before assignment; region count=0 2018-08-03 12:59:58,973 INFO [MASTER_SERVER_OPERATIONS-CHMCISPRBDSN01:16000-2] master.SplitLogManager: dead splitlog workers [chmcisprbddn02.chm.intra,16020,1503542193852] 2018-08-03 12:59:58,973 INFO [MASTER_SERVER_OPERATIONS-CHMCISPRBDSN01:16000-2] master.SplitLogManager: started splitting 1 logs in [hdfs://prodcluster/apps/hbase/data/WALs/chmcisprbddn02.chm.intra,16020,1503542193852-splitting] for [chmcisprbddn02.chm.intra,16020,1503542193852] 2018-08-03 12:59:58,977 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase-unsecure/splitWAL/WALs%2Fchmcisprbddn02.chm.intra%2C16020%2C1503542193852-splitting%2Fchmcisprbddn02.chm.intra%252C16020%252C1503542193852.default.1503542200899 acquired by chmcisprbddn07.chm.intra,16020,1533218036449 2018-08-03 12:59:59,005 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase-unsecure/splitWAL/WALs%2Fchmcisprbddn02.chm.intra%2C16020%2C1503542193852-splitting%2Fchmcisprbddn02.chm.intra%252C16020%252C1503542193852.default.1503542200899 entered state: ERR chmcisprbddn07.chm.intra,16020,1533218036449 2018-08-03 12:59:59,006 WARN [main-EventThread] coordination.SplitLogManagerCoordination: Error splitting /hbase-unsecure/splitWAL/WALs%2Fchmcisprbddn02.chm.intra%2C16020%2C1503542193852-splitting%2Fchmcisprbddn02.chm.intra%252C16020%252C1503542193852.default.1503542200899 2018-08-03 12:59:59,006 WARN [MASTER_SERVER_OPERATIONS-CHMCISPRBDSN01:16000-2] master.SplitLogManager: error while splitting logs in [hdfs://prodcluster/apps/hbase/data/WALs/chmcisprbddn02.chm.intra,16020,1503542193852-splitting] installed = 1 but only 0 done 2018-08-03 12:59:59,006 ERROR [MASTER_SERVER_OPERATIONS-CHMCISPRBDSN01:16000-2] executor.EventHandler: Caught throwable while processing event M_SERVER_SHUTDOWN java.io.IOException: failed log splitting for chmcisprbddn02.chm.intra,16020,1503542193852, will retry at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:378) at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:222) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: error or interrupted while splitting logs in [hdfs://prodcluster/apps/hbase/data/WALs/chmcisprbddn02.chm.intra,16020,1503542193852-splitting] Task = installed = 1 done = 0 error = 1 ... View more

Details in the specified hdfs directory is below- PWAL‹
  "ProtobufLogWriter*rorg.apache.hadoop.hbase.regionserver.wal.WALCellCodec,org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec LAWP ... View more

Hi Ravi, Thanks for writing, below are the required details - Ambari version - HDP-2.6.1.0 Hbase Version - 1.1.2 Pheonix Version - 4.7.0 Regarding the classes I have checked in Phoenix client jar, jar name - phoenix-4.7.0.2.6.1.0-129-client.jar Both the classes are present in the above jar, and this phoenix jar is kept on hbase lib folder. Thanks, Prashant Verma ... View more