Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search

cannot get log reader error in hbase, continuosly coming in all region servers, still hbase is running fine....

Labels:
avatar
author-rank Crif
Contributor

Created ‎08-03-2018 05:40 AM

2018-08-03 10:49:47,087 INFO [RS_LOG_REPLAY_OPS-CHMCISPRBDDN01:16020-1] coordination.ZkSplitLogWorkerCoordination: successfully transitioned task /hbase-unsecure/splitWAL/WALs%2Fchmcisprbddn08.chm.intra%2C16020%2C1503542190128-splitting%2Fchmcisprbddn08.chm.intra%252C16020%252C1503542190128.default.1503542196998 to final state ERR chmcisprbddn01.chm.intra,16020,1533218035807

2018-08-03 10:49:47,088 INFO  [RS_LOG_REPLAY_OPS-CHMCISPRBDDN01:16020-1] handler.WALSplitterHandler: worker chmcisprbddn01.chm.intra,16020,1533218035807 done with task org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$ZkSplitTaskDetails@11ba74a2 in 39ms
2018-08-03 10:49:47,663 INFO  [SplitLogWorker-CHMCISPRBDDN01:16020] coordination.ZkSplitLogWorkerCoordination: worker chmcisprbddn01.chm.intra,16020,1533218035807 acquired task /hbase-unsecure/splitWAL/WALs%2Fchmcisprbddn08.chm.intra%2C16020%2C1503542010122-splitting%2Fchmcisprbddn08.chm.intra%252C16020%252C1503542010122.default.1503542017092
2018-08-03 10:49:47,692 INFO  [RS_LOG_REPLAY_OPS-CHMCISPRBDDN01:16020-0] wal.WALSplitter: Splitting wal: hdfs://prodcluster/apps/hbase/data/WALs/chmcisprbddn08.chm.intra,16020,1503542010122-splitting/chmcisprbddn08.chm.intra%2C16020%2C1503542010122.default.1503542017092, length=153
2018-08-03 10:49:47,692 INFO  [RS_LOG_REPLAY_OPS-CHMCISPRBDDN01:16020-0] wal.WALSplitter: DistributedLogReplay = false
2018-08-03 10:49:47,693 INFO  [RS_LOG_REPLAY_OPS-CHMCISPRBDDN01:16020-0] util.FSHDFSUtils: Recovering lease on dfs file hdfs://prodcluster/apps/hbase/data/WALs/chmcisprbddn08.chm.intra,16020,1503542010122-splitting/chmcisprbddn08.chm.intra%2C16020%2C1503542010122.default.1503542017092
2018-08-03 10:49:47,693 INFO  [RS_LOG_REPLAY_OPS-CHMCISPRBDDN01:16020-0] util.FSHDFSUtils: recoverLease=true, attempt=0 on file=hdfs://prodcluster/apps/hbase/data/WALs/chmcisprbddn08.chm.intra,16020,1503542010122-splitting/chmcisprbddn08.chm.intra%2C16020%2C1503542010122.default.1503542017092 after 0ms
2018-08-03 10:49:47,695 INFO  [RS_LOG_REPLAY_OPS-CHMCISPRBDDN01:16020-0] wal.WALSplitter: Processed 0 edits across 0 regions; edits skipped=0; log file=hdfs://prodcluster/apps/hbase/data/WALs/chmcisprbddn08.chm.intra,16020,1503542010122-splitting/chmcisprbddn08.chm.intra%2C16020%2C1503542010122.default.1503542017092, length=153, corrupted=false, progress failed=false
2018-08-03 10:49:47,695 WARN  [RS_LOG_REPLAY_OPS-CHMCISPRBDDN01:16020-0] regionserver.SplitLogWorker: log splitting of WALs/chmcisprbddn08.chm.intra,16020,1503542010122-splitting/chmcisprbddn08.chm.intra%2C16020%2C1503542010122.default.1503542017092 failed, returning error
java.io.IOException: Cannot get log reader
	at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:355)
	at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:267)
	at org.apache.hadoop.hbase.wal.WALSplitter.getReader(WALSplitter.java:839)
	at org.apache.hadoop.hbase.wal.WALSplitter.getReader(WALSplitter.java:763)
	at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:297)
	at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:235)
	at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:104)
	at org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:72)
	at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.UnsupportedOperationException: Unable to find org.apache.hadoop.hbase.regionserver.wal.WALCellCodec,org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec
	at org.apache.hadoop.hbase.util.ReflectionUtils.instantiateWithCustomCtor(ReflectionUtils.java:36)
	at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec.create(WALCellCodec.java:103)
	at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.getCodec(ProtobufLogReader.java:297)
	at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initAfterCompression(ProtobufLogReader.java:307)
	at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:82)
	at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:164)
	at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:303)
	... 11 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.regionserver.wal.WALCellCodec,org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at org.apache.hadoop.hbase.util.ReflectionUtils.instantiateWithCustomCtor(ReflectionUtils.java:32)
	... 17 more
3,319 Views
0 Kudos
5 REPLIES 5

avatar
author-rank PARTOMIA author-rank
Expert Contributor

Created ‎08-03-2018 06:33 AM

@Prashant Verma

Seems the Split Task is failing.

2018-08-0310:49:47,695 WARN  [RS_LOG_REPLAY_OPS-CHMCISPRBDDN01:16020-0] 
regionserver.SplitLogWorker: log splitting of 
WALs/chmcisprbddn08.chm.intra,16020,1503542010122-splitting/chmcisprbddn08.chm.intra%2C16020%2C1503542010122.default.1503542017092
 failed, returning error java.io.IOException:Cannotget log reader
Also there seems to be issue with the Class. 
Caused by: java.lang.UnsupportedOperationException:Unable to find org.apache.hadoop.hbase.regionserver.wal.WALCellCodec,org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.regionserver.wal.WALCellCodec,org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec

RegionServer seems to be unable to find the class.

What is the HDP, Ambari and HBase version? Share the rpm -qa if that is okay.

Also, can you check the content of the WAL :

# hdfs dfs -cat /apps/hbase/data/WALs/chmcisprbddn08.chm.intra,16020,1503542010122-splitting/chmcisprbddn08.chm.intra%2C16020%2C1503542010122.default.1503542017092
3,008 Views
0 Kudos

avatar
author-rank PARTOMIA author-rank
Expert Contributor

Created ‎08-03-2018 06:34 AM

What do you see in the HBase Master logs during the time of these logs in RegionServer?

3,008 Views
0 Kudos

avatar
author-rank Crif
Contributor

Created ‎08-03-2018 08:37 AM

Hi Ravi,

Thanks for writing, below are the required details -

Ambari version - HDP-2.6.1.0

Hbase Version - 1.1.2

Pheonix Version - 4.7.0

Regarding the classes I have checked in Phoenix client jar, jar name - phoenix-4.7.0.2.6.1.0-129-client.jar

Both the classes are present in the above jar, and this phoenix jar is kept on hbase lib folder.

Thanks,

Prashant Verma

3,008 Views
0 Kudos

avatar
author-rank Crif
Contributor

Created ‎08-03-2018 08:41 AM

Details in the specified hdfs directory is below-

PWAL‹  "ProtobufLogWriter*rorg.apache.hadoop.hbase.regionserver.wal.WALCellCodec,org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec LAWP

3,008 Views
0 Kudos

avatar
author-rank Crif
Contributor

Created ‎08-03-2018 08:48 AM

Dump of Hbase Master..

2018-08-03 12:59:58,968 ERROR [MASTER_SERVER_OPERATIONS-CHMCISPRBDSN01:16000-0] executor.EventHandler: Caught throwable while processing event M_SERVER_SHUTDOWN
java.io.IOException: failed log splitting for chmcisprbddn01.chm.intra,16020,1503542011010, will retry
	at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:378)
	at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:222)
	at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: error or interrupted while splitting logs in [hdfs://prodcluster/apps/hbase/data/WALs/chmcisprbddn01.chm.intra,16020,1503542011010-splitting] Task = installed = 1 done = 0 error = 1
	at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:290)
	at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:429)
	at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:402)
	at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:319)
	at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:215)
	... 4 more
2018-08-03 12:59:58,969 WARN  [MASTER_SERVER_OPERATIONS-CHMCISPRBDSN01:16000-0] hbase.HBaseConfiguration: Config option "hbase.regionserver.lease.period" is deprecated. Instead, use "hbase.client.scanner.timeout.period"
2018-08-03 12:59:58,971 INFO  [MASTER_SERVER_OPERATIONS-CHMCISPRBDSN01:16000-2] handler.ServerShutdownHandler: Splitting logs for chmcisprbddn02.chm.intra,16020,1503542193852 before assignment; region count=0
2018-08-03 12:59:58,973 INFO  [MASTER_SERVER_OPERATIONS-CHMCISPRBDSN01:16000-2] master.SplitLogManager: dead splitlog workers [chmcisprbddn02.chm.intra,16020,1503542193852]
2018-08-03 12:59:58,973 INFO  [MASTER_SERVER_OPERATIONS-CHMCISPRBDSN01:16000-2] master.SplitLogManager: started splitting 1 logs in [hdfs://prodcluster/apps/hbase/data/WALs/chmcisprbddn02.chm.intra,16020,1503542193852-splitting] for [chmcisprbddn02.chm.intra,16020,1503542193852]
2018-08-03 12:59:58,977 INFO  [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase-unsecure/splitWAL/WALs%2Fchmcisprbddn02.chm.intra%2C16020%2C1503542193852-splitting%2Fchmcisprbddn02.chm.intra%252C16020%252C1503542193852.default.1503542200899 acquired by chmcisprbddn07.chm.intra,16020,1533218036449
2018-08-03 12:59:59,005 INFO  [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase-unsecure/splitWAL/WALs%2Fchmcisprbddn02.chm.intra%2C16020%2C1503542193852-splitting%2Fchmcisprbddn02.chm.intra%252C16020%252C1503542193852.default.1503542200899 entered state: ERR chmcisprbddn07.chm.intra,16020,1533218036449
2018-08-03 12:59:59,006 WARN  [main-EventThread] coordination.SplitLogManagerCoordination: Error splitting /hbase-unsecure/splitWAL/WALs%2Fchmcisprbddn02.chm.intra%2C16020%2C1503542193852-splitting%2Fchmcisprbddn02.chm.intra%252C16020%252C1503542193852.default.1503542200899
2018-08-03 12:59:59,006 WARN  [MASTER_SERVER_OPERATIONS-CHMCISPRBDSN01:16000-2] master.SplitLogManager: error while splitting logs in [hdfs://prodcluster/apps/hbase/data/WALs/chmcisprbddn02.chm.intra,16020,1503542193852-splitting] installed = 1 but only 0 done
2018-08-03 12:59:59,006 ERROR [MASTER_SERVER_OPERATIONS-CHMCISPRBDSN01:16000-2] executor.EventHandler: Caught throwable while processing event M_SERVER_SHUTDOWN
java.io.IOException: failed log splitting for chmcisprbddn02.chm.intra,16020,1503542193852, will retry
	at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:378)
	at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:222)
	at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: error or interrupted while splitting logs in [hdfs://prodcluster/apps/hbase/data/WALs/chmcisprbddn02.chm.intra,16020,1503542193852-splitting] Task = installed = 1 done = 0 error = 1
3,008 Views
0 Kudos
Announcements
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
What's New @ Cloudera
New Action menu item in the Cloudera Operational Database UI
What's New @ Cloudera
Get the list of supported instance types in the Cloudera Ope...
View More Announcements