Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

RegionServer throws "not a SequenceFile", oldWALs growing, Hbase-indexer not indexing

avatar
Rising Star

Hi,

 

At some point "test_indexer" stopped indexing new entries from the hbase.

 

The table has the replication scope set to 1. From the logs, the "test_indexer" and the solr server appear to be working just fine.

 

Looking at the RegionServer logs I see the following warning being repeated all day:

 

WARN	org.apache.hadoop.hbase.replication.regionserver.ReplicationSource	
Indexer_test_indexer Got: 
java.io.EOFException: hdfs://tets-node:8020/hbase/oldWALstest-node%2C60020%2C1480189429512.null0.1480189450431 not a SequenceFile
	at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1919)
	at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1878)
	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1827)
	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1841)
	at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:70)
	at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.reset(SequenceFileLogReader.java:168)
	at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.initReader(SequenceFileLogReader.java:177)
	at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:66)
	at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:302)
	at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:267)
	at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:255)
	at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:406)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationWALReaderManager.openReader(ReplicationWALReaderManager.java:70)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceWorkerThread.openReader(ReplicationSource.java:745)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceWorkerThread.run(ReplicationSource.java:541)

 

Folder  /hbase/oldWALs is taking up 8.7Gb and slowly but noticeably increasing.

 

So my guess is the indexer is not marking the WAL entries as replicated and they keep accumulating in the oldWALs folder. How do I determine the cause of the issue?

Could you also explain to me what this exception means? The set-up worked just fine a day ago.

 

Thanks,

Gin

1 ACCEPTED SOLUTION

avatar
Rising Star
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
4 REPLIES 4

avatar
Rising Star
More info. I see in the oldWALs folder the oldest wal file size is 0. Can it be that this is a corrupt file and the indexer cannot get past it to process the rest of WAL records?

avatar
Rising Star
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Super Collaborator

Thx, that was interesting to know !

avatar
Explorer

I am having the same error but I didn't understand the solution, can you please explain it. Thank you.