Hi,   At some point "test_indexer" stopped indexing new entries from the hbase.   The table has the replication scope set to 1. From the logs, the "test_indexer" and the solr server appear to be working just fine.   Looking at the RegionServer logs I see the following warning being repeated all day:   WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource Indexer_test_indexer Got: java.io.EOFException: hdfs://tets-node:8020/hbase/oldWALstest-node%2C60020%2C1480189429512.null0.1480189450431 not a SequenceFile at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1919) at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1878) at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1827) at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1841) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:70) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.reset(SequenceFileLogReader.java:168) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.initReader(SequenceFileLogReader.java:177) at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:66) at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:302) at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:267) at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:255) at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:406) at org.apache.hadoop.hbase.replication.regionserver.ReplicationWALReaderManager.openReader(ReplicationWALReaderManager.java:70) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceWorkerThread.openReader(ReplicationSource.java:745) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceWorkerThread.run(ReplicationSource.java:541)   Folder  /hbase/oldWALs is taking up 8.7Gb and slowly but noticeably increasing.   So my guess is the indexer is not marking the WAL entries as replicated and they keep accumulating in the oldWALs folder. How do I determine the cause of the issue? Could you also explain to me what this exception means? The set-up worked just fine a day ago.   Thanks, Gin ... View more

Hi,   In case a collection has multiple cores, how does the hbase-indexer (or the corresponding batch indexer job) know to which core to write?     Thanks, Gin ... View more

Hi,   We have observed an unsafe behaviour from CDH 5.3.2/5.9 libraries. I am sharing the observations with the hope that the issues are/will be fixed in later releases.   We have a CDH 5.3.2 cluster that has been running fine for months. Yesterday, out of the blue, Region Servers started dropping like flies. There were no error messages in the logs, just an abrupt startup entry with all the classpath info etc. It took me a good hour to narrow down the source of the problem.   Apparently, one of the colleagues tried to fetch some "fresh data" from the cluster using newer CDH 5.9 client libraries. That's it! Whenever he connected to the CDH 5.3.2 cluster and attempted to query a table, all cluster's region servers crashed without an error message. It is really worrying that an accidental connection using newer libraries (5.9) can bring the whole cluster (5.3.2) offline.  So I wonder: do hadoop/hbase architecture have some kind of safety mechanisms in terms of library incompatibility? Maybe this safety mechanism has not been implemented? Or maybe it is non-existent whatsoever?   Thanks, Gin ... View more

Hi,   What is the best way to add a library to lily indexer's classpath? Is there an option to do this via the Cloudera manager? I am looking for an CDH upgrade-friendly approach.   Thanks, Gin ... View more

Hi,   I am using extractHBaseCells morphlines command to index hbase data into solr. How do I get HBase rowkey?   I would like to construct a URI-based ID from the given rowkey as follows:   setValues: { id : "http://hbase_ids.com/@{rowkey}" }   Thanks,  Gin  ... View more