Member since
10-13-2016
31
Posts
4
Kudos Received
5
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2645 | 02-05-2018 03:12 AM | |
4787 | 12-09-2016 08:36 AM | |
1770 | 11-25-2016 07:14 AM | |
3219 | 11-09-2016 09:59 AM | |
3829 | 10-21-2016 10:36 AM |
11-28-2016
07:13 AM
Hi, At some point "test_indexer" stopped indexing new entries from the hbase. The table has the replication scope set to 1. From the logs, the "test_indexer" and the solr server appear to be working just fine. Looking at the RegionServer logs I see the following warning being repeated all day: WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource
Indexer_test_indexer Got:
java.io.EOFException: hdfs://tets-node:8020/hbase/oldWALstest-node%2C60020%2C1480189429512.null0.1480189450431 not a SequenceFile
at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1919)
at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1878)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1827)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1841)
at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:70)
at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.reset(SequenceFileLogReader.java:168)
at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.initReader(SequenceFileLogReader.java:177)
at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:66)
at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:302)
at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:267)
at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:255)
at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:406)
at org.apache.hadoop.hbase.replication.regionserver.ReplicationWALReaderManager.openReader(ReplicationWALReaderManager.java:70)
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceWorkerThread.openReader(ReplicationSource.java:745)
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceWorkerThread.run(ReplicationSource.java:541) Folder /hbase/oldWALs is taking up 8.7Gb and slowly but noticeably increasing. So my guess is the indexer is not marking the WAL entries as replicated and they keep accumulating in the oldWALs folder. How do I determine the cause of the issue? Could you also explain to me what this exception means? The set-up worked just fine a day ago. Thanks, Gin
... View more
Labels:
- Labels:
-
Apache HBase
-
Apache Solr
11-25-2016
07:14 AM
Apologies, a premature post. Just realised that Solr Core != SolrCloud Core and found out about Collection Aliasing
... View more
11-25-2016
07:02 AM
Hi, In case a collection has multiple cores, how does the hbase-indexer (or the corresponding batch indexer job) know to which core to write? Thanks, Gin
... View more
Labels:
- Labels:
-
Apache HBase
-
Apache Solr
11-18-2016
03:00 AM
We are about to perform planned upgrade of the cluster to 5.9 thus the problem will be solved. It is great to know that the issue will become irrelevant. Thanks, Harsh. A fantastic clarification!
... View more
11-18-2016
01:39 AM
Hi, We have observed an unsafe behaviour from CDH 5.3.2/5.9 libraries. I am sharing the observations with the hope that the issues are/will be fixed in later releases. We have a CDH 5.3.2 cluster that has been running fine for months. Yesterday, out of the blue, Region Servers started dropping like flies. There were no error messages in the logs, just an abrupt startup entry with all the classpath info etc. It took me a good hour to narrow down the source of the problem. Apparently, one of the colleagues tried to fetch some "fresh data" from the cluster using newer CDH 5.9 client libraries. That's it! Whenever he connected to the CDH 5.3.2 cluster and attempted to query a table, all cluster's region servers crashed without an error message. It is really worrying that an accidental connection using newer libraries (5.9) can bring the whole cluster (5.3.2) offline. So I wonder: do hadoop/hbase architecture have some kind of safety mechanisms in terms of library incompatibility? Maybe this safety mechanism has not been implemented? Or maybe it is non-existent whatsoever? Thanks, Gin
... View more
Labels:
- Labels:
-
Apache HBase
-
HDFS
11-09-2016
09:59 AM
1 Kudo
Found the solution! https://groups.google.com/forum/#!searchin/hbase-indexer-user/rowkey%7Csort:relevance/hbase-indexer-user/eitg7QquxRI/MfVA6SXInM8J Post by Wolfgang: You can fetch it (edited: the rowkey) by pasting this code snippet into a “java” morphline command:
org.apache.hadoop.hbase.client.Result result = (org.apache.hadoop.hbase.client.Result) record.getFirstValue("_attachment_body");
byte[] rowKey = result.getRow();
record.put("myRowKey", rowKey);
Possible gotcha: Make sure that this java command appears *before* the extractHBaseCells morphline command in the morphline config file.
http://kitesdk.org/docs/current/morphlines/morphlines-reference-guide.html#java Gin
... View more
11-09-2016
06:47 AM
I have looked at the source code. The most maintenance-friendly solution would be to create a custom mapper class and do some "hacks" there. However, the provided mapper classes are final (thus no decent reuse) and I haven't found a way to configure hbase-indexer classpath via Cloudera manager.
... View more
11-09-2016
05:49 AM
Hi,
What is the best way to add a library to lily indexer's classpath?
Is there an option to do this via the Cloudera manager?
I am looking for an CDH upgrade-friendly approach.
Thanks,
Gin
... View more
Labels:
- Labels:
-
Apache HBase
-
Apache Solr
-
Cloudera Manager
11-08-2016
07:27 AM
Hi, I am using extractHBaseCells morphlines command to index hbase data into solr. How do I get HBase rowkey? I would like to construct a URI-based ID from the given rowkey as follows: setValues: {
id : "http://hbase_ids.com/@{rowkey}"
} Thanks, Gin
... View more
Labels:
- Labels:
-
Apache Solr
10-25-2016
01:12 AM
Fantastic! Thanks, whosch.
... View more
- « Previous
-
- 1
- 2
- Next »