Lily Indexer: ensuring Hbase and Solr record consistency

ginsul — Fri, 16 Sep 2022 10:45:20 GMT

Hi,

There are a few theoretical gaps that I need to be filled. Could you please help me out with that?

I have looked at the Lily documentation provided on https://github.com/NGDATA/hbase-indexer/wiki , but I am not certain regarding a few aspects.

Consider a scenario where HBase table is constantly being added new rows. Two Lily indexers L1 and L2 then asynchronously process the added data.

1. Since I could not find WAL-related properties in Lily Indexer's configuration, I presume the indexer uses HBase WAL. Is that right?
2. How do lily indexers L1 and L2 coordinate in order not to index the same data twice?
3. What happens if one indexer crashes while indexing an HBase row. Do other indexers then reindex the data that the crashed indexer has failed to index?
4. What is the best approach to verify that all HBase records have been indexed and are present in Solr? E.g. an occasional batch job?

Thanks,

Gin

Re: Lily Indexer: ensuring Hbase and Solr record consistency

whosch — Mon, 24 Oct 2016 17:43:42 GMT

Here is a useful related read: http://www.ngdata.com/the-hbase-side-effect-processor-and-hbase-replication-monitoring/

Re: Lily Indexer: ensuring Hbase and Solr record consistency

ginsul — Tue, 25 Oct 2016 08:12:35 GMT

Fantastic! Thanks, whosch.

question Lily Indexer: ensuring Hbase and Solr record consistency in Archives of Support Questions (Read Only)

Lily Indexer: ensuring Hbase and Solr record consistency

Re: Lily Indexer: ensuring Hbase and Solr record consistency

Re: Lily Indexer: ensuring Hbase and Solr record consistency