Key-Value Store Indexer for NRT with number of morphline configs

[#1]

Dear Team,

Presently, I'm using Solr batch indexing on hbase tables to create solr documents. I have to enable NRT feature on 3 hbase tables. How to handle multiple morphlines for "Key-Value Store Indexer" service? I know, Morphlines file holds an array of 'id's, but how do we handle the SOLR_LOCATOR section where we have to specify the collection name? can we give a comma seperated list of collection names?

I'm using CDH 5.3.1 and CM 5.3.0.

Presently I have 3 morphline files say morphlineX.conf, morphlineY.conf and morphlineZ.conf

Thanks,

YBSNR

[#2 · Accepted solution]

The SOLR_LOCATOR is a variable that works via simple text substitution (ala unix shell scripts). You can define as many variables as you like within the same morphline config file. For example along these lines:

SOLR_LOCATOR_1 : { collection : collection1, zkHost : ${ZK_HOST} }
SOLR_LOCATOR_2 : { collection : collection2, zkHost : ${ZK_HOST} }

morphlines : [
{
id : morphline1
...
{ loadSolr { solrLocator : ${SOLR_LOCATOR_1} } }
}

{
id : morphline2
...
{ loadSolr { solrLocator : ${SOLR_LOCATOR_2} } }
}
]

Wolfgang

[#3]

Thanks for the quick response.

After I enable Key-Value Store Indexer service and edit service wide morphline (textbox) config as you suggested above, should I maintain the morphlines.conf file in /etc/hbase-solr/conf directory where I have created for the batch indexing purpose?

In other words, if I create morphlinesX.conf, morphlinesY.conf and morphlinesZ.conf, should I update the service wide morphline configuration on KV Store Indexer also?

My observation is, when I have those 3 morphlines files in /etc/hbase-solr/conf directory, and enable KV Store indexer service with default configuration, the corresponding 3 collections are active and started generating solr index documents.

Further, after updating the KV Store indexer service --> configuration --> servicewide --> morphlines --> morphlines file (textbox), and deploy the client configuration, where it will be updated? Where could I verify the deployed configuraiton from CM admin console?

Please clarify.