Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Setting up high availability Solr cluster on HDP cluster 2.5.6

Setting up high availability Solr cluster on HDP cluster 2.5.6

New Contributor

I have setup a solr cluster using the steps provided in the links -

1. https://community.hortonworks.com/articles/30482/how-to-deploy-apache-solr-as-solrcloud-on-hdfs-in.h...

2. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_solr-search-installation/content/ch_hdp-...


I have setup a two nodes lucidworks solr cluster and using zookeeper which comes with HDP. I am configuring solr to store indexes on HDFS.

There are two problems i am facing -

1. I am providing a name node to store indexes on HDFS in the configset
using(as mentioned in the link)
<strname="solr.hdfs.home">hdfs://<namenode1:port>/user/solr</str>
But when my primary namenode goes down and switch over takes place.The solr starts giving error as it doesnt switch over by itself to the new active name node.

2. When ever I ingest data into solr using solrJ, It doesn't show up in solr. But, when i restart the solr it shows the data into solr. Answer : I was able to resolve this by changing the <openSearcher>false</openSearcher> setting to true in hard commit settings.


Thanks,

9 REPLIES 9
Highlighted

Re: Setting up high availability Solr cluster on HDP cluster 2.5.6

Expert Contributor
@Jetinder Singh

when you are using NameNode (NN) HA (High Availability) is there any reason that you are giving the individual hostname of the NN instead of specifying the NameSpace which can be found from fs.defaultFS from core-site.xml

i.e. you can use the <strname="solr.hdfs.home"> hdfs://<NAMESERVICE>:8020/path/to/solr/data </str>instead of <strname="solr.hdfs.home">hdfs://<namenode1:port>/user/solr</str> 
Highlighted

Re: Setting up high availability Solr cluster on HDP cluster 2.5.6

New Contributor

Thanks for the reply @Venkata Sudheer Kumar M
I tried putting the nameservice instead of the individual server, It doesnt work either. I get an error when i try to create a collection
using the command

/opt/lucidworks-hdpsearch/solr/bin/solr create -c tests -d /opt/lucidworks-hdpsearch/solr/server/solr/configsets/data_driven_schema_configs_hdfs -n testconfig -s 2 -rf 2

The error I get is.


{ "responseHeader":{ "status":0, "QTime":7087}, "failure":
: Error CREATEing SolrCore 'tests_shard1_replica2': Unable to create core [tests_shard1_replica2] Caused by: mynameservice"}}

So i guess the config expects only a host name as the documentation also says Hostname: full name of your name node on the cluster. Eg. Namenode01.company.com

Thanks,

Highlighted

Re: Setting up high availability Solr cluster on HDP cluster 2.5.6

Expert Contributor
@Jetinder Singh

I have tested the Nameservice usage with HDP2.3.4 and HDP 2.5.3 and it works with Nameservice for sure. and from the error it is giving the Error CREATEing SolrCore 'tests_shard1_replica2': Unable to create core [tests_shard1_replica2] Caused by: mynameservice"}}

it looks like it is not able to place the 2nd replica.

Can you please check that your SOLR is running in distributed mode which you can find form solr UI by selecting the cloud link and also from zookeeper cli using :

/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server <ZK_HOST>:2181 ls /solr/live_nodes

this will help you identify that both of your solr nodes are running the cluster not standalone.

Thanks

Venkat

Highlighted

Re: Setting up high availability Solr cluster on HDP cluster 2.5.6

New Contributor

@Venkata Sudheer Kumar M
Well, I see both my nodes listed in /live_nodes in admin console.

Sorry my bad for not pasting the complete logs earlier. I am getting the same error for all replicas.

Error CREATEing SolrCore 'tests_shard2_replica1': Unable to create core [tests_shard2_replica1] Caused by: mynameservice

Error CREATEing SolrCore 'tests_shard2_replica2': Unable to create core [tests_shard2_replica2] Caused by: mynameservice

Error CREATEing SolrCore 'tests_shard1_replica1': Unable to create core [tests_shard1_replica1] Caused by: mynameservice

Error CREATEing SolrCore 'tests_shard1_replica2': Unable to create core [tests_shard1_replica2] Caused by: mynameservice

for the same reason

Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: mynameserviceAnd it works when i change the service "mynameservice" to the active name node host name.

Thanks,

Highlighted

Re: Setting up high availability Solr cluster on HDP cluster 2.5.6

Expert Contributor
@Jetinder Singh

can you please enable the SOLR debug logging from /opt/lucidworks-hdpsearch/solr/server/resources/log4j.properties (default hdpsearch installtion path) by changing:

log4j.rootLogger=INFO, file, CONSOLE

to

log4j.rootLogger=DEBUG, file, CONSOLE

and give the logs and also give the solrconfig.xml file you are using for the service

Thanks

Highlighted

Re: Setting up high availability Solr cluster on HDP cluster 2.5.6

New Contributor
Highlighted

Re: Setting up high availability Solr cluster on HDP cluster 2.5.6

Expert Contributor
@Jetinder Singh

I have tested this functionality and works well with HDFS HA NameSpace in SOLR.

From the solrconfig.xml file you have given i have noticed that the below lines:

<updateHandler class="solr.DirectUpdateHandler2">

</updateHandler>

are not commented which affect the indexes with HdfsDirectoryFactor

please find the attachment solrconfig.xml which i have used.

Thanks

Venkat

Highlighted

Re: Setting up high availability Solr cluster on HDP cluster 2.5.6

Expert Contributor
@Jetinder Singh

It looks like this is an issue and SOLR community is already aware of this https://issues.apache.org/jira/browse/SOLR-10215 but this is referring to SOLR 6.4.1 and there is a community article for this as well: https://community.hortonworks.com/questions/85132/solr-v641-unable-to-create-collection-when-using-h... but the lucidworks-hdpsearch solr is 5.2.1 which is also having similar issue.

Thanks

Venkat

Highlighted

Re: Setting up high availability Solr cluster on HDP cluster 2.5.6

New Contributor

Thanks @Venkata Sudheer Kumar M
Was able to resolve this by upgrading solr to 6.5.1.

Don't have an account?
Coming from Hortonworks? Activate your account here