Error CREATEing SolrCore 'Test_shard1_replica1': Unable to create core [Test_shard1_replica1] Caused by: Index locked for write for core Test_shard1_replica1

[#1]

Hello,

We are testing SOLR 5.2.1 , below error pops up when creating a collection with more than 1 shard & more than 1 replica. The process fails after it creates 1 shard as it is unable to remove the write lock that is created under /solr/index on hdfs.

745698 [OverseerThreadFactory-4-thread-1-processing-{node_name=<hostname>:8983_solr}] ERROR org.apache.solr.cloud.OverseerCollectionProcessor [Test ] – Error from shard: http://<hostname>:8983/solr org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://<hostname>:8983/solr: Error CREATEing SolrCore 'Test_shard1_replica1': Unable to create core [Test_shard1_replica1] Caused by: Index locked for write for core Test_shard1_replica1 at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227) at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:218) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:183) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744)

Thanks,

Jagdish

[#2 · Accepted solution]

Take a look at this article https://community.hortonworks.com/articles/15159/securing-solr-collections-with-ranger-kerberos.html especially the following section

--------

Since all Solr data will be stored in the Hadoop Filesystem, it is important to adjust the time Solr will take to shutdown or "kill" the Solr process (whenever you execute "service solr stop/restart"). If this setting is not adjusted, Solr will try to shutdown the Solr process and because it takes a bit more time when using HDFS, Solr will simply kill the process and most of the time lock the Solr Indexes of your collections. If the index of a collection is locked the following exception is shown after the startup routine "org.apache.solr.common.SolrException: Index locked for write"

Increase the sleep time from 5 to 30 seconds in /opt/lucidworks-hdpsearch/solr/bin/solr

sed -i 's/(sleep 5)/(sleep 30)/g'/opt/lucidworks-hdpsearch/solr/bin/solr

--------

You can try the following solution: Make sure nobody is writing to the Solr Index, remove all write.lock files in your HDFS Solr folder (/solr/), Restart Solr.

Also make sure the solr user or the user that is running your Solr instance is able to remove the write.lock file.

[#3]

@Jonas Straub : Thanks Jonas , the above issue was more related to a hard coded value for index lock in solr.in.sh but I am also facing issue as you mentioned above on one of the solr cloud nodes. It kind of makes sense to increase the sleep # for solr graceful shutdown process .