Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

expection in ambari infra solr instance

Highlighted

expection in ambari infra solr instance

Below exception is being thrown in the ambari infra solr logs. Any idea how to work around this problem ?

2016-10-19 15:41:53,003 [qtp100555887-308] ERROR [c:hadoop_logs s:shard0 r:core_node1 x:hadoop_logs_shard0_replica1] org.apache.solr.common.SolrException (SolrException.java:148) - org.apache.solr.common.SolrException: no servers hosting shard: shard1

	at org.apache.solr.handler.component.HttpShardHandler.prepDistributed(HttpShardHandler.java:451)

	at org.apache.solr.handler.component.SearchHandler.getAndPrepShardHandler(SearchHandler.java:215)

	at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:241)

	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)

	at org.apache.solr.core.SolrCore.execute(SolrCore.java:2102)

	at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)

	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:460)

	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)

	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)

	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)

	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)

	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)

	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)

	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)

	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)

	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)

	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)

	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)

	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)

	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

	at org.eclipse.jetty.server.Server.handle(Server.java:499)

	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)

	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)

	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)

	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)

	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)

	at java.lang.Thread.run(Thread.java:745)

4 REPLIES 4
Highlighted

Re: expection in ambari infra solr instance

@Santhosh B Gowda

I guess you are hitting https://issues.apache.org/jira/browse/SOLR-6335. Please go through this link for more details. Also, note that this issue is still in open state.

Highlighted

Re: expection in ambari infra solr instance

It is also good to know, how solr reached to this state? Did you run any queries? Are your solr nodes in active/recovery state?

Re: expection in ambari infra solr instance

Can you restart the solr instance that is hosting the shard? After the restart, does the shard recover or stay dead?

Did the shard just die or was the solr instance killed, ran out of resources, etc.?

Highlighted

Re: expection in ambari infra solr instance

New Contributor

@Santhosh B Gowda

We had a similar issue where one of the cores wasn't allocated to a shard.
the root cause that made all this mess was that all servers had a very low configuration of open file descriptor limit (ulimit -n was 1024)
I found the following helped after fixing the ulimit issue:

on the "Cloud" view you can see the graph of correlation between collections, shards, cores, and solr nodes:

we were missing one of the links between shard to solr instance.

on the "Core Admin" view you can add the missing core, I kept the same standard as with all other cores:

nameaudit_logs_shard1_replica1
instanceDir /opt/ambari_infra_solr/data/audit_logs_shard1_replica1
dataDir/opt/ambari_infra_solr/data/audit_logs_shard1_replica1/data/
configsolrconfig.xml
schemaschema.xml
collectionaudit_logs
shardshard1

hope this helps!

Don't have an account?
Coming from Hortonworks? Activate your account here