Specified config does not exist in ZooKeeper (Log Search is broken)

[#1]

Fresh cluster (2.6.4) was working, restarted zookeeper and now seeing multiple errors. How can I fix this and why doesn't Solr just send new data to zookeeper?

solr-problem.png

One Log Example:

null:org.apache.solr.common.SolrException: SolrCore 'hadoop_logs_shard0_replica1' is not available due to init failure: Specified config does not exist in ZooKeeper: hadoop_logs
	at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1071)
	at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:252)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:414)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
	at org.eclipse.jetty.server.Server.handle(Server.java:499)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.cloud.ZooKeeperException: Specified config does not exist in ZooKeeper: hadoop_logs
	at org.apache.solr.common.cloud.ZkStateReader.readConfigName(ZkStateReader.java:161)
	at org.apache.solr.cloud.CloudConfigSetService.createCoreResourceLoader(CloudConfigSetService.java:36)
	at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:75)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:818)
	at org.apache.solr.core.CoreContainer.access$000(CoreContainer.java:90)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:473)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:464)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more

[#2 · Accepted solution]

For those facing this issue with hadoop_logs (Log Search)

• Delete the Log Search service from Ambari UI
• Go into the zookeeper shell /usr/hdp/current/zookeeper-client/bin/zkCli.sh
• rmr /infra-solr/configs/hadoop_logs
  
• Open up solr and delete the shards/collection for hadoop_logs
• SSH into your SOLR servers as root/sudo
• Delete /opt/ambari_infra_solr/data/hadoop_logs*
• Make sure you have at least 2 instances of SOLR installed
• Install Log Search on a new node (a SOLR instance should be installed on this node as well)
• Log search should be running now