Could not load solr.xml from zookeeper: node not found

[#1]

Newly installed cluster which began with CDH 5.5 and was recently upgraded to CDH 5.7.

Errors encountered during startup of cluster service are identical for both versions.

I would greatly appreciate any suggestions on how to proceed. Thanks!

ROLE LOG OUTPUT:

May 24, 3:43:17.052 PM ERROR org.apache.solr.servlet.SolrDispatchFilter Could not start Solr. Check solr/home property and the logs

May 24, 3:43:17.087 PM ERROR org.apache.solr.core.SolrCore null:org.apache.solr.common.SolrException: Could not load solr.xml from zookeeper at org.apache.solr.servlet.SolrDispatchFilter.loadConfigSolr(SolrDispatchFilter.java:180) at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:196) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:145) at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:298) at org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:119) at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4076) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4730) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:802) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583) at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1080) at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1003) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:507) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1068) at org.apache.catalina.core.StandardHost.start(StandardHost.java:822) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1060) at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463) at org.apache.catalina.core.StandardService.start(StandardService.java:525) at org.apache.catalina.core.StandardServer.start(StandardServer.java:759) at org.apache.catalina.startup.Catalina.start(Catalina.java:595) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) Caused by: org.apache.solr.common.SolrException: Could not load solr.xml from zookeeper: node not found at org.apache.solr.servlet.SolrDispatchFilter.loadConfigSolr(SolrDispatchFilter.java:176) ... 28 more

LAST ENTRIES IN STDERR LOG OUTPUT..

+ SOLR_HOME_TMP=/var/lib/solr
+ export SOLR_HOME=/opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/solr
+ SOLR_HOME=/opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/solr
+ '[' '' = true ']'
+ eval /opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/solr/bin/solrctl.sh --zk slrs01va6f1clm1.a6.ad.digital.accenture.com:2181/data/solr cluster --set-property urlScheme http
++ /opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/solr/bin/solrctl.sh --zk slrs01va6f1clm1.a6.ad.digital.accenture.com:2181/data/solr cluster --set-property urlScheme http
+ '[' 0 == 0 ']'
+ echo 'Successfully configured urlScheme property for Solr cluster in Zookeeper'
+ export SOLR_HOME=/var/lib/solr
+ SOLR_HOME=/var/lib/solr
+ mkdir -p /var/run/cloudera-scm-agent/process/478-solr-SOLR_SERVER/temp
+ exec /opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/solr/bin/solrd run
May 24, 2016 3:43:16 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
May 24, 2016 3:43:16 PM org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-8983
May 24, 2016 3:43:16 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 487 ms
May 24, 2016 3:43:16 PM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
May 24, 2016 3:43:16 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.44
May 24, 2016 3:43:16 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory solr
May 24, 2016 3:43:17 PM org.apache.catalina.core.StandardContext start
SEVERE: Error filterStart
May 24, 2016 3:43:17 PM org.apache.catalina.core.StandardContext start
SEVERE: Context [/solr] startup failed due to previous errors
May 24, 2016 3:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/solr] appears to have started a thread named [main-SendThread(slrs01va6f1clm1.a6.ad.digital.accenture.com:2181)] but has failed to stop it. This is very likely to create a memory leak.
May 24, 2016 3:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/solr] appears to have started a thread named [main-SendThread(slrs01va6f1clm1.a6.ad.digital.accenture.com:2181)] but has failed to stop it. This is very likely to create a memory leak.
May 24, 2016 3:43:17 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory ROOT
May 24, 2016 3:43:17 PM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-8983
May 24, 2016 3:43:17 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 1160 ms

[#2 · Accepted solution]

It looks like zookeeper didn't get initialized, can you shut down the solr service and run the "Initialize Solr" action from the solr service page?

If that doesn't work, then try the following from the command line:

solrctl init --force

(Note: this is a descructive command which will remove all solr configurations in zk and reinitialize. Do not use if it you have solr configuration you need to keep)

-pd

[#3 · Accepted solution]

It looks like zookeeper didn't get initialized, can you shut down the solr service and run the "Initialize Solr" action from the solr service page?

If that doesn't work, then try the following from the command line:

solrctl init --force

(Note: this is a descructive command which will remove all solr configurations in zk and reinitialize. Do not use if it you have solr configuration you need to keep)

-pd

[#4 · Accepted solution]

Hello pdvorak
Initialize from service page did not work - message = "Skipping because Solr service is already initialized."

I executed "solrctl init --force" as you suggested and service started OK for the first time. 🙂

I am now seeing a different issue in solr log, but service remains up and running. I will file case for following issue, but this issue can be considered resolved. Thanks!

2016-05-25 10:56:31,870 INFO org.apache.solr.servlet.SolrDispatchFilter: [admin] webapp=null path=/admin/cores params={action=STATUS&wt=json} status=0 QTime=0
2016-05-25 10:57:31,792 INFO org.apache.solr.servlet.SolrDispatchFilter: Could not consume full client request
java.io.IOException: Stream closed

[#5]

There is a regression that was introduced in CDH5.7.0 that throws that stack trace (at INFO level) for every update, or /admin/cores command.

The workaround is to set the following in the logging safety valve for the solr service:

log4j.logger.org.apache.solr.servlet.SolrDispatchFilter=WARN

This will be resolved in CDH5.7.1.

-pd

[#6]

Hi @pdvorak,

from today, I'm having the same error, but I can't execute the command: solrctl init --force in Prod enviroment, is there any other solution in order to fix this issue?

(2018-05-07 14:47:14,451 WARN org.apache.zookeeper.ClientCnxn: Session 0x1633aa1e11a0014 for server [server name],
unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Unreasonable length = 1051274
        at org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:127)
        at org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:92)
        at org.apache.zookeeper.proto.GetDataResponse.deserialize(GetDataResponse.java:54)
        at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:814)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2018-05-07 14:47:14,451 ERROR org.apache.solr.servlet.SolrDispatchFilter: Could not start Solr. Check solr/home property and the logs
2018-05-07 14:47:14,511 ERROR org.apache.solr.core.SolrCore: null:java.lang.NullPointerException)

Many thanks in advance for the kind cooperation.

Regards,

Alex

[#7]

This seems to be indicating that the jute.maxbuffer has been exceeded.

You can increase this on the command line side by exporting the following:
export ZKCLI_JVM_FLAGS=-Djute.maxbuffer=4194304

You may need to confirm on the ZK service configuration, if the jute.maxbuffer size is also 4 MB

-pd