Specified config does not exist in ZooKeeper

[#1]

Hi all,

I have managed to setup SOLR via Cloudera Manager, on 2 hosts.

I followed the steps up to Creating Your First Solr Collection... 

$ solrctl instancedir --generate $HOME/solr_configs

$ solrctl instancedir --create collection1 $HOME/solr_configs

$ solrctl collection --create collection1 -s 2

And I get the following error:

org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:Error CREATEing SolrCore 'collection1_shard1_replica1': Unable to create core: collection1_shard1_replica1 Caused by: Could not find configName for collection collection1 found:null

So I thought I need to add in the -c param:

$ solrctl collection --create collection1 -s 2 -c collection1

And instead I got this error:

org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:Error CREATEing SolrCore 'collection1_shard2_replica1': Unable to create core: collection1_shard2_replica1 Caused by: Specified config does not exist in ZooKeeper:collection1

Here are some of the corresponding Java error trace:

org.apache.solr.common.cloud.ZooKeeperException: Specified config does not exist in ZooKeeper:collection1
	at org.apache.solr.cloud.ZkController.readConfigName(ZkController.java:708)
	at org.apache.solr.core.CoreContainer.createFromZk(CoreContainer.java:937)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1035)
	at org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:509)
	at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:145)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
	at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:615)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:206)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:155)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.solr.servlet.SolrHadoopAuthenticationFilter$1.doFilter(SolrHadoopAuthenticationFilter.java:97)
	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:384)
	at org.apache.solr.servlet.SolrHadoopAuthenticationFilter.doFilter(SolrHadoopAuthenticationFilter.java:105)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
	at java.lang.Thread.run(Thread.java:662) 

I got a feeling that I need to modify some ZK config somewhere. Can someone point me to the right directions?

Many thanks in advance!

[#2]

Make sure your /etc/default/solr SOLR_ZK_ENSEMBLE property is set the same as CM. CM uses a /solr zk chroot by default, so make sure that chroot is also used in the /etc/default/solr config. Alternatively, you can pass an explicit zk address when making the call via:

solrctl --zk zkaddress:port/solr instancedir --create collection1 $HOME/solr_configs

To debug this, you might want to use a tool to inspect ZooKeeper. The configuration files that you uploaded should be in /solr/configs/{name}. Do you see anything there - or perhaps you see it in /configs/{name}? That would likely be the case if the configured chroots did not match.

- Mark

[#3]

My /etc/default/solr is configured as below:

SOLR_PORT=8983
SOLR_ADMIN_PORT=8984
SOLR_LOG=/var/log/solr
SOLR_ZK_ENSEMBLE=master:2181,n1:2181
SOLR_HDFS_HOME=hdfs://master:8020/solr
SOLR_HDFS_CONFIG=/etc/hadoop/conf

Where master is my hdfs namenode as well as a solr instance. n1 is another solr instance.

I've tried the explicit manner, but it came back with the exact same error.

How do I inspect zookeper?

[#4]

Can you paste the command you tried when being explicit?

I would say the likely problem is that you should use the following zk address:

SOLR_ZK_ENSEMBLE=master:2181,n1:2181/solr

CM will use a root of /solr by default - you need to chroot onto that node, and so your final address in the ensemble string should include the chroot. To inspect ZooKeeper you have many options. If Solr is up, you can inspect ZooKeeper via it's admin UI. Solr has to be able to connect to ZooKeeper first though - and will see a limited chroot'd view. ZooKeeper also comes with some simple command line tools - you can get started with those here: http://zookeeper.apache.org/doc/r3.4.5/zookeeperStarted.html There are also some graphical tools - I like to use a plugin for eclipse to browse ZK. Google should provide some other alternatives.

[#5]

I tried the following variants with similar results:

solrctl --zk master:2181/solr,n1:2181/solr collection --create collection1 -s 2
solrctl --zk master:2181/solr,n1:2181/solr collection --create collection1 -s 2 -c collection1
solrctl --zk master:2181,n1:2181collection --create collection1 -s 2
solrctl --zk master:2181,n1:2181collection --create collection1 -s 2 -c collection1

I noticed in your example, only the n1 is postfixed with /solr... is that a typo?

I didn't see any means of inspecting Zookeeper via SOLR's admin UI though. 

[#6]

Hi @adams

You mentioned that you deployed Solr using Cloudera Manager (CM)

• If this is the case, CM neither reads nor updates the values saved in "/etc/default/solr". It maintains its own own database of these values.
• You can view/edit the current properties of Solr if you click on "Configuration" in CM's SOLR-1 homepage
• And if you want to check out the configs being deployed to all the Solr processes, there is an option to download the client configs under "Actions" of SOLR-1 homepage

-Vamsee

[#7]

Can you login to any host and run:

[cconner@cdh44-1 ~]$ zookeeper-client

[zk: localhost:2181(CONNECTED) 0]
[zk: localhost:2181(CONNECTED) 0] ls /solr/collections
[collection2, collection1]
[zk: localhost:2181(CONNECTED) 1] ls /solr/collections/collection1
[leader_elect, leaders]

If collection1 is there, you should see it when you enter the ZK client and run ls /solr/collections.

Thanks

Chris

[#8]

Hi Chris,

I tried your suggestions... and various alternatives:

[zk: localhost:2181(CONNECTED) 8] ls /
[configs, solr,n1:2181, zookeeper, solr.xml, solr,n1.hdc:2181, solr]
[zk: localhost:2181(CONNECTED) 9] ls /solr 
[clusterstate.json, aliases.json, solr.xml, live_nodes, overseer, collections, overseer_elect]
[zk: localhost:2181(CONNECTED) 10] ls /solr/collections
[]
[zk: localhost:2181(CONNECTED) 11] ls /solr,n1:2181             
[solr]
[zk: localhost:2181(CONNECTED) 12] ls /solr,n1:2181/solr        
[configs]
[zk: localhost:2181(CONNECTED) 13] ls /solr,n1:2181/solr/configs
[collection4]
[zk: localhost:2181(CONNECTED) 14] ls /solr,n1.hdc:2181/solr    
[configs]
[zk: localhost:2181(CONNECTED) 15] ls /solr,n1.hdc:2181/solr/configs
[collection2, collection1]
[zk: localhost:2181(CONNECTED) 16] 

Btw, my nodes are all in the domain hdc.