Reply
Explorer
Posts: 16
Registered: ‎12-27-2016

Solr error: No live SolrServers available to handle this request and peer not authenticated

I am working on Play Framework using Scala  and trying to query my Solr collection using scala code from windows(i.e. outside of the cluster where solr is installed).  My cluster is secured through Kerberos.

Below is the code I am using to connect with Solr.

val kbfile = play.Play.application().configuration().getString("mykeytab");

val user = "userid@INTERNAL.XYZ.COM";

UserGroupInformation.loginUserFromKeytab(user, kbfile)

val jaas=play.Play.application().configuration().getString("myjaasfile");

System.setProperty("java.security.auth.login.config", jaas);

HttpClientUtil.setConfigurer(new Krb5HttpClientConfigurer());

val zkHostString = "server1:2181,server2:2181,server3:2181/solr"

val cloudSolrServer = new CloudSolrServer(zkHostString);

cloudSolrServer.setDefaultCollection("testindex3");

cloudSolrServer.connect();

val query:SolrQuery = new SolrQuery();

query.addFilterQuery("Supplier_ID:*");

val params = SolrParams.toSolrParams(query.toNamedList());

val response:QueryResponse = cloudSolrServer.query(params);

 

I am fine upto the point where the cloudsolrserver is getting connected. But the below line is giving the error

val response:QueryResponse = cloudSolrServer.query(params);

 

Below is the job log

 

17/04/17 01:39:34 INFO play: Application started (Dev)
17/04/17 01:39:45 INFO security.UserGroupInformation: Login successful for user userid@INTERNAL.XYZ.COM using keytab file Y:/PlayPOC/PlayProject2/conf/mykeytab
17/04/17 01:39:45 INFO impl.HttpClientUtil: Setting up SPNego auth with config:Y:/PlayPOC/PlayProject2/conf/devl.spfjaas.conf
17/04/17 01:39:45 INFO cloud.SolrZkClient: Using default ZkCredentialsProvider
17/04/17 01:39:45 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=server1:2181,server2:2181,server3:2181/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@6d5ebdc0
17/04/17 01:39:45 INFO cloud.ConnectionManager: Waiting for client to connect to ZooKeeper
17/04/17 01:39:45 INFO client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
17/04/17 01:39:45 INFO zookeeper.ClientCnxn: Opening socket connection to server server1/xxx.xx.xxx.xx:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
17/04/17 01:39:45 INFO zookeeper.ClientCnxn: Socket connection established to server1/xxx.xx.xxx.xx:2181, initiating session
17/04/17 01:39:45 INFO zookeeper.ClientCnxn: Session establishment complete on server server1/xxx.xx.xxx.xx:2181, sessionid = 0x35b4ec3916b6d20, negotiated timeout = 10000
17/04/17 01:39:45 INFO cloud.ConnectionManager: Watcher org.apache.solr.common.cloud.ConnectionManager@6d5ebdc0 name:ZooKeeperConnection Watcher:server1:2181,server2:2181,server3:2181/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
17/04/17 01:39:45 INFO cloud.ConnectionManager: Client is connected to ZooKeeper
17/04/17 01:39:45 INFO cloud.SolrZkClient: Using default ZkACLProvider
17/04/17 01:39:45 INFO cloud.ConnectionManager: Watcher org.apache.solr.common.cloud.ConnectionManager@6d5ebdc0 name:ZooKeeperConnection Watcher:server1:2181,server2:2181,server3:2181/solr
got event WatchedEvent state:SaslAuthenticated type:None path:null path:null type:None
17/04/17 01:39:45 INFO cloud.ZkStateReader: Updating cluster state from ZooKeeper...
17/04/17 01:39:46 INFO cloud.SolrZkClient: Using default ZkCredentialsProvider
17/04/17 01:39:46 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=server1:2181,server2:2181,server3:2181/solr sessionTimeout=90000000 watcher=org.apache.solr.common.cloud.ConnectionManager@285c2e0f
17/04/17 01:39:46 INFO cloud.ConnectionManager: Waiting for client to connect to ZooKeeper
17/04/17 01:39:46 INFO client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
17/04/17 01:39:46 INFO zookeeper.ClientCnxn: Opening socket connection to server server2/xxx.xx.xxx.xx:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
17/04/17 01:39:46 INFO zookeeper.ClientCnxn: Socket connection established to server2/xxx.xx.xxx.xx:2181, initiating session
17/04/17 01:39:46 INFO zookeeper.ClientCnxn: Session establishment complete on server server2/xxx.xx.xxx.xx:2181, sessionid = 0x25b4ec3941f6cda , negotiated timeout = 60000
17/04/17 01:39:46 INFO cloud.ConnectionManager: Watcher org.apache.solr.common.cloud.ConnectionManager@285c2e0f name:ZooKeeperConnection Watcher:server1:2181,server2:2181,server3:2181/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
17/04/17 01:39:46 INFO cloud.ConnectionManager: Client is connected to ZooKeeper
17/04/17 01:39:46 INFO cloud.SolrZkClient: Using default ZkACLProvider


Below is the error I am getting
Execution exception in Y:\PlayPOC\PlayProject2\app\libs\solr\scala\SolrClient.scala:88
at play.api.Application$class.handleError(Application.scala:293)
at play.api.DefaultApplication.handleError(Application.scala:399)
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$12$$anonfun$apply$1.applyOrElse(PlayDefaultUpstreamHandler.scala:165)
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$12$$anonfun$apply$1.applyOrElse(PlayDefaultUpstreamHandler.scala:162)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:185)
at scala.util.Try$.apply(Try.scala:161)
at scala.util.Failure.recover(Try.scala:185)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:387)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:387)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:29)
at play.api.libs.iteratee.Execution$$anon$1.execute(Execution.scala:43)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:37)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:133)
at scala.concurrent.Promise$class.complete(Promise.scala:55)
at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:58)
at scala.concurrent.Future$$anonfun$flatMap$1$$anonfun$apply$3.apply(Future.scala:278)
at scala.concurrent.Future$$anonfun$flatMap$1$$anonfun$apply$3.apply(Future.scala:278)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:29)
at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.processBatch$1(Future.scala:731)
at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.apply$mcV$sp(Future.scala:746)
at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.apply(Future.scala:723)
at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.apply(Future.scala:723)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at scala.concurrent.Future$InternalCallbackExecutor$Batch.run(Future.scala:722)
at scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:782)
at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:773)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:37)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:133)
at scala.concurrent.Promise$class.complete(Promise.scala:55)
at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:58)
at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:274)
at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:274)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:29)
at play.api.libs.iteratee.Execution$$anon$2.execute(Execution.scala:70)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:37)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:133)
at scala.concurrent.Promise$class.complete(Promise.scala:55)
at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:58)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:23)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://SolrServer1:8985/solr/testindex3, https://SolrServer2:8985/solr/testindex3, https://SolrServer3:8985/solr/testindex3]
at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:324)
at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:684)
at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
at libs.solr.scala.SolrClient.<init>(SolrClient.scala:88)
at libs.SearchLib$.buildQuery(SearchLib.scala:85)
at libs.SearchLib$.get(SearchLib.scala:36)
at controllers.Application$$anonfun$shiv$1.apply(Application.scala:22)
at controllers.Application$$anonfun$shiv$1.apply(Application.scala:21)
at play.api.mvc.ActionBuilder$$anonfun$apply$10.apply(Action.scala:221)
at play.api.mvc.ActionBuilder$$anonfun$apply$10.apply(Action.scala:220)
at play.api.mvc.Action$.invokeBlock(Action.scala:357)
at play.api.mvc.ActionBuilder$$anon$1.apply(Action.scala:309)
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:109)
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:109)
at play.utils.Threads$.withContextClassLoader(Threads.scala:18)
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:108)
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:107)
at scala.Option.map(Option.scala:145)
at play.api.mvc.Action$$anonfun$apply$1.apply(Action.scala:107)
at play.api.mvc.Action$$anonfun$apply$1.apply(Action.scala:100)
at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:481)
at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:481)
at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:517)
at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:517)
at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$13.apply(Iteratee.scala:493)
at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$13.apply(Iteratee.scala:493)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
... 6 more
Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://SolrServer1:8985/solr/testindex3
at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:633)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:228)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:224)
at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:345)
at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:306)
... 34 more
Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at sun.security.ssl.SSLSessionImpl.getPeerCertificates(Unknown Source)
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:126)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:437)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:643)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:513)
... 38 more

 

In summary, I am getting the below 3 errors even though I am connected with Zookeeper using keytab and jaas file.

1. org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://SolrServer1:8985/solr/testindex3, https://SolrServer2:8985/solr/testindex3, https://SolrServer3:8985/solr/testindex3]

2. org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://SolrServer1:8985/solr/testindex3

3. javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated

 

To make sure if my nodes are live, I used the below code to check if zookeeper can access the nodes and the reqired nodes are live and I got a good list of servers including the solr collection server I am getting error.

 

val solrZK = ZkClient.getSolrZooKeeper();

  println("Printing the output of zookeeper live nodes")

println(solrZK.getChildren("/live_nodes", true));

 

Please advise what I am missing or doing wrong.

Posts: 341
Topics: 0
Kudos: 35
Solutions: 27
Registered: ‎08-16-2016

Re: Solr error: No live SolrServers available to handle this request and peer not authenticated

I don't have the answer but the error is when it is trying to verify the SSL certificate of the SOLR instance.

This article and code is working to getting around doing the SSL verification. I don't think this is what you want but you could lead you in the correct direction.
Explorer
Posts: 16
Registered: ‎12-27-2016

Re: Solr error: No live SolrServers available to handle this request and peer not authenticated

I couldn't see any article posted. Can you please post the link.

Explorer
Posts: 16
Registered: ‎12-27-2016

Re: Solr error: No live SolrServers available to handle this request and peer not authenticated

For the SSL verification, I am using the below code in scala.

 

Am I in the right direction?

 

System.setProperty("javax.net.ssl.keyStore", keystorejks);
System.setProperty("javax.net.ssl.keyStorePassword", "secret");
System.setProperty("javax.net.ssl.trustStore", truststorejks);
System.setProperty("javax.net.ssl.trustStorePassword", "secret");

 

I am not sure if the keystore or truststore password is secret or something else. The cluster has been setup by the Admin people. I am not sure if they will provide the password for these. 

 

Do we have any other way to do the SSL verification. Please advise.

Posts: 341
Topics: 0
Kudos: 35
Solutions: 27
Registered: ‎08-16-2016

Re: Solr error: No live SolrServers available to handle this request and peer not authenticated

Announcements