I am working on Play Framework using Scala and trying to query my Solr collection using scala code from windows(i.e. outside of the cluster where solr is installed). My cluster is secured through Kerberos.
Below is the code I am using to connect with Solr.
val kbfile = play.Play.application().configuration().getString("mykeytab");
val user = "userid@INTERNAL.XYZ.COM";
UserGroupInformation.loginUserFromKeytab(user, kbfile)
val jaas=play.Play.application().configuration().getString("myjaasfile");
System.setProperty("java.security.auth.login.config", jaas);
HttpClientUtil.setConfigurer(new Krb5HttpClientConfigurer());
val zkHostString = "server1:2181,server2:2181,server3:2181/solr"
val cloudSolrServer = new CloudSolrServer(zkHostString);
cloudSolrServer.setDefaultCollection("testindex3");
cloudSolrServer.connect();
val query:SolrQuery = new SolrQuery();
query.addFilterQuery("Supplier_ID:*");
val params = SolrParams.toSolrParams(query.toNamedList());
val response:QueryResponse = cloudSolrServer.query(params);
I am fine upto the point where the cloudsolrserver is getting connected. But the below line is giving the error
val response:QueryResponse = cloudSolrServer.query(params);
Below is the job log
17/04/17 01:39:34 INFO play: Application started (Dev)
17/04/17 01:39:45 INFO security.UserGroupInformation: Login successful for user userid@INTERNAL.XYZ.COM using keytab file Y:/PlayPOC/PlayProject2/conf/mykeytab
17/04/17 01:39:45 INFO impl.HttpClientUtil: Setting up SPNego auth with config:Y:/PlayPOC/PlayProject2/conf/devl.spfjaas.conf
17/04/17 01:39:45 INFO cloud.SolrZkClient: Using default ZkCredentialsProvider
17/04/17 01:39:45 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=server1:2181,server2:2181,server3:2181/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@6d5ebdc0
17/04/17 01:39:45 INFO cloud.ConnectionManager: Waiting for client to connect to ZooKeeper
17/04/17 01:39:45 INFO client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
17/04/17 01:39:45 INFO zookeeper.ClientCnxn: Opening socket connection to server server1/xxx.xx.xxx.xx:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
17/04/17 01:39:45 INFO zookeeper.ClientCnxn: Socket connection established to server1/xxx.xx.xxx.xx:2181, initiating session
17/04/17 01:39:45 INFO zookeeper.ClientCnxn: Session establishment complete on server server1/xxx.xx.xxx.xx:2181, sessionid = 0x35b4ec3916b6d20, negotiated timeout = 10000
17/04/17 01:39:45 INFO cloud.ConnectionManager: Watcher org.apache.solr.common.cloud.ConnectionManager@6d5ebdc0 name:ZooKeeperConnection Watcher:server1:2181,server2:2181,server3:2181/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
17/04/17 01:39:45 INFO cloud.ConnectionManager: Client is connected to ZooKeeper
17/04/17 01:39:45 INFO cloud.SolrZkClient: Using default ZkACLProvider
17/04/17 01:39:45 INFO cloud.ConnectionManager: Watcher org.apache.solr.common.cloud.ConnectionManager@6d5ebdc0 name:ZooKeeperConnection Watcher:server1:2181,server2:2181,server3:2181/solr
got event WatchedEvent state:SaslAuthenticated type:None path:null path:null type:None
17/04/17 01:39:45 INFO cloud.ZkStateReader: Updating cluster state from ZooKeeper...
17/04/17 01:39:46 INFO cloud.SolrZkClient: Using default ZkCredentialsProvider
17/04/17 01:39:46 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=server1:2181,server2:2181,server3:2181/solr sessionTimeout=90000000 watcher=org.apache.solr.common.cloud.ConnectionManager@285c2e0f
17/04/17 01:39:46 INFO cloud.ConnectionManager: Waiting for client to connect to ZooKeeper
17/04/17 01:39:46 INFO client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
17/04/17 01:39:46 INFO zookeeper.ClientCnxn: Opening socket connection to server server2/xxx.xx.xxx.xx:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
17/04/17 01:39:46 INFO zookeeper.ClientCnxn: Socket connection established to server2/xxx.xx.xxx.xx:2181, initiating session
17/04/17 01:39:46 INFO zookeeper.ClientCnxn: Session establishment complete on server server2/xxx.xx.xxx.xx:2181, sessionid = 0x25b4ec3941f6cda , negotiated timeout = 60000
17/04/17 01:39:46 INFO cloud.ConnectionManager: Watcher org.apache.solr.common.cloud.ConnectionManager@285c2e0f name:ZooKeeperConnection Watcher:server1:2181,server2:2181,server3:2181/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
17/04/17 01:39:46 INFO cloud.ConnectionManager: Client is connected to ZooKeeper
17/04/17 01:39:46 INFO cloud.SolrZkClient: Using default ZkACLProvider
Below is the error I am getting
Execution exception in Y:\PlayPOC\PlayProject2\app\libs\solr\scala\SolrClient.scala:88
at play.api.Application$class.handleError(Application.scala:293)
at play.api.DefaultApplication.handleError(Application.scala:399)
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$12$$anonfun$apply$1.applyOrElse(PlayDefaultUpstreamHandler.scala:165)
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$12$$anonfun$apply$1.applyOrElse(PlayDefaultUpstreamHandler.scala:162)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:185)
at scala.util.Try$.apply(Try.scala:161)
at scala.util.Failure.recover(Try.scala:185)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:387)
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:387)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:29)
at play.api.libs.iteratee.Execution$$anon$1.execute(Execution.scala:43)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:37)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:133)
at scala.concurrent.Promise$class.complete(Promise.scala:55)
at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:58)
at scala.concurrent.Future$$anonfun$flatMap$1$$anonfun$apply$3.apply(Future.scala:278)
at scala.concurrent.Future$$anonfun$flatMap$1$$anonfun$apply$3.apply(Future.scala:278)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:29)
at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.processBatch$1(Future.scala:731)
at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.apply$mcV$sp(Future.scala:746)
at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.apply(Future.scala:723)
at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.apply(Future.scala:723)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at scala.concurrent.Future$InternalCallbackExecutor$Batch.run(Future.scala:722)
at scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:782)
at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:773)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:37)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:133)
at scala.concurrent.Promise$class.complete(Promise.scala:55)
at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:58)
at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:274)
at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:274)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:29)
at play.api.libs.iteratee.Execution$$anon$2.execute(Execution.scala:70)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:37)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:133)
at scala.concurrent.Promise$class.complete(Promise.scala:55)
at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:58)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:23)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://SolrServer1:8985/solr/testindex3, https://SolrServer2:8985/solr/testindex3, https://SolrServer3:8985/solr/testindex3]
at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:324)
at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:684)
at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
at libs.solr.scala.SolrClient.<init>(SolrClient.scala:88)
at libs.SearchLib$.buildQuery(SearchLib.scala:85)
at libs.SearchLib$.get(SearchLib.scala:36)
at controllers.Application$$anonfun$shiv$1.apply(Application.scala:22)
at controllers.Application$$anonfun$shiv$1.apply(Application.scala:21)
at play.api.mvc.ActionBuilder$$anonfun$apply$10.apply(Action.scala:221)
at play.api.mvc.ActionBuilder$$anonfun$apply$10.apply(Action.scala:220)
at play.api.mvc.Action$.invokeBlock(Action.scala:357)
at play.api.mvc.ActionBuilder$$anon$1.apply(Action.scala:309)
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:109)
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:109)
at play.utils.Threads$.withContextClassLoader(Threads.scala:18)
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:108)
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:107)
at scala.Option.map(Option.scala:145)
at play.api.mvc.Action$$anonfun$apply$1.apply(Action.scala:107)
at play.api.mvc.Action$$anonfun$apply$1.apply(Action.scala:100)
at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:481)
at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:481)
at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:517)
at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:517)
at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$13.apply(Iteratee.scala:493)
at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$13.apply(Iteratee.scala:493)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
... 6 more
Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://SolrServer1:8985/solr/testindex3
at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:633)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:228)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:224)
at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:345)
at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:306)
... 34 more
Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at sun.security.ssl.SSLSessionImpl.getPeerCertificates(Unknown Source)
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:126)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:437)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:643)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:513)
... 38 more
In summary, I am getting the below 3 errors even though I am connected with Zookeeper using keytab and jaas file.
1. org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://SolrServer1:8985/solr/testindex3, https://SolrServer2:8985/solr/testindex3, https://SolrServer3:8985/solr/testindex3]
2. org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://SolrServer1:8985/solr/testindex3
3. javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
To make sure if my nodes are live, I used the below code to check if zookeeper can access the nodes and the reqired nodes are live and I got a good list of servers including the solr collection server I am getting error.
val solrZK = ZkClient.getSolrZooKeeper();
println("Printing the output of zookeeper live nodes")
println(solrZK.getChildren("/live_nodes", true));
Please advise what I am missing or doing wrong.
