Support Questions
Find answers, ask questions, and share your expertise

Help on Kerberos communication between 2 clusters within same domain and Realm

Explorer

Hi,

I have 2 clusters within same domain and within same REALM.

Cluster1 has services like Spark,HDFS,HBase, Kafka and other important services

and

Cluster2 has services like Solr, Zookeeper, HDFS and other important services.

I want to use zookeeper service in cluster2 from cluster1 and I cannot do that.

Can some one help me commands to initiate zookeeper of cluster2 from cluster1 and how to debug the kerberos issue?

Thanks a lot.

4 REPLIES 4

Super Collaborator

What is your kerberos issue? And how do you want to initiate zookeeper? Are you using Ambari for managing the services?

If I assume you want to let Kafka from cluster1 use zookeeper from cluster2, than in principal it is as easy as just providing the zookeeper nodes as list in the parameter zookeeper.connect. <<hostname>>:<<port>>, so basically cluster2-node:2181.

But it will be important to see if both clusters are fully kerberos enabled, and how the services on cluster1 are configured if they don't use zookeeper currently.

Explorer

@ Harald,

Thanks for your time on this.

We want spark in cluster1 to use zookeeper from cluster2 and our designer has complained that he is getting an error like below:

17/12/26 06:30:51 WARN SolrRelation: Unknown parameters passed to query: Set(inferschema) 17/12/26 06:30:51 WARN ClientCnxn: SASL configuration failed: javax.security.auth.login.LoginException: Configuration Error - useKeyTab should be set to true to use the keytab/etc/security/keytabs/hdfs.headless.keytab Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. 17/12/26 06:30:51 WARN ConnectionManager: Watcher org.apache.solr.common.cloud.ConnectionManager@28dbc172 name: ZooKeeperConnection Watcher:172.31.XX.YY:2181,172.31.xx.yy:2181,172.31.xx.yy:2181 got event WatchedEvent state:AuthFailed type:None path:null path: null type: None 17/12/26 06:30:51 WARN ConnectionManager: zkClient received AuthFailed com.google.common.util.concurrent.UncheckedExecutionException: org.apache.solr.common.SolrException: Cannot connect to cluster at 172.31.XX.YY:2181,172.31.xx.yy:2181,172.31.xx.yy:2181: cluster not found/not ready

From admin perspective I would like to connect to zookeeper in cluster2 from cluster1 to verify if it is connection issue or command issue ( designer might not have used correct command )

So, I want to verify connectivity towards zookeeper of cluster2 from cluster1 via command line.

Hope my point is clear.

If not, please let me know.

Super Collaborator

from the message it looks like solr isn't configured to use kerberos to connect to zookeeper. In many cases the zookeeper in fact allows read-only communication even without previous authentication, so it is possible that you receive the warning, but things are still working, but it also possible that solr fails due to this authentication issue.

In any case you can install a zookeeper client on cluster1 to try to connect to zookeeper. But if I am right, your zookeeper client will work without issues, and you still see the warning from solr. you need to change the solre configuration. You will have to create or change a JAAS file. I am not sure where the file is located for solr, but to let the zookeeper client authenticate via kerberos it typically has a section like this

Client {
       com.sun.security.auth.module.Krb5LoginModule required
       useKeyTab=true
       keyTab="/path/to/client/keytab"
       storeKey=true
       useTicketCache=false
       principal="yourzookeeperclient";
};

The principal for solr is typically something like 'HTTP/solrnode@KRBREALM'

You can check for more details here:

https://lucene.apache.org/solr/guide/6_6/kerberos-authentication-plugin.html

Explorer

Both the clusters are fully kerberized.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.