@yjiang @Pardeep @khireswar Kalita @rmaruthiyodan
I am having an issue starting up kafka from kafka trying to connect to zookeeper. We have the jaas file on kafka setup as others have shown with KafkaServer and Client (Ex https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_security/content/secure-kafka-config-opt...) and have the zookeeper nodes with jaas specified for Server. We are setting -Djava.security.auth.login.config to the jaas file locations and -Djava.security.krb5.conf to the krb5.conf file location for the startup of both zookeeper and kafka. From both zookeeper and kafka the keytabs were generated and can be used to kinit against kdc. Updated zookeeper.properties to be secured and zookeeper starts up fine. On startup, Kafka is able to generate a valid "Client" tgt from the jaas and we can also see in the logs "Socket connection established to <zkserver>".
Then, zookeeper state changes and the error "Server not found in Kerberos database" exception is seen. Kafka fails to start. Do you know of any other parameters that need to be set in order to overcome this error? Please let me know if you would like me to clarify any configs/etc. Thanks.
....
INFO Waiting for keeper state SaslAuthenticated (org.I0Itec.zkclient.ZkClient)
INFO Client successfully logged in. (org.apache.zookeeper.Login)
INFO TGT refresh thread started. (org.apache.zookeeper.Login)
INFO TGT valid starting at: ....(org.apache.zookeeper.Login)
INFO Session establishment complete on server
....
INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
ERROR An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state. (org.apache.zookeeper.client.ZooKeeperSaslClient)
....
