Cloudera Community
chirunimmala
user rank
Explorer
My Accepted Solutions
Show All

Re: Kafka broker SASL connection failing with Zook...

by Explorer in Support Questions
‎09-11-2020 05:48 PM
‎09-11-2020 05:48 PM
I am able to fix this issue, posting the resolution here just in case if it helps anyone. I was using the embedded zookeeper which comes with Kafka. I see the below code is missing in the zookeeper server start script which will initialize the environment for the zookeeper. After adding the below code, the jvm process is able to pick the jaas file properly and sasl configuration is complete. After this Kafka is able to SASL auth to zookeeper without any issues.   if [ "x$KAFKA_OPTS" = "x" ]; then     export KAFKA_OPTS="-Djava.security.auth.login.config=/home/zookeeper/zookeeper_jaas.conf" fi ... View more

Kafka broker SASL connection failing with Zookeepe...

by Explorer in Support Questions
‎09-07-2020 02:05 AM
‎09-07-2020 02:05 AM
Dear experts,   I have installed apache kafka 2.4 on one node. I am having SSL and SASL(Kerberos) enabled for kafka broker and now enabled SASL for zookeeper. However when starting the broker , i am getting the below error, could you please help on this ?   --error log---   [2020-09-07 14:11:09,761] DEBUG Client principal is "kafka/broker0@KAFKA.SECURE". (org.apache.zookeeper.Login) [2020-09-07 14:11:09,761] DEBUG Server principal is "krbtgt/KAFKA.SECURE@KAFKA.SECURE". (org.apache.zookeeper.Login) [2020-09-07 14:11:09,764] INFO TGT valid starting at: Mon Sep 07 14:11:09 IST 2020 (org.apache.zookeeper.Login) [2020-09-07 14:11:09,765] INFO TGT expires: Tue Sep 08 14:11:09 IST 2020 (org.apache.zookeeper.Login) [2020-09-07 14:11:09,765] INFO TGT refresh sleeping until: Tue Sep 08 09:30:58 IST 2020 (org.apache.zookeeper.Login) [2020-09-07 14:11:09,765] INFO Client will use GSSAPI as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient) [2020-09-07 14:11:09,766] DEBUG creating sasl client: Client=kafka/broker0@KAFKA.SECURE;service=kafka;serviceHostname=broker0 (org.apache.zookeeper.client.ZooKeeperSaslClient) [2020-09-07 14:11:09,773] INFO Opening socket connection to server broker0/X.X.X.X:2181. Will attempt to SASL-authenticate using Login Context section 'ZkClient' (org.apache.zookeeper.ClientCnxn) [2020-09-07 14:11:09,778] INFO Socket connection established, initiating session, client: /X.X.X.X:54728, server: broker0/X.X.X.X:2181 (org.apache.zookeeper.ClientCnxn) [2020-09-07 14:11:09,780] DEBUG Session establishment request sent on broker0/X.X.X.X:2181 (org.apache.zookeeper.ClientCnxn) [2020-09-07 14:11:09,785] INFO Session establishment complete on server broker0/X.X.X.X:2181, sessionid = 0x100000039900003, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn) [2020-09-07 14:11:09,786] DEBUG ClientCnxn:sendSaslPacket:length=0 (org.apache.zookeeper.client.ZooKeeperSaslClient) [2020-09-07 14:11:09,787] DEBUG saslClient.evaluateChallenge(len=0) (org.apache.zookeeper.client.ZooKeeperSaslClient) [2020-09-07 14:11:09,789] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient) [2020-09-07 14:11:09,811] ERROR SASL authentication failed using login context 'ZkClient' with exception: {} (org.apache.zookeeper.client.ZooKeeperSaslClient) javax.security.sasl.SaslException: Error in authenticating with a Zookeeper Quorum member: the quorum member's saslToken is null. at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:312) at org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:275) at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:882) at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:101) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:363) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) [2020-09-07 14:11:09,814] ERROR [ZooKeeperClient Kafka server] Auth failed. (kafka.zookeeper.ZooKeeperClient) [2020-09-07 14:11:09,833] INFO EventThread shut down for session: 0x100000039900003 (org.apache.zookeeper.ClientCnxn) [2020-09-07 14:11:09,889] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /consumers   ---- kafka jaas file --- KafkaServer { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true debug=true serviceName="kafka" keyTab="/home/kafka/kafka.service.keytab" principal="kafka/broker0@KAFKA.SECURE"; }; // ZooKeeper client authentication ZkClient{ com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true debug=true serviceName="zookeeper" keyTab="/home/kafka/kafka.service.keytab" principal="kafka/broker0@KAFKA.SECURE"; };   -- zookeeper jaas--   QuorumServer { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/home/zookeeper/zookeeper.service.keytab" storeKey=true useTicketCache=false debug=false principal="zookeeper/broker0@EXAMPLE.COM"; }; QuorumLearner { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/home/zookeeper/zookeeper.service.keytab" storeKey=true useTicketCache=false debug=false principal="zookeeper/broker0@EXAMPLE.COM"; }; Server { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true useTicketCache=false storeKey=true debug=true keytab="/home/zookeeper/zookeeper.service.keytab" principal="zookeeper/broker0@KAFKA.SECURE"; };   Thanks, Chiranjeevi ... View more
Labels:

Re: Cloudera agent SSL error

by Explorer in Support Questions
‎03-27-2019 09:27 PM
‎03-27-2019 09:27 PM
Hi,   Yes i have completed the steps in "Enable server certificate verification" . Please find details below,    Note: The verifier.pem file has both RootCA and IntermediateCA certificates and cmhost.antuit.internal.pem has the signed certificate + IntermediateCA certificate   # A file of CA certificates in PEM format. The file can contain several CA # certificates identified by # # -----BEGIN CERTIFICATE----- # ... (CA certificate in base64 encoding) ... # -----END CERTIFICATE----- # # sequences. Before, between, and after the certificates text is allowed which # can be used e.g. for descriptions of the certificates. # # The file is loaded once, the first time an HTTPS connection is attempted. A # restart of the agent is required to pick up changes to the file. # # Note that if neither verify_cert_file or verify_cert_dir is set, certificate # verification will not be performed. verify_cert_file=/opt/cloudera/security/pki/verifier.pem [root@cmhost pki]# openssl s_client -connect cmhost.antuit.internal:7182 -CAfile verifier.pem -cert cmhost.antuit.internal.pem -key agent.key Enter pass phrase for agent.key: CONNECTED(00000003) depth=2 C = IN, ST = KA, L = BNG, O = Antuit, OU = DE, CN = Antuit Root CA verify return:1 depth=1 C = IN, ST = KA, O = Antuit, OU = DE, CN = Antuit Inter CA verify return:1 depth=0 C = IN, ST = KA, L = BNG, O = Antuit, OU = DE, CN = cmhost.antuit.internal verify return:1 140606215886736:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:s3_pkt.c:1493:SSL alert number 46 140606215886736:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177: --- Certificate chain 0 s:/C=IN/ST=KA/L=BNG/O=Antuit/OU=DE/CN=cmhost.antuit.internal i:/C=IN/ST=KA/O=Antuit/OU=DE/CN=Antuit Inter CA 1 s:/C=IN/ST=KA/O=Antuit/OU=DE/CN=Antuit Inter CA i:/C=IN/ST=KA/L=BNG/O=Antuit/OU=DE/CN=Antuit Root CA --- ... View more

Cloudera agent SSL error

by Explorer in Support Questions
‎03-27-2019 04:30 AM
‎03-27-2019 04:30 AM
Hi team,   I am trying to enable SSL in transit for my cloudera cluster using the document https://www.cloudera.com/documentation/enterprise/latest/topics/how_to_configure_cm_tls.html   I am able to successfully configure till the step "Enable Server Certificate Verification on Cloudera Manager Agents" , however once i completed "Configure Agent Certificate Authentication" , i am receiving the below error and all the hosts are in bad health state, could you please help ?   [27/Mar/2019 11:12:37 +0000] 1022 MainThread agent ERROR Heartbeating to cmhost.antuit.internal:7182 failed. Traceback (most recent call last): File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1388, in _send_heartbeat self.cfg.max_cert_depth) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__ self.conn.connect() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 80, in connect sock.connect((self.host, self.port)) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 305, in connect ret = self.connect_ssl() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 292, in connect_ssl return m2.ssl_connect(self.ssl, self._timeout) SSLError: sslv3 alert certificate unknown   Notes: -----   1. I am using a private intermediate CA to sign the certificates for each host. 2. I have imported both root and intermediate CA certs into jssecacerts in the cloudera manager host  3. I am able to manually verify the signed certs cat ca.cert.pem intermediate.cert.pem > verify.pem sudo openssl verify -CAfile verifier.pem cmhost.XX.YY.pem cmhost.XX.YY.pem: OK   Thanks, Chiranjeevi ... View more
Labels:
Contact Me
Online
Offline
Last Visited
‎09-23-2020 11:51 AM
Community Statistics
Member Since ‎03-27-2019 04:31 AM
Last Visited ‎09-23-2020 11:51 AM
Posts 4