Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

AuthFailedException in Storm while connecting remote non secure Kafka from Kerberized cluster

AuthFailedException in Storm while connecting remote non secure Kafka from Kerberized cluster

Explorer

Hi all,

We are using Storm to pull data from a remote Kafka cluster. Recently our cluster running Storm has been Kerberized. So now our Storm (0.10.0.2.4.3.2-1) is running on a secure environment and connecting Kafka 0.10.x on a non secure node. We have done the required changed to submit Storm job in Kerberized node but once submitted we are getting the below exception in Storm's worker log

2017-07-25 14:39:49.818 o.a.z.c.ZooKeeperSaslClient [ERROR] An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7))]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2017-07-25 14:39:49.819 o.a.z.ClientCnxn [ERROR] SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7))]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2017-07-25 14:39:49.819 o.a.c.ConnectionState [ERROR] Authentication failed
2017-07-25 14:39:49.822 b.s.util [ERROR] Async loop died!
java.lang.RuntimeException: java.lang.RuntimeException: org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /brokers/topics/test_topic/partitions
         at storm.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:82) ~[stormjar.jar:?]
         at storm.kafka.trident.ZkBrokerReader.<init>(ZkBrokerReader.java:42) ~[stormjar.jar:?]
         at storm.kafka.KafkaUtils.makeBrokerReader(KafkaUtils.java:57) ~[stormjar.jar:?]
         at storm.kafka.KafkaSpout.open(KafkaSpout.java:86) ~[stormjar.jar:?]
         at backtype.storm.daemon.executor$fn__5541$fn__5556.invoke(executor.clj:558) ~[storm-core-0.10.0.2.4.3.2-1.jar:0.10.0.2.4.3.2-1]
         at backtype.storm.util$async_loop$fn__545.invoke(util.clj:477) [storm-core-0.10.0.2.4.3.2-1.jar:0.10.0.2.4.3.2-1]
         at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
         at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]

To debug the issue we have created a simple topology with one Kafka Spout connected to a bolt which just prints the tuple. For KafkaSpout we have used the below protocol:

spoutConfig.securityProtocol=PLAINTEXTSASL

We have also tried using "PLAINTEXT" as security protocol but without any luck. Using the below maven dependency

<groupId>org.apache.storm</groupId>
<artifactId>storm-kafka</artifactId>
<version>0.10.0.2.4.3.2-1</version>
<groupId>org.apache.storm</groupId>
<artifactId>storm-core</artifactId>
<version>0.10.0.2.4.3.2-1</version>

Additionally one very strange thing is we were able to use zkCli.sh to connect to remote zk but post kerberization we need to run the below command (otherwise it shows AUTH_FAILED) :

export JVMFLAGS="-Djava.security.auth.login.config=/etc/zookeeper/conf/zookeeper_jaas.conf"

But there were no "zookeeper_jaas.conf" file inside that folder instead "zookeeper_client_jaas.conf" was found with the below content:

Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=false
useTicketCache=true;
};

But for any reason the export command works and allows us to connect to the remote zk. Did this multiple times with same result, not really sure why?

So at this are we missing anything here ? Within storm the only option to pass the security details inside spout using that "spoutConfig.securityProtocol" params which is not working here. Any help please ? Pleas let me know if any other information is required here.

5 REPLIES 5
Highlighted

Re: AuthFailedException in Storm while connecting remote non secure Kafka from Kerberized cluster

Contributor

@Amardeep Sarkar Based on your worker logs, you are not able to connect to zookeeper. You don't need to use spoutConfig.securityProtocol=PLAINTEXTSASL since that is to connect to Kafka SASL listener. It looks like your Kafka cluster is still not kerberized. So the default PLAINTEXT should work fine. You can check a few things here. Is your zookeeper also kerberized? If so, from your supervisor log you can check the java command that is used to start a worker and if it has -Djava.security.auth.login.config set correctly to a jaas config file. If so, does that file have a section named Client? That will be used by zookeeper client running in storm worker to connect to zookeeper. Note that if that section has an entry useTicketCache=true; it wont work. It needs a keytab. Can you check these things? If possible attach the file that is set as value for system property java.security.auth.login.config in your worker start java command

Highlighted

Re: AuthFailedException in Storm while connecting remote non secure Kafka from Kerberized cluster

Explorer

@pshah thank for your reply .

The cluster where Storm is running does have kerberized zookeeper, however the remote cluster that we are connecting does not have kerberized Zk and Kafka.

From Storm's supervisor.childopts params we do see that

-Djava.security.auth.login.config=/path/to/storm-nimbus/conf/storm_jaas.conf

And the client section looks like

Client {
   com.sun.security.auth.module.Krb5LoginModule required
   useKeyTab=true
   keyTab="/etc/security/keytabs/storm.headless.keytab"
   storeKey=true
   useTicketCache=false
   serviceName="zookeeper"
   principal="storm-user@COMPANY.DOMAIN.COM";
};

Please let me know your thoughts on this

Highlighted

Re: AuthFailedException in Storm while connecting remote non secure Kafka from Kerberized cluster

@Amardeep Sarkar were you able to find the solution? Even, I am facing the same issue.

Highlighted

Re: AuthFailedException in Storm while connecting remote non secure Kafka from Kerberized cluster

@pshah Can you please help? I am still facing the same issue. I also noticed one thing, when i try to connect zookeeper on kafka server from kerberized cluster without taking any ticket, I am able to connect, with below logs:

p.p1 {margin: 0.0px 0.0px 10.0px 0.0px; line-height: 18.0px; font: 15.0px Arial; color: #404041; -webkit-text-stroke: #404041} span.s1 {font-kerning: none}

WARN [main-SendThread(kafka01.nix.xyz.com:2181):ZooKeeperSaslClient$ClientCallbackHandler@496] - Could not login: the client is being asked for a password, but the Zookeeper client code does not currently support obtaining a password from the user. Make sure that the client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)' and restart the client. If you still get this message after that, the TGT in the ticket cache has expired and must be manually refreshed. To do so, first determine if you are using a password or a keytab. If the former, run kinit in a Unix shell in the environment of the user who is running this Zookeeper client using the command 'kinit <princ>' (where <princ> is the name of the client's Kerberos principal). If the latter, do 'kinit -k -t <keytab> <princ>' (where <princ> is the name of the Kerberos principal, and <keytab> is the location of the keytab file). After manually refreshing your cache, restart this client. If you continue to see this message after manually refreshing your cache, ensure that your KDC host's clock is in sync with this host's clock. 2017-10-23 15:56:40,982 - WARN [main-SendThread(kafka01.nix.xyz.com:2181):ClientCnxn$SendThread@1001] - SASL configuration failed: javax.security.auth.login.LoginException: No password provided Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. 2017-10-23 15:56:40,984 - INFO [main-SendThread(kafka01.nix.xyz.com:2181):ClientCnxn$SendThread@1019] - Opening socket connection to server kafka01.nix.xyz.com/10.72.19.66:2181

But when i take the zookeeper ticket on the same server, then it fails:

2017-10-23 16:01:47,877 - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=sta-needs01-kafka01 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@4534b60d Welcome to ZooKeeper! JLine support is enabled 2017-10-23 16:01:47,989 - INFO [main-SendThread(kafka01.nix.xyz.com:2181):Login@294] - successfully logged in. 2017-10-23 16:01:47,991 - INFO [Thread-0:Login$1@127] - TGT refresh thread started. 2017-10-23 16:01:47,995 - INFO [main-SendThread(kafka01.nix.xyz.com:2181):ZooKeeperSaslClient$1@289] - Client will use GSSAPI as SASL mechanism. 2017-10-23 16:01:48,013 - INFO [Thread-0:Login@302] - TGT valid starting at: Mon Oct 23 16:01:44 CEST 2017 2017-10-23 16:01:48,013 - INFO [Thread-0:Login@303] - TGT expires: Tue Oct 24 02:01:44 CEST 2017 2017-10-23 16:01:48,013 - INFO [Thread-0:Login$1@181] - TGT refresh sleeping until: Tue Oct 24 00:06:45 CEST 2017 [zk: sta-needs01-kafka01(CONNECTING) 0] 2017-10-23 16:01:48,052 - INFO [main-SendThread(kafka01.nix.xyz.com:2181):ClientCnxn$SendThread@1019] - Opening socket connection to server kafka01.nix.xyz.com/10.72.19.66:2181. Will attempt to SASL-authenticate using Login Context section 'Client' 2017-10-23 16:01:48,058 - INFO [main-SendThread(kafka01.nix.xyz.com:2181):ClientCnxn$SendThread@864] - Socket connection established to kafka01.nix.xyz.com/10.72.19.66:2181, initiating session 2017-10-23 16:01:48,067 - INFO [main-SendThread(kafka01.nix.xyz.com:2181):ClientCnxn$SendThread@1279] - Session establishment complete on server kafka01.nix.xyz.com/10.72.19.66:2181, sessionid = 0x15f496f70e70049, negotiated timeout = 30000 WATCHER:: WatchedEvent state:SyncConnected type:None path:null 2017-10-23 16:01:48,140 - ERROR [main-SendThread(kafka01.nix.xyz.com:2181):ZooKeeperSaslClient@388] - An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Message stream modified (41))]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state. 2017-10-23 16:01:48,140 - ERROR [main-SendThread(kafka01.nix.xyz.com:2181):ClientCnxn$SendThread@1059] - SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Message stream modified (41))]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state.

Highlighted

Re: AuthFailedException in Storm while connecting remote non secure Kafka from Kerberized cluster

Contributor

@Amardeep Sarkar @Saurabh Singh I am guessing you are using ZkHosts in your Kafka spout config when you build the topology. That is the main issue I think. Reason is ZkHosts also tries to read from Zk the metadata for Kafka brokers. However, Zk cluster for Kafka is unsecured. By default zookeeper client library uses Client section of the jaas config passed as a system property -Djava.security.auth.login.config anytime it tries to connect to Zk cluster.

Because storm and its zookeeper is secured you need that Client section in jaas config to successfully connect to that Zk cluster. However, ZkHosts uses zookeeper client in the same jvm process that also uses that Client section to connect to non kerberized Kafka cluster.

One solution to fix this problem is to use StaticHosts instead of ZkHosts in spout config for BorkerHosts interface implementation. However, static hosts is very "static" as the name suggests and will have to be reconfigured if the leaders for a topic partition change down the line,

Don't have an account?
Coming from Hortonworks? Activate your account here