Authentication and Authorization errors on simple Storm topology submission in a Kerberized cluster

[#1]

I have a newly created HDP 2.5.3 cluster with Kerberos enabled that I'm having trouble getting a simple Storm topology submitted. I do NOT have Ranger installed. I'm following the validation instructions at the bottom of http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_command-line-installation/content/validat... to run the included simple WordCount topology which reads as the following.

storm jar /usr/hdp/current/storm-client/contrib/storm-starter/storm-starter-topologies-*.jar org.apache.storm.starter.WordCountTopology wordcount

I tried this two different ways with two different results.

** FIRST ATTEMPT ** (the authentication problem!!)

I created a Kerberos ticket for one of my users, student2, as shown below.

[student2@ip-172-30-0-42 target]$ klist
Ticket cache: FILE:/tmp/krb5cc_432201241
Default principal: student2@LAB.HORTONWORKS.NET
Valid starting       Expires              Service principal
03/07/2017 02:57:33  03/07/2017 12:57:33  krbtgt/LAB.HORTONWORKS.NET@LAB.HORTONWORKS.NET
         renew until 03/14/2017 02:57:29

Then I run the earlier topology submission command and get the following excerpt (full output at student2.txt).

976  [main] INFO  o.a.s.s.a.AuthUtils - Got AutoCreds []
1001 [main] WARN  o.a.s.s.a.k.ClientCallbackHandler - Could not login: the client is being asked for a password, but the  client code does not currently support obtaining a password from the user. Make sure that the client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)' and restart the client. If you still get this message after that, the TGT in the ticket cache has expired and must be manually refreshed. To do so, first determine if you are using a password or a keytab. If the former, run kinit in a Unix shell in the environment of the user who is running this client using the command 'kinit <princ>' (where <princ> is the name of the client's Kerberos principal). If the latter, do 'kinit -k -t <keytab> <princ>' (where <princ> is the name of the Kerberos principal, and <keytab> is the location of the keytab file). After manually refreshing your cache, restart this client. If you continue to see this message after manually refreshing your cache, ensure that your KDC host's clock is in sync with this host's clock.
1002 [main] ERROR o.a.s.s.a.k.KerberosSaslTransportPlugin - Server failed to login in principal:javax.security.auth.login.LoginException: No password provided
javax.security.auth.login.LoginException: No password provided
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:919) ~[?:1.8.0_121]
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760) ~[?:1.8.0_121]

To me... this looks like student2's kerb ticket is not making the journey and thus, the authentication exception is being thrown.

QUESTION: Is there anything special I need to be doing in order to have the ticket be leveraged at submission time?

** SECOND ATTEMPT ** (the authorization problem!!)

I then figured I'd try to run the command again, but this time with a valid ticket for the storm user thinking that its God-like powers should persevere.

[root@ip-172-30-0-42 simplestorm]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: storm-telus_training@LAB.HORTONWORKS.NET
Valid starting       Expires              Service principal
03/07/2017 03:37:16  03/07/2017 13:37:16  krbtgt/LAB.HORTONWORKS.NET@LAB.HORTONWORKS.NET
         renew until 03/14/2017 03:37:16

I submitted the WC topology again and this time got this excerpt (full output at storm.txt).

2269 [main] INFO  o.a.s.StormSubmitter - Successfully uploaded topology jar to assigned location: /hadoop/storm/nimbus/inbox/stormjar-cac76801-cea6-4c4e-9420-44d69bd7cb9b.jar
2278 [main] INFO  o.a.s.m.n.Login - successfully logged in.
2302 [main] INFO  o.a.s.m.n.Login - successfully logged in.
2310 [main] INFO  o.a.s.StormSubmitter - Submitting topology wordcount in distributed mode with conf {"storm.zookeeper.topology.auth.scheme":"digest","storm.zookeeper.topology.auth.payload":"-5661685876145720659:-8904469779744658388","topology.workers":3,"topology.debug":true}
Exception in thread "main" java.lang.RuntimeException: AuthorizationException(msg:wordcount-2-1488857970-stormconf.ser does not appear to be a valid blob key)
at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:255)
at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:310)

To me... it looks like I got hung up on an authorization problem this time (which probably answers my earlier question about if anything special is needed for the kerb ticket to be passed along) although I'm not sure what that "does not appear to be a valid blob key" message is saying.

QUESTION: What settings do I need to check in Ambari that would tell Storm to allow all secured users to be able to submit a topology? << reminder; I do NOT have Ranger installed

Any assistance, even a hint, would be greatly appreciated!!

[#2 · Accepted solution]

@Lester Martin

To allow user to deploy a storm topology in secure mode

1. create user level storm.yaml under /home/username/.storm/storm.yaml . This storm.yaml can only contain few required settings or you can just copy the entire storm.yaml from /etc/storm/conf/storm.yaml

required settings if you are adding are

1. nimbus.seeds

2. storm.thrift.transport: "org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin"

3. java.security.auth.login.config: "/etc/storm/conf/client_jaas.conf"

4. Make sure /etc/storm/conf/client_jaas.conf contents looks like this

StormClient {
   com.sun.security.auth.module.Krb5LoginModule required
   doNotPrompt=false
   useTicketCache=true
   serviceName="nimbus";
};

5. Run kinit with user's keytab before running storm jar command

By default Storm uses SimpleACLAuthorizer as authorizer to give permissions to user. This will allow all user's with valid keytab to submit topologies.

[#3]

Beautiful!! This fixes me!! Especially, when I noticed the storm.yaml goes in a .storm file in the home dir (I didn't create that subdir initially). The only thing I did different is leave the default /etc/storm/conf/client_jaas.conf which reads as the following.

StormClient {
  com.sun.security.auth.module.Krb5LoginModule required
  useTicketCache=true
  renewTicket=true
  serviceName="nimbus";
};

1,000,000 thanks!! A shout out to @Vipin Rathor for the second set of eyes!!

[#4]

If i am setting up the entire storm cluster on one node only, my jaas.conf lookslikethe one mentioned in the apache docs, where can i intergrate point number 4? If i append it in the existing jaas.conf, it gives error when starting nimbus daemon.

[#5]

Hi,

I have got 3 node cluster running kerberized hdp 2.6.2 with Ranger but without Ranger Storm plugin.

I also see the errors when I try to run command "storm list".

Storm sevice check runs fine. I get the following error when I use underprivileged user account with a valid token.

Any clues most apprecieted.

2290 [main] WARN  o.a.s.s.a.k.ClientCallbackHandler - Could not login: the client is being asked for a password, but the  client code does not currently support obtaining a password from the user. Make sure that the client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)' and restart the client. If you still get this message after that, the TGT in the ticket cache has expired and must be manually refreshed. To do so, first determine if you are using a password or a keytab. If the former, run kinit in a Unix shell in the environment of the user who is running this client using the command 'kinit <princ>' (where <princ> is the name of the client's Kerberos principal). If the latter, do 'kinit -k -t <keytab> <princ>' (where <princ> is the name of the Kerberos principal, and <keytab> is the location of the keytab file). After manually refreshing your cache, restart this client. If you continue to see this message after manually refreshing your cache, ensure that your KDC host's clock is in sync with this host's clock.
2298 [main] ERROR o.a.s.s.a.k.KerberosSaslTransportPlugin - Server failed to login in principal:javax.security.auth.login.LoginException: No password provided
javax.security.auth.login.LoginException: No password provided
        at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:919) ~[?:1.8.0_112]
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760) ~[?:1.8.0_112]
        at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617) ~[?:1.8.0_112]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]