Support Questions
Find answers, ask questions, and share your expertise

Help w/ Securing Storm in Ambari-managed Cluster

Explorer

I've read the documentation https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_secure-storm-ambari/content/ch_secure-st..., but am confused.

The section in the link above "Use an Existing Storm Node" mentions creating a .storm directory for each user, but doesn't say whether or not anything should be put in the new directory.

Also, the next step talks about Adding settings to /etc/storm/conf/storm.yaml, but it looks like all of those settings already exist, but 2 have different values:

settingcurrent valuedocs suggest
nimbus.thrift.port66276667
java.security.auth.login.config'/usr/hdp/current/storm-supervisor/conf/storm_jaas.conf'"/etc/storm/conf/client_jaas.conf"

It's unclear to me whether or not I should change the settings to match what the docs suggest. The port difference could just be how it happens to be set on my cluster, or it could be that a different port is used by the client vs. storm processes communicating with each other. If I make the java.security change, with this node only be usable as a client?

9 REPLIES 9

Rising Star

Client jaas file is only used by Storm client not Nimbus or the Supervisor. The ports should be matching the configured nimbus port.

Explorer

Based on additional digging since I posted this question, it looks like the HWX docs are wrong to refer to port 6667, it appears 6627 is the default and what is currently set on my cluster.

I'm still fuzzy on whether I should change java.security.auth.login.config in storm.yaml to point to client_jaas.conf rather than storm_jaas.conf, and whether that would mean the node where this change is made would ONLY function as a client.

Rising Star

You are right, seems like the documentation is incorrect; 6667 is default port for Kafka, 6627 is for Storm.

Changing storm.yaml file will impact anything else (Storm) running on the node.

I would recommend trying a manual override first. You can do so by running storm <command> -c java.security.auth.login.config=<file path>

This way you can avoid changing the configuration of the node and validate that this change works. (don't forget to manually Kinit)

@Vincent Romeo is the Storm client currently not able to connect to nimbus? HDP Kerberos wizard takes care of kerberizing both client and server and updating settings accordingly.

Explorer

I was suspecting configs were not correct because trying to run hortonworks' WordCountTopology sample was not working. Here is what I was seeing:

-bash-4.1$ storm jar storm-starter-0.0.1-storm-0.9.0.1.jar storm.starter.WordCountTopology WordCount

Running: /opt/java/hotspot/7/64_bit/jdk1.7.0_79/bin/java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdp/2.4.2.0-258/storm -Dstorm.log.dir=/var/hadoop/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -cp /usr/hdp/2.4.2.0-258/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/hadoop-auth-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.2.0-258/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.2.0-258/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.2.0-258/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.2.0-258/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.2.0-258/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.2.0-258/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.2.0-258/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.2.0-258/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.2.0-258/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.2.0-258/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.2.0-258/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.2.0-258/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.2.0-258/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.2.0-258/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.2.0-258/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.2.0-258/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.2.0-258/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.2.0-258/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.2.0-258/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/asm-4.0.jar:/usr/hdp/2.4.2.0-258/storm/lib/zookeeper.jar:/usr/hdp/2.4.2.0-258/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/storm-core-0.10.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.2.0-258/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.2.0-258/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.2.0-258/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.2.0-258/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.2.0-258/storm/lib/clj-time-0.8.0.jar:storm-starter-0.0.1-storm-0.9.0.1.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/2.4.2.0-258/storm/bin -Dstorm.jar=storm-starter-0.0.1-storm-0.9.0.1.jar storm.starter.WordCountTopology WordCount 18:49:19.481 [main] INFO b.s.u.Utils - Using defaults.yaml from resources 18:49:19.551 [main] INFO b.s.u.Utils - Using storm.yaml from resources 18:49:19.611 [main] INFO b.s.u.Utils - Using defaults.yaml from resources 18:49:19.631 [main] INFO b.s.u.Utils - Using storm.yaml from resources 18:49:19.648 [main] INFO b.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -6595191808170807148:-7705041539986139533 18:49:19.649 [main] INFO b.s.s.a.AuthUtils - Got AutoCreds [] 18:49:19.664 [main] INFO b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [2000] the maxSleepTimeMs [60000] the maxRetries [5] 18:49:19.723 [main] WARN b.s.s.a.k.ClientCallbackHandler - Could not login: the client is being asked for a password, but the client code does not currently support obtaining a password from the user. Make sure that the client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)' and restart the client. If you still get this message after that, the TGT in the ticket cache has expired and must be manually refreshed. To do so, first determine if you are using a password or a keytab. If the former, run kinit in a Unix shell in the environment of the user who is running this client using the command 'kinit <princ>' (where <princ> is the name of the client's Kerberos principal). If the latter, do 'kinit -k -t <keytab> <princ>' (where <princ> is the name of the Kerberos principal, and <keytab> is the location of the keytab file). After manually refreshing your cache, restart this client. If you continue to see this message after manually refreshing your cache, ensure that your KDC host's clock is in sync with this host's clock. 18:49:19.725 [main] ERROR b.s.s.a.k.KerberosSaslTransportPlugin - Server failed to login in principal:javax.security.auth.login.LoginException: No password provided javax.security.auth.login.LoginException: No password provided

....

Now that I've tried passing the path to the client_jaas.conf on the command line, it seems things get a little further, and there is a different error:

-bash-4.1$ storm jar storm-starter-0.0.1-storm-0.9.0.1.jar storm.starter.WordCountTopology WordCount -c java.security.auth.login.config=/etc/storm/conf/client_jaas.conf

Running: /opt/java/hotspot/7/64_bit/jdk1.7.0_79/bin/java -client -Ddaemon.name= -Dstorm.options=java.security.auth.login.config%3D%2Fetc%2Fstorm%2Fconf%2Fclient_jaas.conf -Dstorm.home=/usr/hdp/2.4.2.0-258/storm -Dstorm.log.dir=/var/hadoop/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -cp /usr/hdp/2.4.2.0-258/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/hadoop-auth-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.2.0-258/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.2.0-258/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.2.0-258/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.2.0-258/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.2.0-258/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.2.0-258/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.2.0-258/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.2.0-258/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.2.0-258/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.2.0-258/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.2.0-258/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.2.0-258/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.2.0-258/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.2.0-258/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.2.0-258/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.2.0-258/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.2.0-258/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.2.0-258/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.2.0-258/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/asm-4.0.jar:/usr/hdp/2.4.2.0-258/storm/lib/zookeeper.jar:/usr/hdp/2.4.2.0-258/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/storm-core-0.10.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.2.0-258/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.2.0-258/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.2.0-258/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.2.0-258/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.2.0-258/storm/lib/clj-time-0.8.0.jar:storm-starter-0.0.1-storm-0.9.0.1.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/2.4.2.0-258/storm/bin -Dstorm.jar=storm-starter-0.0.1-storm-0.9.0.1.jar storm.starter.WordCountTopology WordCount 18:54:08.293 [main] INFO b.s.u.Utils - Using defaults.yaml from resources 18:54:08.362 [main] INFO b.s.u.Utils - Using storm.yaml from resources 18:54:08.421 [main] INFO b.s.u.Utils - Using defaults.yaml from resources 18:54:08.441 [main] INFO b.s.u.Utils - Using storm.yaml from resources 18:54:08.458 [main] INFO b.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -6645485375203566088:-8607446551035289369 18:54:08.459 [main] INFO b.s.s.a.AuthUtils - Got AutoCreds [] 18:54:08.474 [main] INFO b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [2000] the maxSleepTimeMs [60000] the maxRetries [5] 18:54:08.532 [main] INFO o.a.s.z.Login - successfully logged in. 18:54:08.782 [main] INFO b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [2000] the maxSleepTimeMs [60000] the maxRetries [5] 18:54:08.783 [main] INFO o.a.s.z.Login - successfully logged in. 18:54:08.895 [main] INFO b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [2000] the maxSleepTimeMs [60000] the maxRetries [5] 18:54:08.896 [main] INFO o.a.s.z.Login - successfully logged in. 18:54:09.015 [main] INFO b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [2000] the maxSleepTimeMs [60000] the maxRetries [5] 18:54:09.017 [main] INFO o.a.s.z.Login - successfully logged in. 18:54:09.137 [main] INFO b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [2000] the maxSleepTimeMs [60000] the maxRetries [5] 18:54:09.138 [main] INFO o.a.s.z.Login - successfully logged in. 18:54:09.261 [main] INFO b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [2000] the maxSleepTimeMs [60000] the maxRetries [5] 18:54:09.262 [main] INFO o.a.s.z.Login - successfully logged in. Exception in thread "main" java.lang.RuntimeException: AuthorizationException(msg:fileUpload is not authorized) at backtype.storm.StormSubmitter.submitJarAs(StormSubmitter.java:399) at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:229) at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271) at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157) at storm.starter.WordCountTopology.main(WordCountTopology.java:77) Caused by: AuthorizationException(msg:fileUpload is not authorized) at backtype.storm.generated.Nimbus$beginFileUpload_result$beginFileUpload_resultStandardScheme.read(Nimbus.java:13616) at backtype.storm.generated.Nimbus$beginFileUpload_result$beginFileUpload_resultStandardScheme.read(Nimbus.java:13594) at backtype.storm.generated.Nimbus$beginFileUpload_result.read(Nimbus.java:13536) at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:78) at backtype.storm.generated.Nimbus$Client.recv_beginFileUpload(Nimbus.java:462) at backtype.storm.generated.Nimbus$Client.beginFileUpload(Nimbus.java:450) at backtype.storm.StormSubmitter.submitJarAs(StormSubmitter.java:370) ... 4 more

Rising Star

Authorization problem can be resolved by adding the users to Storm using these two properties and then restarting Storm Nimbus.

nimbus.supervisor.users and nimbus.admins

The account you are using doesn't have permissions to deploy the topology.

Explorer

nimbus.supervisor.users and nimbus.admins need to be added manually even when Ranger is being used?

I'm in a group that has the following Permissions: Submit Topology, File Upload, Get Nimbus Conf, Get Cluster Info, File Download, Kill Topology, Rebalance, Activate, Deactivate, Get Topology Conf, Get Topology, Get User Topology, Get Topology Info. And 'Delegate Admin' is checked.

Rising Star

Not if Ranger - Storm integration is in place. Please validate that through ambari, check configuration for Storm containing com.xasecure.authorization.storm.authorizer.XaSecureStormAuthorizer

Explorer

Using HDP 2.4.2, Ambari 2.2.2.0, I see

nimbus.authorizer: org.apache.ranger.authorization.storm.authorizer.RangerStormAuthorizer

Rising Star

Please double check that the kerberos user has access in Ranger.

If it's still not resolved, I would recommend checking with Hortonworks Support if there is a configuration or integration issue.