It's been a while since this question was asked. Does Knox now "play nicer" with Kafka? Regardless of volume, it would sure be nice to insulate sources that want to push data to the HDP/HDF ecosystem from kerberos. ... View more

Anyone find an answer to this? We're going to need to do something similar.... ... View more

In a nutshell, few were using it, and it really was just a bit easier-to-use skin over other tools (primarily Oozie); and as they attempted to add features, it was looking more like the underlying tools, so.... It will be interesting to see how the marketing/messaging shifts over the coming months. This related question and answer seems to suggest Atlas isn't going anywhere, and it seems Atlas and Oozie were the primary underpinnings of Falcon (I think): https://community.hortonworks.com/questions/97570/apache-falcon-in-hdp-30.html ... View more

I'll reach out to my account team, but it still seems odd to announce a component as deprecated when you can't point to an announced product that will be taking on its functionality. The whole point of deprecating is normally to give folks a heads-up they should start moving to the replacement.... ... View more

@ibalaji9, have you found a solution for redirecting the sqoop working/scratch directory? We're running into issues where users moving a lot of data w/ sqoop are bumping into personal quotas in their /user/ directory even though there is plenty of space in /tmp and in the destination folder where they're trying to land data. Thx, -Vince ... View more

hive.exec.scratchdir on this cluster is /tmp/hive. Don't know why the user appears to be exceeding quota on a personal directory. ... View more

Using HDP 2.4.2, Ambari 2.2.2.0, I see nimbus.authorizer: org.apache.ranger.authorization.storm.authorizer.RangerStormAuthorizer ... View more

nimbus.supervisor.users and nimbus.admins need to be added manually even when Ranger is being used? I'm in a group that has the following Permissions: Submit Topology, File Upload, Get Nimbus Conf, Get Cluster Info, File Download, Kill Topology, Rebalance, Activate, Deactivate, Get Topology Conf, Get Topology, Get User Topology, Get Topology Info. And 'Delegate Admin' is checked. ... View more

I was suspecting configs were not correct because trying to run hortonworks' WordCountTopology sample was not working. Here is what I was seeing: -bash-4.1$ storm jar storm-starter-0.0.1-storm-0.9.0.1.jar storm.starter.WordCountTopology WordCount Running: /opt/java/hotspot/7/64_bit/jdk1.7.0_79/bin/java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdp/2.4.2.0-258/storm -Dstorm.log.dir=/var/hadoop/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -cp /usr/hdp/2.4.2.0-258/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/hadoop-auth-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.2.0-258/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.2.0-258/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.2.0-258/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.2.0-258/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.2.0-258/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.2.0-258/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.2.0-258/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.2.0-258/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.2.0-258/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.2.0-258/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.2.0-258/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.2.0-258/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.2.0-258/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.2.0-258/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.2.0-258/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.2.0-258/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.2.0-258/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.2.0-258/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.2.0-258/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/asm-4.0.jar:/usr/hdp/2.4.2.0-258/storm/lib/zookeeper.jar:/usr/hdp/2.4.2.0-258/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/storm-core-0.10.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.2.0-258/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.2.0-258/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.2.0-258/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.2.0-258/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.2.0-258/storm/lib/clj-time-0.8.0.jar:storm-starter-0.0.1-storm-0.9.0.1.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/2.4.2.0-258/storm/bin -Dstorm.jar=storm-starter-0.0.1-storm-0.9.0.1.jar storm.starter.WordCountTopology WordCount 18:49:19.481 [main] INFO b.s.u.Utils - Using defaults.yaml from resources 18:49:19.551 [main] INFO b.s.u.Utils - Using storm.yaml from resources 18:49:19.611 [main] INFO b.s.u.Utils - Using defaults.yaml from resources 18:49:19.631 [main] INFO b.s.u.Utils - Using storm.yaml from resources 18:49:19.648 [main] INFO b.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -6595191808170807148:-7705041539986139533 18:49:19.649 [main] INFO b.s.s.a.AuthUtils - Got AutoCreds [] 18:49:19.664 [main] INFO b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [2000] the maxSleepTimeMs [60000] the maxRetries [5] 18:49:19.723 [main] WARN b.s.s.a.k.ClientCallbackHandler - Could not login: the client is being asked for a password, but the client code does not currently support obtaining a password from the user. Make sure that the client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)' and restart the client. If you still get this message after that, the TGT in the ticket cache has expired and must be manually refreshed. To do so, first determine if you are using a password or a keytab. If the former, run kinit in a Unix shell in the environment of the user who is running this client using the command 'kinit <princ>' (where <princ> is the name of the client's Kerberos principal). If the latter, do 'kinit -k -t <keytab> <princ>' (where <princ> is the name of the Kerberos principal, and <keytab> is the location of the keytab file). After manually refreshing your cache, restart this client. If you continue to see this message after manually refreshing your cache, ensure that your KDC host's clock is in sync with this host's clock. 18:49:19.725 [main] ERROR b.s.s.a.k.KerberosSaslTransportPlugin - Server failed to login in principal:javax.security.auth.login.LoginException: No password provided javax.security.auth.login.LoginException: No password provided .... Now that I've tried passing the path to the client_jaas.conf on the command line, it seems things get a little further, and there is a different error: -bash-4.1$ storm jar storm-starter-0.0.1-storm-0.9.0.1.jar storm.starter.WordCountTopology WordCount -c java.security.auth.login.config=/etc/storm/conf/client_jaas.conf Running: /opt/java/hotspot/7/64_bit/jdk1.7.0_79/bin/java -client -Ddaemon.name= -Dstorm.options=java.security.auth.login.config%3D%2Fetc%2Fstorm%2Fconf%2Fclient_jaas.conf -Dstorm.home=/usr/hdp/2.4.2.0-258/storm -Dstorm.log.dir=/var/hadoop/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -cp /usr/hdp/2.4.2.0-258/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/hadoop-auth-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.2.0-258/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.2.0-258/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.2.0-258/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.2.0-258/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.2.0-258/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.2.0-258/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.2.0-258/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.2.0-258/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.2.0-258/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.2.0-258/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.2.0-258/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.2.0-258/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.2.0-258/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.2.0-258/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.2.0-258/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.2.0-258/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.2.0-258/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.2.0-258/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.2.0-258/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/asm-4.0.jar:/usr/hdp/2.4.2.0-258/storm/lib/zookeeper.jar:/usr/hdp/2.4.2.0-258/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/storm-core-0.10.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.2.0-258/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.2.0-258/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.2.0-258/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.2.0-258/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.2.0-258/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.2.0-258/storm/lib/clj-time-0.8.0.jar:storm-starter-0.0.1-storm-0.9.0.1.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/2.4.2.0-258/storm/bin -Dstorm.jar=storm-starter-0.0.1-storm-0.9.0.1.jar storm.starter.WordCountTopology WordCount 18:54:08.293 [main] INFO b.s.u.Utils - Using defaults.yaml from resources 18:54:08.362 [main] INFO b.s.u.Utils - Using storm.yaml from resources 18:54:08.421 [main] INFO b.s.u.Utils - Using defaults.yaml from resources 18:54:08.441 [main] INFO b.s.u.Utils - Using storm.yaml from resources 18:54:08.458 [main] INFO b.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -6645485375203566088:-8607446551035289369 18:54:08.459 [main] INFO b.s.s.a.AuthUtils - Got AutoCreds [] 18:54:08.474 [main] INFO b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [2000] the maxSleepTimeMs [60000] the maxRetries [5] 18:54:08.532 [main] INFO o.a.s.z.Login - successfully logged in. 18:54:08.782 [main] INFO b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [2000] the maxSleepTimeMs [60000] the maxRetries [5] 18:54:08.783 [main] INFO o.a.s.z.Login - successfully logged in. 18:54:08.895 [main] INFO b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [2000] the maxSleepTimeMs [60000] the maxRetries [5] 18:54:08.896 [main] INFO o.a.s.z.Login - successfully logged in. 18:54:09.015 [main] INFO b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [2000] the maxSleepTimeMs [60000] the maxRetries [5] 18:54:09.017 [main] INFO o.a.s.z.Login - successfully logged in. 18:54:09.137 [main] INFO b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [2000] the maxSleepTimeMs [60000] the maxRetries [5] 18:54:09.138 [main] INFO o.a.s.z.Login - successfully logged in. 18:54:09.261 [main] INFO b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [2000] the maxSleepTimeMs [60000] the maxRetries [5] 18:54:09.262 [main] INFO o.a.s.z.Login - successfully logged in. Exception in thread "main" java.lang.RuntimeException: AuthorizationException(msg:fileUpload is not authorized) at backtype.storm.StormSubmitter.submitJarAs(StormSubmitter.java:399) at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:229) at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271) at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157) at storm.starter.WordCountTopology.main(WordCountTopology.java:77) Caused by: AuthorizationException(msg:fileUpload is not authorized) at backtype.storm.generated.Nimbus$beginFileUpload_result$beginFileUpload_resultStandardScheme.read(Nimbus.java:13616) at backtype.storm.generated.Nimbus$beginFileUpload_result$beginFileUpload_resultStandardScheme.read(Nimbus.java:13594) at backtype.storm.generated.Nimbus$beginFileUpload_result.read(Nimbus.java:13536) at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:78) at backtype.storm.generated.Nimbus$Client.recv_beginFileUpload(Nimbus.java:462) at backtype.storm.generated.Nimbus$Client.beginFileUpload(Nimbus.java:450) at backtype.storm.StormSubmitter.submitJarAs(StormSubmitter.java:370) ... 4 more ... View more

Based on additional digging since I posted this question, it looks like the HWX docs are wrong to refer to port 6667, it appears 6627 is the default and what is currently set on my cluster. I'm still fuzzy on whether I should change java.security.auth.login.config in storm.yaml to point to client_jaas.conf rather than storm_jaas.conf, and whether that would mean the node where this change is made would ONLY function as a client. ... View more

I think @Eric Brosch's question is around multi-tenancy... I found the following link, but none of the answers really get to the details of running topologies in an enterprise multi-tenant environment: https://community.hortonworks.com/questions/1705/storm-multi-tenancy-best-practices.html The primary recommendations seem to be that one must 1. have a secure cluster and 2. set supervisor.worker.run.as.user to true. In the docs I've seen, it's not clear whether there's a good way to have groups of users where they can manage topologies within the group, but not mess with topologies belonging to another group. ... View more

I thought those 2 settings pre-dated the introduction of ACID tables. I can understand the "External tables cannot be ACID tables..." part, but I would think those settings could be used to allow users to issue an "exclusive lock" on an external table to prevent reading from it thru hive while external jobs manipulate the underlying files.... ... View more

From what I'm hearing from other sources this answer was inaccurate and totally fails to take into consideration how our cluster is being used. I disagree with it being tagged "best answer". ... View more

Appreciate the quick response, but if that jira is out there, I'm not seeing it. ... View more

@Vamsi Jonnadula, Did this get resolved? We're seeing issues with small ORC files within a partition as well. ... View more