About Aaron_Dossett

Aaron_Dossett · ‎05-14-2018

To keep us on our toes!

Aaron_Dossett · ‎02-22-2017

I recently had a hive query (executed via a JDBC connection) hang for several hours due to a metastore issue. Is there a way that I set a maximum query time with my JDBC connection? This query should take just a few seconds, so I'd rather kill it after a few minutes than let it block for hours. Some searching of the Hive JIRA suggests that this isn't possible, but asking here if anyone has advice.

Aaron_Dossett · ‎01-17-2017

@Jasper I think it's correct to say that actions added by .addRotationAction are AT MOST ONCE, even for Trident. I'm not sure using a sync policy would work either, I believe those guarantee a minimum sync frequency but additional syncs are still possible (I could be wrong, I haven't looked at that code in a while). Said another way, you want to treat the unit of work as an HDFS file, but in Storm itself the unit of work is a tuple. Thus it will be hard to meet your goal in a clean way.

Aaron_Dossett · ‎01-12-2017

This is right, but I am VERY wary about using `RotationAction`s because if a worker crashes and restarts the action will never be performed. For that reason I would never use a `RotationAction` in production.

Aaron_Dossett · ‎01-12-2017

Hi Jasper -- No, you can't tell just by the filename. When I had to solve that same problem I ended up using the HDFS API to test whether or not the file was open. This wasn't too painful since the application I wrote was already using the Java API.

Aaron_Dossett · ‎04-05-2016

Thanks @Kuldeep Kulkarni. Since the cluster I'm on is shared by many users for many different workloads I won't be able to make that change easily. Are there oozie JIRAs regarding this change I could read or was this specific to HWX packaging? I'd like to learn more about it since it seems very disruptive to java actions, e.g. having a 4+ year old version of guava in the classpath. Thanks!

Aaron_Dossett · ‎04-04-2016

Thanks @Artem Ervits and @Kuldeep Kulkarni. Yes, I can definitely control the versions that my java application includes. The problem is that when the java action runs, the jar files that I created are in the "current directory" of the java action along with the other versions that oozie share lib is including. These jar files have the same classpath precedence so it appears to be random which jar is picked up. For example, com.fasterxml.jackson.core.JsonFactory might be found in my own jar or in jackson-core-2.2.3.jar provided by oozie.

Aaron_Dossett · ‎04-04-2016

Thank you @Kuldeep Kulkarni! It's causing me some problems because files like jackson-*.jar and guava appear early in the classpath (the current directory of the oozie action) and late in the classpath (mr-framework/hadoop/share/hadoop/common/lib/). I would effectively have to pin the dependencies of my java action to those precise versions. Any way around that?

Aaron_Dossett · ‎04-04-2016

Hi, we recently upgraded from HDP 2.2.X to 2.3.4 and I notice that /user/oozie/share/lib/oozie contains a lot more jar files now. See below. The presence of these jar files is puzzling since they are otherwise in the classpath of, say, java oozie actions. e.g. in directories like `mr-framework/hadoop/share/hadoop/common/lib/` Does anyone know 1) why these jars in share/lib/oozie and 2) if they can be safely removed? Thanks! 2.3.4: -rw-r--r-- 3 oozie hadoop 11948376 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/aws-java-sdk-1.7.4.jar -rw-r--r-- 3 oozie hadoop 666763 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/azure-storage-2.2.0.jar -rw-r--r-- 3 oozie hadoop 412739 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/commons-lang3-3.3.2.jar -rw-r--r-- 3 oozie hadoop 1648200 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/guava-11.0.2.jar -rw-r--r-- 3 oozie hadoop 103052 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/hadoop-aws-2.7.1.2.3.4.0-3485.jar -rw-r--r-- 3 oozie hadoop 137439 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/hadoop-azure-2.7.1.2.3.4.0-3485.jar -rw-r--r-- 3 oozie hadoop 33483 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/jackson-annotations-2.2.3.jar -rw-r--r-- 3 oozie hadoop 192699 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/jackson-core-2.2.3.jar -rw-r--r-- 3 oozie hadoop 865838 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/jackson-databind-2.2.3.jar -rw-r--r-- 3 oozie hadoop 570478 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/joda-time-2.1.jar -rw-r--r-- 3 oozie hadoop 16046 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/json-simple-1.1.jar -rw-r--r-- 3 oozie hadoop 12294 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/oozie-hadoop-utils-hadoop-2-4.2.0.2.3.4.0-3485.jar -rw-r--r-- 3 oozie hadoop 49474 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/oozie-sharelib-oozie-4.2.0.2.3.4.0-3485.jar 2.2 -rw-r--r-- 3 hdfs hadoop 7832 2015-09-02 12:57 /user/oozie/share/lib/oozie/core-site.xml -rw-r--r-- 3 hdfs hadoop 9517 2015-09-02 12:57 /user/oozie/share/lib/oozie/hdfs-site.xml -rwxr-xr-x 3 oozie hadoop 16046 2015-06-17 23:08 /user/oozie/share/lib/oozie/json-simple-1.1.jar -rw-r--r-- 3 hdfs hadoop 6671 2015-09-02 12:57 /user/oozie/share/lib/oozie/mapred-site.xml -rwxr-xr-x 3 oozie hadoop 3242 2015-06-17 23:08 /user/oozie/share/lib/oozie/oozie-hadoop-utils-2.6.0.2.2.4.12-1.oozie-4.1.0.2.2.4.12-1.jar -rwxr-xr-x 3 oozie hadoop 41903 2015-06-17 23:08 /user/oozie/share/lib/oozie/oozie-sharelib-oozie-4.1.0.2.2.4.12-1.jar -rw-r--r-- 3 hdfs hadoop 18164 2015-11-04 18:55 /user/oozie/share/lib/oozie/yarn-site.xml

Aaron_Dossett · ‎03-11-2016

I should have done a bit more research... this is a problem in Storm 0.10, but it looks like the upcoming Storm 1.0 release will handle this situation by allowing properties to be passed directly to the Kafka Bolt instead of indirectly through the Topology configuration: https://github.com/apache/storm/blob/master/external/storm-kafka/src/jvm/org/apache/storm/kafka/bolt/KafkaBolt.java#L87-L90

Online	Offline
Last Visited	‎05-14-2018 09:23 PM

Member Since	‎11-30-2015 11:16 PM
Last Visited	‎05-14-2018 09:23 PM
Posts	39
Kudos received	23

Cloudera Community

Re: How can I specify a Java class in the kafka bo...

Re: Can I ensure that my own jars have classpath p...

Re: How can I pass "-D" arguments to Oozie java ac...

Re: Why does Storm fail with this deserialization ...

Can I set a maximum query time with Hive JDBC?

Re: Storm HDFS Bolt question (Trident api)

Re: Storm HDFS Bolt question (Trident api)

Re: Storm HDFS Bolt question (Trident api)

Re: oozie share lib for oozie contains a lot of ja...

Re: oozie share lib for oozie contains a lot of ja...

Re: oozie share lib for oozie contains a lot of ja...

oozie share lib for oozie contains a lot of jar fi...

Re: How can I specify a Java class in the kafka bo...