Member since
11-30-2015
39
Posts
23
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1006 | 03-11-2016 07:31 PM | |
5651 | 12-17-2015 12:33 AM | |
1222 | 12-16-2015 10:46 PM |
05-14-2018
09:23 PM
To keep us on our toes!
... View more
02-22-2017
05:07 PM
1 Kudo
I recently had a hive query (executed via a JDBC connection) hang for several hours due to a metastore issue. Is there a way that I set a maximum query time with my JDBC connection? This query should take just a few seconds, so I'd rather kill it after a few minutes than let it block for hours. Some searching of the Hive JIRA suggests that this isn't possible, but asking here if anyone has advice.
... View more
Labels:
- Labels:
-
Apache Hive
01-17-2017
08:44 PM
@Jasper I think it's correct to say that actions added by .addRotationAction are AT MOST ONCE, even for Trident. I'm not sure using a sync policy would work either, I believe those guarantee a minimum sync frequency but additional syncs are still possible (I could be wrong, I haven't looked at that code in a while). Said another way, you want to treat the unit of work as an HDFS file, but in Storm itself the unit of work is a tuple. Thus it will be hard to meet your goal in a clean way.
... View more
01-12-2017
10:16 PM
This is right, but I am VERY wary about using `RotationAction`s because if a worker crashes and restarts the action will never be performed. For that reason I would never use a `RotationAction` in production.
... View more
01-12-2017
10:14 PM
Hi Jasper -- No, you can't tell just by the filename. When I had to solve that same problem I ended up using the HDFS API to test whether or not the file was open. This wasn't too painful since the application I wrote was already using the Java API.
... View more
04-05-2016
12:57 PM
Thanks @Kuldeep Kulkarni. Since the cluster I'm on is shared by many users for many different workloads I won't be able to make that change easily. Are there oozie JIRAs regarding this change I could read or was this specific to HWX packaging? I'd like to learn more about it since it seems very disruptive to java actions, e.g. having a 4+ year old version of guava in the classpath. Thanks!
... View more
04-04-2016
09:11 PM
Thanks @Artem Ervits and @Kuldeep Kulkarni. Yes, I can definitely control the versions that my java application includes. The problem is that when the java action runs, the jar files that I created are in the "current directory" of the java action along with the other versions that oozie share lib is including. These jar files have the same classpath precedence so it appears to be random which jar is picked up. For example, com.fasterxml.jackson.core.JsonFactory might be found in my own jar or in jackson-core-2.2.3.jar provided by oozie.
... View more
04-04-2016
01:29 PM
1 Kudo
Thank you @Kuldeep Kulkarni! It's causing me some problems because files like jackson-*.jar and guava appear early in the classpath (the current directory of the oozie action) and late in the classpath (mr-framework/hadoop/share/hadoop/common/lib/). I would effectively have to pin the dependencies of my java action to those precise versions. Any way around that?
... View more
04-04-2016
01:03 PM
2 Kudos
Hi, we recently upgraded from HDP 2.2.X to 2.3.4 and I notice that /user/oozie/share/lib/oozie contains a lot more jar files now. See below. The presence of these jar files is puzzling since they are otherwise in the classpath of, say, java oozie actions. e.g. in directories like `mr-framework/hadoop/share/hadoop/common/lib/` Does anyone know 1) why these jars in share/lib/oozie and 2) if they can be safely removed? Thanks! 2.3.4: -rw-r--r-- 3 oozie hadoop 11948376 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/aws-java-sdk-1.7.4.jar
-rw-r--r-- 3 oozie hadoop 666763 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/azure-storage-2.2.0.jar
-rw-r--r-- 3 oozie hadoop 412739 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/commons-lang3-3.3.2.jar
-rw-r--r-- 3 oozie hadoop 1648200 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/guava-11.0.2.jar
-rw-r--r-- 3 oozie hadoop 103052 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/hadoop-aws-2.7.1.2.3.4.0-3485.jar
-rw-r--r-- 3 oozie hadoop 137439 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/hadoop-azure-2.7.1.2.3.4.0-3485.jar
-rw-r--r-- 3 oozie hadoop 33483 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/jackson-annotations-2.2.3.jar
-rw-r--r-- 3 oozie hadoop 192699 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/jackson-core-2.2.3.jar
-rw-r--r-- 3 oozie hadoop 865838 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/jackson-databind-2.2.3.jar
-rw-r--r-- 3 oozie hadoop 570478 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/joda-time-2.1.jar
-rw-r--r-- 3 oozie hadoop 16046 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/json-simple-1.1.jar
-rw-r--r-- 3 oozie hadoop 12294 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/oozie-hadoop-utils-hadoop-2-4.2.0.2.3.4.0-3485.jar
-rw-r--r-- 3 oozie hadoop 49474 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/oozie-sharelib-oozie-4.2.0.2.3.4.0-3485.jar 2.2 -rw-r--r-- 3 hdfs hadoop 7832 2015-09-02 12:57 /user/oozie/share/lib/oozie/core-site.xml
-rw-r--r-- 3 hdfs hadoop 9517 2015-09-02 12:57 /user/oozie/share/lib/oozie/hdfs-site.xml
-rwxr-xr-x 3 oozie hadoop 16046 2015-06-17 23:08 /user/oozie/share/lib/oozie/json-simple-1.1.jar
-rw-r--r-- 3 hdfs hadoop 6671 2015-09-02 12:57 /user/oozie/share/lib/oozie/mapred-site.xml
-rwxr-xr-x 3 oozie hadoop 3242 2015-06-17 23:08 /user/oozie/share/lib/oozie/oozie-hadoop-utils-2.6.0.2.2.4.12-1.oozie-4.1.0.2.2.4.12-1.jar
-rwxr-xr-x 3 oozie hadoop 41903 2015-06-17 23:08 /user/oozie/share/lib/oozie/oozie-sharelib-oozie-4.1.0.2.2.4.12-1.jar
-rw-r--r-- 3 hdfs hadoop 18164 2015-11-04 18:55 /user/oozie/share/lib/oozie/yarn-site.xml
... View more
Labels:
- Labels:
-
Apache Oozie
03-11-2016
07:31 PM
2 Kudos
I should have done a bit more research... this is a problem in Storm 0.10, but it looks like the upcoming Storm 1.0 release will handle this situation by allowing properties to be passed directly to the Kafka Bolt instead of indirectly through the Topology configuration: https://github.com/apache/storm/blob/master/external/storm-kafka/src/jvm/org/apache/storm/kafka/bolt/KafkaBolt.java#L87-L90
... View more