Member since
08-05-2016
25
Posts
5
Kudos Received
0
Solutions
02-07-2017
01:55 PM
Thx @Predrag Minovic one last question, the variable OOZIE_BASE_URL set in oozie-site or oozie-env is global for all the jobs scheduled or each job must define its own OOZIE_BASE_URL?
... View more
02-07-2017
09:18 AM
Hi @Predrag Minovic , thank you for the answer. Can you please tell me if there is something wrong not changing the oozie_base_url for the scheduled jobs, for example leaving it for localhost:11000/oozie ? I think that for the case of scheduled jobs, in terms of HA, if I use or not the load balancer as oozie_base_url nothing will change. This is because having two oozie servers sharing the same info, one of them will execute the job. However the load balancer starts to get relevance when we have a jobs on demand. I am on the right track? Thx @Kuldeep Kulkarni for the very relevant info. I am planing to use kerberos
... View more
02-06-2017
01:25 PM
Hi, I have some jobs that must run every night, these jobs are scheduled in oozie. By the moment I make oozie ha, the oozie servers will share these scheduling, My quetion is, Will these jobs scheduled be executed twice ? ( I suppose this is a no, but why?) And should I change the value of the variable oozie_base_url for these jobs to localhost or to my load balancer adress ?
... View more
Labels:
- Labels:
-
Apache Oozie
02-06-2017
01:05 PM
@Laurent Edel what about the jobs are planified to be run? if every oozie server know that there are jobs that must be executed, how to decide which execute the job? If it is done by locking using zookeeper, it does the job submissionpasses trhoughtout the load balancer /dns round robin / VIP?
... View more
01-31-2017
03:46 PM
Thank you, acctually I was missing the package oozie-2-3-2-0-2950-server. I paste the procedure apt-get install oozie-2-3-2-0-2950-server and then hadoop fs -put /usr/hdp/current/oozie-server/libtools/oozie-tools-4.2.0.2.3.2.0-2950.jar /user/oozie/share/lib/lib_xxxxxxx/oozie/
... View more
01-31-2017
10:15 AM
Hello guys I am following the documentation to make oozie HA http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.1/bk_Ambari_Users_Guide/content/_adding_an_oozie_server_component.html And after installing and changing the conf, when I restart oozie I got this error Error: Could not find or load main class org.apache.oozie.tools.OozieDBCLI
Do I forget something?
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Oozie
10-11-2016
08:10 AM
Yes, It was me who created the ticket.
... View more
09-29-2016
09:21 AM
@zyang @Constantin Stanca I created the ticket https://issues.apache.org/jira/browse/TEZ-3451 Thank you
... View more
09-23-2016
02:58 PM
2 Kudos
Hello, I have a table in cassandra, and I use the driver hive-cassandra to do selects over it. This is the table CREATE TABLE table1 (
campaign_id text,
sid text,
name text,
ts timestamp,
PRIMARY KEY (campaign_id, sid)
) WITH CLUSTERING ORDER BY (sid ASC) And I have only 3 partitions At the moment to query my table using hive like that hive -e "select count(*) from table1;" I got this error Status: Failed
Vertex failed, vertexName=Map 1,
vertexId=vertex_1474275943985_0179_1_00, diagnostics=[Task failed,
taskId=task_1474275943985_0179_1_00_000001, diagnostics=[TaskAttempt 0
failed, info=[Error: Failure while running
task:java.lang.RuntimeException:
org.apache.tez.dag.api.TezUncheckedException: Expected length: 12416
actual length: 9223372036854775711
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.tez.dag.api.TezUncheckedException: Expected length: 12416 actual length: 9223372036854775711
at org.apache.hadoop.mapred.split.TezGroupedSplit.readFields(TezGroupedSplit.java:128)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
at org.apache.tez.mapreduce.hadoop.MRInputHelpers.createOldFormatSplitFromUserPayload(MRInputHelpers.java:177)
at org.apache.tez.mapreduce.lib.MRInputUtils.getOldSplitDetailsFromEvent(MRInputUtils.java:136)
at org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:643)
at org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:621)
at org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:145)
at org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:109)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:390)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:128)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:147)
... 14 more
So far I understand that in readfields we are getting more data that we are expecting. But considering the size of the table, I dont think the data is a problem. @Constantin Stanca has helped me trying to find the problem, I am re lauching the subjet 🙂 Another thing to add is that if I do select * it works perfectly fine with tez 🙂 . Using the engine mp, select count(*) and select * works fine also. We are using hortonworks version 2.3.2
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Hive
-
Apache Tez