Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Vertex error on Hive Oozie Action

Vertex error on Hive Oozie Action

New Contributor

We have a oozie workflow that calls a hive action. The hive SQL runs without any issue when run directly in the edge node in hive shell. When run through oozie , it aborts randomly. Our version is HDP 2.3

E.g in the below error social_hx is the table. This is a bucketed hive internal table. We are performing Delete and Insert operations on this table.

Please let me know if anyone has run into this error.

Error
2016-01-18 01:37:12,781 [ERROR] [Dispatcher thread {Central}] |impl.VertexImpl|: Vertex Input: social_hx_ept_update_ct initializer failed, vertex=vertex_1452786548451_5145_3_00 [Map 1]
org.apache.tez.dag.app.dag.impl.AMUserCodeException: java.lang.IllegalArgumentException: Illegal Capacity: -1
  at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallback.onFailure(RootInputInitializerManager.java:292)
  at com.google.common.util.concurrent.Futures$4.run(Futures.java:1140)
  at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)
  at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:150)
  at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:135)
  at com.google.common.util.concurrent.ListenableFutureTask.done(ListenableFutureTask.java:91)
  at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:380)
  at java.util.concurrent.FutureTask.setException(FutureTask.java:247)
  at java.util.concurrent.FutureTask.run(FutureTask.java:267)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Illegal Capacity: -1
  at java.util.ArrayList.<init>(ArrayList.java:142)
  at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:332)
  at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:305)
  at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:407)
  at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:155)
  at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:246)
  at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:240)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:415)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
  at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:240)
  at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:227)
  at java.util.concurrent.FutureTask.run(FutureTask.java:262)
  ... 3 more
5 REPLIES 5

Re: Vertex error on Hive Oozie Action

Mentor

@Maharaj Muthusamy can you run service check on YARN, Hive, Tez and MR and confirm they work.

Re: Vertex error on Hive Oozie Action

@Maharaj Muthusamy

"This is a bucketed hive internal table. We are performing Delete and Insert operations on this table."

I highly recommend to be careful of ACID operations on hive tables. There is very high probability that you may be hitting an unknown bug.

Highlighted

Re: Vertex error on Hive Oozie Action

Rising Star

@Maharaj Muthusamy

To turn on Hive transactions, just check the values of these parameters in your hive-site.xml are set as below

These parameters must also have non-default values to turn on Hive transactions:

If a table is to be used in ACID writes (insert, update, delete) then the table property "transactional=true" must be set on that table.

Also try to run with tez.am.log.level to DEBUG in your tez-site.xml, re-run it and see if you get an speocfic errors.

Re: Vertex error on Hive Oozie Action

Mentor

@Maharaj Muthusamy are you still having issues with this? Can you accept best answer or provide your workaround?

Re: Vertex error on Hive Oozie Action

New Contributor

I have a same problem like you . This problem possiblely like this issue HIVE-12957

Because fair-scheduler give a native resource value ( like -4096 MB, -1 vcore , Maybe this is a fair-scheduler's bug ,this issue doesn't inffect MR job,but Tez job does.

How to slove it ? Two solutions.

1. change fair-scheduler to capacity-scheduler. ( HDP recommend capacity-scheduler, but we use fair-scheduler, so we use solution 2 ).

2. patch HIVE-12957 , it has two versions. You can patch HIVE-12957.1.patch,it handle the negative resource value to make sure running well.