Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive action fails

Hive action fails

Explorer

When I try to alter the in hive find some errors,

 

Error in acquireLock... FAILED: Error in acquiring locks: Locks on the underlying objects cannot be acquired. retry after some time

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.Hive Main], exit code [10]

 

 

13 REPLIES 13
Highlighted

Re: Hive action fails

Master Collaborator

Ashok,

 

  Can you provide us a little more detail on this issue?  What is your oozie action trying to do in Hive?  Any specific commands or a larger log snippet you can share?

 

Regards,


Clint

Re: Hive action fails

Expert Contributor

Hey Ashok,

 

Can you possibly strip out the important details and provide your workflow.xml and job.properties?  Also, are you using Kerberos?

 

Below is a sample workflow.xml and job.properties for hive.  Note, hive-site.xml must exist in HDFS and job-xml should point to the hive-site.xml in HDFS.

 

Thanks

 

 

job.properties:

 

nameNode=hdfs://cdh42-1.test.com:8020
jobTracker=cdh42-1.test.com:8021
queueName=default
baseDir=${nameNode}/user/${user.name}/hive_wf
hiveSite=hive-site.xml
sqlScript=test5.sql
hiveLogLevel=DEBUG
oozie.use.system.libpath=true

 

 

workflow.xml:

 

 

<workflow-app
xmlns ="uri:oozie:workflow:0.2" name="chris-hive-wf">
<start to="action1"/>

<action name="action1">
<hive
xmlns ="uri:oozie:hive-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<job-xml>${hiveSite}</job-xml>
<configuration>
<property>
<name>oozie.hive.log.level</name>
<value>${hiveLogLevel}</value>
</property>
</configuration>
<script>${sqlScript}</script>
</hive>
<ok to="end"/>
<error to="fail"/>
</action>

<kill name="fail">
<message>Map/Reduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>

Re: Hive action fails

New Contributor

I'm having a similar issue.  How do I see workflow.xml and job.properties to get this issue debugged?  I'm a Hive novice so I may require step by step directions.

Re: Hive action fails

Expert Contributor

Hey Chardin,

 

How are you running the oozie workflow that is having the hive issue?  That will help to determine where to look for the workflow.xml and job.properties.

 

Thanks

Re: Hive action fails

New Contributor

I don't know how to answer that.  I submitted my Hive query via the Thrift interface.  I'm unaware of setting up any specific thing other than that.  I didn't configure the server, but I have root on it and can determine answers to questions if I am given reasonably specific directions.

Re: Hive action fails

New Contributor

Note:  We are using Cloudera Manager with pretty much all of the default options.

Re: Hive action fails

New Contributor

To clarify further:  I get no message about "Failing Oozie Launcher".  I only get:

 

Error in acquireLocks...
FAILED: Error in acquiring locks: Locks on the underlying objects cannot be acquired. retry after some time

Re: Hive action fails

New Contributor

Adding more information in the hope of a reply:

 

The problem seems to be that consecutive Hive queries build up the number of connections over time.  For every query we execute, the command

 

/usr/sbin/lsof -p <hiveserver_pid> | grep -i tcp

 

grows by two lines at the beginning of a query, then shrinks by one sometime during or after the map portion of the query.  Over time this number grows to 65 lines, at which point the Hive server presumably runs out of connections and gives up.

 

We are using Hive version 0.10.0+198 and Hive Server version 1.cdh4.4.0.p0.15.el6, according to our RedHat packages.

Re: Hive action fails

Master Guru
Can you clarify if you use HiveServer (i.e. HiveServer1/HS1) or the newer, supported HiveServer2?

If the former, then its best to simply disable the concurrency property for it specifically, as it doesn't matter to it anyway.
Don't have an account?
Coming from Hortonworks? Activate your account here