Reply
Explorer
Posts: 18
Registered: ‎08-13-2013

Hive action fails

When I try to alter the in hive find some errors,

 

Error in acquireLock... FAILED: Error in acquiring locks: Locks on the underlying objects cannot be acquired. retry after some time

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.Hive Main], exit code [10]

 

 

Posts: 416
Topics: 51
Kudos: 85
Solutions: 49
Registered: ‎06-26-2013

Re: Hive action fails

Ashok,

 

  Can you provide us a little more detail on this issue?  What is your oozie action trying to do in Hive?  Any specific commands or a larger log snippet you can share?

 

Regards,


Clint

Cloudera Employee
Posts: 90
Registered: ‎07-31-2013

Re: Hive action fails

Hey Ashok,

 

Can you possibly strip out the important details and provide your workflow.xml and job.properties?  Also, are you using Kerberos?

 

Below is a sample workflow.xml and job.properties for hive.  Note, hive-site.xml must exist in HDFS and job-xml should point to the hive-site.xml in HDFS.

 

Thanks

 

 

job.properties:

 

nameNode=hdfs://cdh42-1.test.com:8020
jobTracker=cdh42-1.test.com:8021
queueName=default
baseDir=${nameNode}/user/${user.name}/hive_wf
hiveSite=hive-site.xml
sqlScript=test5.sql
hiveLogLevel=DEBUG
oozie.use.system.libpath=true

 

 

workflow.xml:

 

 

<workflow-app
xmlns ="uri:oozie:workflow:0.2" name="chris-hive-wf">
<start to="action1"/>

<action name="action1">
<hive
xmlns ="uri:oozie:hive-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<job-xml>${hiveSite}</job-xml>
<configuration>
<property>
<name>oozie.hive.log.level</name>
<value>${hiveLogLevel}</value>
</property>
</configuration>
<script>${sqlScript}</script>
</hive>
<ok to="end"/>
<error to="fail"/>
</action>

<kill name="fail">
<message>Map/Reduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>

New Contributor
Posts: 5
Registered: ‎09-19-2013

Re: Hive action fails

I'm having a similar issue.  How do I see workflow.xml and job.properties to get this issue debugged?  I'm a Hive novice so I may require step by step directions.

Cloudera Employee
Posts: 90
Registered: ‎07-31-2013

Re: Hive action fails

Hey Chardin,

 

How are you running the oozie workflow that is having the hive issue?  That will help to determine where to look for the workflow.xml and job.properties.

 

Thanks

New Contributor
Posts: 5
Registered: ‎09-19-2013

Re: Hive action fails

I don't know how to answer that.  I submitted my Hive query via the Thrift interface.  I'm unaware of setting up any specific thing other than that.  I didn't configure the server, but I have root on it and can determine answers to questions if I am given reasonably specific directions.

New Contributor
Posts: 5
Registered: ‎09-19-2013

Re: Hive action fails

Note:  We are using Cloudera Manager with pretty much all of the default options.

New Contributor
Posts: 5
Registered: ‎09-19-2013

Re: Hive action fails

To clarify further:  I get no message about "Failing Oozie Launcher".  I only get:

 

Error in acquireLocks...
FAILED: Error in acquiring locks: Locks on the underlying objects cannot be acquired. retry after some time

New Contributor
Posts: 5
Registered: ‎09-19-2013

Re: Hive action fails

Adding more information in the hope of a reply:

 

The problem seems to be that consecutive Hive queries build up the number of connections over time.  For every query we execute, the command

 

/usr/sbin/lsof -p <hiveserver_pid> | grep -i tcp

 

grows by two lines at the beginning of a query, then shrinks by one sometime during or after the map portion of the query.  Over time this number grows to 65 lines, at which point the Hive server presumably runs out of connections and gives up.

 

We are using Hive version 0.10.0+198 and Hive Server version 1.cdh4.4.0.p0.15.el6, according to our RedHat packages.

Posts: 1,768
Kudos: 386
Solutions: 283
Registered: ‎07-31-2013

Re: Hive action fails

Can you clarify if you use HiveServer (i.e. HiveServer1/HS1) or the newer, supported HiveServer2?

If the former, then its best to simply disable the concurrency property for it specifically, as it doesn't matter to it anyway.
Announcements