Hive server connections start to fail after a short period of running correctly.

[#1]

Hi,

I am running a four node instance of CHD on  AWS(m1.mediums).  We are running into some very odd behaviour that I am hoping someone can help me troubleshoot.   We have imported some data into hive using a few different methods(Sqoop, and direct import), And we then connecting Hive to an instance of  JaserSoft, to run build some basic reports.   Everything works fine for a period of a few hours(4-6)  then the connections to Hive start to fail, for no reason that I can see.  I have  tried looking through what logs I see, but nothing is jumping out at me. I am seeing a number of logs that look like this:

/var/log/hadoop-0.20-mapreduce/history/done/ip-10-139-5-19.ec2.internal_1391232018142_/2014/02/02/000000# less job_201402010520_0032_1391311952515_root_select+count%28\*%29+as+trp_erro...error_code%3D203%28Stage

but the test in them doesn't indicate any problems.  I am very new to the Hadoop world, so forgive me if this is a simple issue, but can anyone give me any hints at how I might troubleshoot this issue?  I have rebuilt the entire cluster from scratch once already, so I know this must be some sort of configuration issue. 

We are running the laster CDH instance,  I started the thrift server in the ubunut home directory using:

hive --service hiveserver -p 10000 & 

Thanks

Andrew

[#2]

Hi Andrew,

Are you trying to run HiveServer or HiveServer2? The command you're running will start the old HiveServer, which is discouraged due to a lot of bugs and architectural issues.

I'm assuming you are using Cloudera Manager since this is a Cloudera Manager forum. If not, then you may want to re-post in the Hive forum:
http://community.cloudera.com/t5/Batch-SQL-Apache-Hive/bd-p/Hive

This blog post can help explain some important aspects of using Hive in Cloudera Manager:
http://blog.cloudera.com/blog/2013/03/how-to-set-up-cloudera-manager-4-5-for-apache-hive/

If using Cloudera Manager, you can search for any error messages from Hive, HDFS, and MapReduce (or YARN, whichever Hive is configured to use) Log Search. The log you pointed at is just the MR log, but if you are losing connections to hive, then the client and hive server (2) logs will be more interesting. You can also usually check /var/log/hive for hive server (2) logs.

Thanks,
Darren

[#3]

Hi Darren, 

I have tried both HiveServer and HiveServer2,with basically the same results.  after about eight hours things just stop working. Connection tests still pass, but whenever we attempt to pull data, the process just hangs(we have tried from mutiple applications, Eclipse,SquirrelSQL, and JasperServer).  We are using the latest version of CDH.   I wll have a look at the Blog post.  Thanks!

Andrew

[#4]

I would try in the Hive forum I mentioned before to see if they can help. Would be nice to try and find a relevant log error as well, maybe in hdfs or hive.

[#5 · Accepted solution]

We managed to get this issue resloved.  The problem acutally had nothing to do with Hive but Zookeeper.  By default there is a limit of 60 active connections from a single IP address. We were running over this limit becuase we have such a small cluster(4 servers).  We upped the limit on connections to 200 and everything started working properly.