Hiveserver2 fails to connect after a while,

jeyanth_k — Wed, 15 Feb 2017 11:42:46 GMT

We recently upgraded to HDP 2.5 and now hiveserver2 fails to connect after a day or two

org.apache.hive.service.cli.HiveSQLException: Failed to open new session: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.IllegalArgumentException: java.net.UnknownHostException: NAMENODE_FQDN at org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:289) at org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:204) at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:421) at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:316) at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1257) at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1242) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.IllegalArgumentException: java.net.UnknownHostException: NAMENODE_FQDN at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:83) at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) at com.sun.proxy.$Proxy23.open(Unknown Source) at org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:281) ... 12 more Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: java.net.UnknownHostException: NAMENODE_FQDN at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:543) at org.apache.hive.service.cli.session.HiveSessionImpl.open(HiveSessionImpl.java:144) at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ... 20 more Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: NAMENODE_FQDN at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:438) at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:311) at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:688) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:629) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:159) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2761) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2795) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2777) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:386) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:179) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:526) ... 25 more Caused by: java.net.UnknownHostException: NAMENODE_FQDN

I do have my NAMENODE_FQDN in /etc/hosts file. But after restart of hiveserver2 in ambari it works just fine but fails again after a day or two.

Re: Hiveserver2 fails to connect after a while,

rrrajesh — Fri, 17 Feb 2017 12:18:39 GMT

@Jeyanth Kumar Kathiresan

It could be a bug or network issue between hive server and namenode server.

Try to start with hive server log and name node log, see if you can find any timeouts or errors during that time.

If you found no issues with the logs, then work with Hortonworks support, it could be a bug.

Re: Hiveserver2 fails to connect after a while,

jeyanth_k — Fri, 24 Feb 2017 12:23:24 GMT

hey Rajesh,

After looking through logs for a while i found out that hiveserver couldn't able to open new sockets due to limits in ulimits. Increasing ulimits for files fix this problem.

question Re: Hiveserver2 fails to connect after a while, in Archives of Support Questions (Read Only)

Hiveserver2 fails to connect after a while,

Re: Hiveserver2 fails to connect after a while,

Re: Hiveserver2 fails to connect after a while,