Reply
Highlighted
Explorer
Posts: 24
Registered: ‎06-24-2018
Accepted Solution

Hive Hiveserver2 is continuously exiting, it doesn't start

hi, 

 

can anybody help me ? when i restart hiveserver2 it work for 2 minute and then role goes to bad healht stating "

  • The health test result for HIVESERVER2_UNEXPECTED_EXITS has become good: This role encountered 0 unexpected exit(s) in the previous 5 minute(s)"

 

Here are the logs:

at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:89)
	at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:793)
	at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:666)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.net.BindException: Address already in use
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:444)
	at sun.nio.ch.Net.bind(Net.java:436)
Posts: 749
Topics: 1
Kudos: 175
Solutions: 95
Registered: ‎04-22-2014

Re: Hive Hiveserver2 is continuously exiting, it doesn't start

@hadoopNoob,

 

"Address already in use" means that HiveServer2 cannot bind to a port to listen because it is already in use by another server.

 

you can run "netstat -nap |grep <port>" if your HiveServer2 uses the default.

 

HiveServer2 uses the following ports by default: 10000 and 10002

Once you find out what process is using the port, you can stop or kill it (using the proper caution depending on what it is).

 

 

Explorer
Posts: 24
Registered: ‎06-24-2018

Re: Hive Hiveserver2 is continuously exiting, it doesn't start

tcp 0 0 0.0.0.0:10002 0.0.0.0:* LISTEN 22735/java
tcp 0 0 185.38.3.144:10002 87.92.98.123:53233 ESTABLISHED 22735/java

this is what i got and i have killed it and stopped it many time, should i reassign a new port to hiveserver2 ? if yes how ?

Posts: 749
Topics: 1
Kudos: 175
Solutions: 95
Registered: ‎04-22-2014

Re: Hive Hiveserver2 is continuously exiting, it doesn't start

10002 is the HiveServer2 web UI port and it should be freed up when HiveServer2 shuts down.

The netstat output shows that some client is connected to your HiveServer2 UI port.

You could try to figure out what client that may be and what it is doing since it is a bit unusual that a connection to the HiveServer2 UI would last very long.  Finding out what client is running on 87.92.98.123 may be a good thing.

 

How are you stopping HiveServer2?  If you are stopping it from Cloudera Manager and it fails to stop completely that indicates that HiveServer2 itself may not be able to shut down due to the client holding the port open.  If that happens, you can try getting some jstack output by going to the HiveServer2 page in the Cloudera Manager UI and choosing Collect Stack Traces from the "Actions" menu.

 

That should help explain why HiveServer2 isn't stopping

 

Oh, and get a "ps aux" output against the PID found in the "netstat" output (22735 in the one you shared).

We should verify that the process is indeed HiveServer2

 

Explorer
Posts: 24
Registered: ‎06-24-2018

Re: Hive Hiveserver2 is continuously exiting, it doesn't start

[ Edited ]

ok, 87.92.98.123 --> this here is my router's default configuration page, surprisingly.
No error occurs while starting or stopping hiveserver2 or hive. It will start correctly and then after few minutes will be pulled down.
I verified 22735, it is hive but not specifically hiveserver2

Explorer
Posts: 24
Registered: ‎06-24-2018

Re: Hive Hiveserver2 is continuously exiting, it doesn't start

[ Edited ]

I almost have tried everything , i am not sure what is causing this problem. the problem is with unexpected exits

 

tcp 0 0 0.0.0.0:10002 0.0.0.0:* LISTEN 22735/java

 

port is free now, when it unexpectedly exit

 

Explorer
Posts: 24
Registered: ‎06-24-2018

Re: Hive Hiveserver2 is continuously exiting, it doesn't start

[ Edited ]

+ [[ -f /run/cloudera-scm-agent/process/527-hive-HIVESERVER2/ ]]
+ exec /opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hive/bin/hive --config /run/cloudera-scm-agent/process/527-hive-HIVESERVER2 --service hiveserver2
18/08/06 19:19:40 WARN conf.HiveConf: HiveConf of name hive.server2.idle.session.timeout_check_operation does not exist
18/08/06 19:19:40 WARN conf.HiveConf: HiveConf of name hive.entity.capture.input.URI does not exist


this is what i got from recent stderr

 

 

and this from role log

 

Error starting HiveServer2: could not start ThriftBinaryCLIService
org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address 0.0.0.0/0.0.0.0:10000.
Announcements