Hive Hiveserver2 is continuously exiting, it doesn't start

[#1]

hi, 

can anybody help me ? when i restart hiveserver2 it work for 2 minute and then role goes to bad healht stating "

• The health test result for HIVESERVER2_UNEXPECTED_EXITS has become good: This role encountered 0 unexpected exit(s) in the previous 5 minute(s)"

Here are the logs:

at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:89)
	at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:793)
	at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:666)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.net.BindException: Address already in use
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:444)
	at sun.nio.ch.Net.bind(Net.java:436)

[#2]

@hadoopNoob,

"Address already in use" means that HiveServer2 cannot bind to a port to listen because it is already in use by another server.

you can run "netstat -nap |grep <port>" if your HiveServer2 uses the default.

HiveServer2 uses the following ports by default: 10000 and 10002

Once you find out what process is using the port, you can stop or kill it (using the proper caution depending on what it is).

[#3]

tcp 0 0 0.0.0.0:10002 0.0.0.0:* LISTEN 22735/java
tcp 0 0 185.38.3.144:10002 87.92.98.123:53233 ESTABLISHED 22735/java

this is what i got and i have killed it and stopped it many time, should i reassign a new port to hiveserver2 ? if yes how ?

[#4 · Accepted solution]

10002 is the HiveServer2 web UI port and it should be freed up when HiveServer2 shuts down.

The netstat output shows that some client is connected to your HiveServer2 UI port.

You could try to figure out what client that may be and what it is doing since it is a bit unusual that a connection to the HiveServer2 UI would last very long.  Finding out what client is running on 87.92.98.123 may be a good thing.

How are you stopping HiveServer2?  If you are stopping it from Cloudera Manager and it fails to stop completely that indicates that HiveServer2 itself may not be able to shut down due to the client holding the port open.  If that happens, you can try getting some jstack output by going to the HiveServer2 page in the Cloudera Manager UI and choosing Collect Stack Traces from the "Actions" menu.

That should help explain why HiveServer2 isn't stopping

Oh, and get a "ps aux" output against the PID found in the "netstat" output (22735 in the one you shared).

We should verify that the process is indeed HiveServer2

[#5]

ok, 87.92.98.123 --> this here is my router's default configuration page, surprisingly.
No error occurs while starting or stopping hiveserver2 or hive. It will start correctly and then after few minutes will be pulled down.
I verified 22735, it is hive but not specifically hiveserver2

[#6]

I almost have tried everything , i am not sure what is causing this problem. the problem is with unexpected exits

tcp 0 0 0.0.0.0:10002 0.0.0.0:* LISTEN 22735/java

port is free now, when it unexpectedly exit

[#7]

+ [[ -f /run/cloudera-scm-agent/process/527-hive-HIVESERVER2/ ]]
+ exec /opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hive/bin/hive --config /run/cloudera-scm-agent/process/527-hive-HIVESERVER2 --service hiveserver2
18/08/06 19:19:40 WARN conf.HiveConf: HiveConf of name hive.server2.idle.session.timeout_check_operation does not exist
18/08/06 19:19:40 WARN conf.HiveConf: HiveConf of name hive.entity.capture.input.URI does not exist


this is what i got from recent stderr

and this from role log

Error starting HiveServer2: could not start ThriftBinaryCLIService
org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address 0.0.0.0/0.0.0.0:10000.