08-06-2018 07:52 AM
can anybody help me ? when i restart hiveserver2 it work for 2 minute and then role goes to bad healht stating "
Here are the logs:
at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:89) at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:793) at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:666) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:444) at sun.nio.ch.Net.bind(Net.java:436)
08-06-2018 08:09 AM
"Address already in use" means that HiveServer2 cannot bind to a port to listen because it is already in use by another server.
you can run "netstat -nap |grep <port>" if your HiveServer2 uses the default.
HiveServer2 uses the following ports by default: 10000 and 10002
Once you find out what process is using the port, you can stop or kill it (using the proper caution depending on what it is).
08-06-2018 08:12 AM
08-06-2018 08:26 AM
10002 is the HiveServer2 web UI port and it should be freed up when HiveServer2 shuts down.
The netstat output shows that some client is connected to your HiveServer2 UI port.
You could try to figure out what client that may be and what it is doing since it is a bit unusual that a connection to the HiveServer2 UI would last very long. Finding out what client is running on 18.104.22.168 may be a good thing.
How are you stopping HiveServer2? If you are stopping it from Cloudera Manager and it fails to stop completely that indicates that HiveServer2 itself may not be able to shut down due to the client holding the port open. If that happens, you can try getting some jstack output by going to the HiveServer2 page in the Cloudera Manager UI and choosing Collect Stack Traces from the "Actions" menu.
That should help explain why HiveServer2 isn't stopping
Oh, and get a "ps aux" output against the PID found in the "netstat" output (22735 in the one you shared).
We should verify that the process is indeed HiveServer2
08-06-2018 09:07 AM - edited 08-06-2018 10:45 AM
ok, 22.214.171.124 --> this here is my router's default configuration page, surprisingly.
No error occurs while starting or stopping hiveserver2 or hive. It will start correctly and then after few minutes will be pulled down.
I verified 22735, it is hive but not specifically hiveserver2
08-06-2018 09:10 AM - edited 08-06-2018 09:12 AM
I almost have tried everything , i am not sure what is causing this problem. the problem is with unexpected exits
tcp 0 0 0.0.0.0:10002 0.0.0.0:* LISTEN 22735/java
port is free now, when it unexpectedly exit
08-06-2018 09:23 AM - edited 08-06-2018 09:30 AM
+ [[ -f /run/cloudera-scm-agent/process/527-hive-HIVESERVER2/ ]]
+ exec /opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hive/bin/hive --config /run/cloudera-scm-agent/process/527-hive-HIVESERVER2 --service hiveserver2
18/08/06 19:19:40 WARN conf.HiveConf: HiveConf of name hive.server2.idle.session.timeout_check_operation does not exist
18/08/06 19:19:40 WARN conf.HiveConf: HiveConf of name hive.entity.capture.input.URI does not exist
this is what i got from recent stderr
and this from role log
Error starting HiveServer2: could not start ThriftBinaryCLIService org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address 0.0.0.0/0.0.0.0:10000.