Unable to start Hiveserver2 on my ambari server (HDP 3.0)

[#1]

Hello community,

I already installed my HDP 3.0 and started my required services.

However, even if all my services started well without any issues, my Hive service is still not running.

I have an issue with hiveserver2 start, I'm getting the error given below :

2018-10-25 12:33:59,215 - Will retry 6 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2018-10-25 12:34:09,223 - Process with pid 29602 is not running. Stale pid file at /var/run/hive/hive-server.pid

Can someone help me please ?

It's an emergency...

Thank you in advance 🙂

[#2]

HiveServer now uses a remote instead of an embedded metastore; consequently, Ambari no longer starts the metastore using hive.metastore.uris=' '. You no longer set key=value commands on the command line to configure Hive Metastore. You configure properties in hive-site.xml. The Hive catalog resides in Hive Metastore, which is RDBMS-based as it was in earlier releases. Using this architecture, Hive can take advantage of RDBMS resources in a cloud deployments.

So please check or share your hive-site.xml. for hive.server2. properties.

[#3]

HI @Gangadhar Kadam,

Thank you so much for your reply.

I got the same issue even if I make hive.metastore.uris=' ' in my Hive configs.

You find below the hiveserve2 properties :

<property>
  <name>hive.server2.allow.user.substitution</name>
  <value>true</value>
</property>

<property>
  <name>hive.server2.authentication</name>
  <value>NONE</value>
</property>

<property>
  <name>hive.server2.authentication.spnego.keytab</name>
  <value>HTTP/_HOST@EXAMPLE.COM</value>
</property>

<property>
  <name>hive.server2.authentication.spnego.principal</name>
  <value>/etc/security/keytabs/spnego.service.keytab</value>
</property>

<property>
  <name>hive.server2.enable.doAs</name>
  <value>false</value>
</property>

<property>
  <name>hive.server2.idle.operation.timeout</name>
  <value>6h</value>
</property>

<property>
  <name>hive.server2.idle.session.timeout</name>
  <value>1d</value>
</property>

<property>
  <name>hive.server2.logging.operation.enabled</name>
  <value>true</value>
</property>

<property>
  <name>hive.server2.logging.operation.log.location</name>
  <value>/tmp/hive/operation_logs</value>
</property>

<property>
  <name>hive.server2.max.start.attempts</name>
  <value>5</value>
</property>

<property>
  <name>hive.server2.support.dynamic.service.discovery</name>
  <value>true</value>
</property>

<property>
  <name>hive.server2.table.type.mapping</name>
  <value>CLASSIC</value>
</property>

<property>
  <name>hive.server2.tez.default.queues</name>
  <value>default</value>
</property>

<property>
  <name>hive.server2.tez.initialize.default.sessions</name>
  <value>false</value>
</property>

<property>
  <name>hive.server2.tez.sessions.per.default.queue</name>
  <value>1</value>
</property>

<property>
  <name>hive.server2.thrift.http.path</name>
  <value>cliservice</value>
</property>

<property>
  <name>hive.server2.thrift.http.port</name>
  <value>10001</value>
</property>

<property>
  <name>hive.server2.thrift.sasl.qop</name>
  <value>auth</value>
</property>

<property>
  <name>hive.server2.transport.mode</name>
  <value>binary</value>
</property>

<property>
  <name>hive.server2.use.SSL</name>
  <value>false</value>
</property>

<property>
  <name>hive.server2.webui.cors.allowed.headers</name>
  <value>X-Requested-With,Content-Type,Accept,Origin,X-Requested-By,x-requested-by</value>
</property>

<property>
  <name>hive.server2.webui.enable.cors</name>
  <value>true</value>
</property>

<property>
  <name>hive.server2.webui.port</name>
  <value>10002</value>
</property>

<property>
  <name>hive.server2.webui.use.ssl</name>
  <value>false</value>
</property>

<property>
  <name>hive.server2.zookeeper.namespace</name>
  <value>hiveserver2</value>
</property>

Thank you in advance 🙂

[#4]

Hi @Gangadhar Kadam,

Can you please help me with this error ?

I'm still nor resolving it, and it's an emergency for me.

That's my hiveserver2 properties from hive-site.xml.

Thank you in advance 🙂

[#5]

@Ayoub Bounfour

Posting my response from other thread. here : https://community.hortonworks.com/comments/226829/view.html

As we see the message:

Will retry 6 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet

.

So it looks like the following command is returning Empty JSON. Please replace the xxxxxxx with your Zookeeper hostname.

Also it is better to have more than one zookeeper to maintain a zookeeper quorum so that if one ZK does not work still we have other Zookeepers working.

# /usr/hdp/current/zookeeper-client/bin/zkCli.sh -server xxxxxxxxx.internal:2181 ls /hiveserver2 | grep 'serverUri=''

Can you try running that command manually to validate if your Zookeeper is accessible and running fine and is returning the znode properly?

It might happen if your Zookeepers are not healthy (not running) or showing some errors in their logs.

[#6]

Thank you @Jay Kumar SenSharma for your reply.

I think there is something to add in "serverUri=' ' ? So the command can't run and returning : Exception in thread "main" java.lang.IllegalArgumentException: A HostProvider may not be empty!

Also, when I check Zookeeper service in my Ambari interface it's working and running very well (I restart it and still running properly).

Thank you in advance 🙂

[#7]

@Ayoub Bounfour

Exception in thread "main" java.lang.IllegalArgumentException: A HostProvider may not be empty!

The above error that you reported is coming from The "StaticHostProvider.java" class.

which has the following code:

https://github.com/apache/zookeeper/blob/release-3.4.6/src/java/main/org/apache/zookeeper/client/Sta...

                    this.serverAddresses.add(
                            new InetSocketAddress(InetAddress.getByAddress(
                                    address.getHostName(),
                                    resolvedAddress.getAddress()), 
                                    address.getPort()));
                } else {
                    this.serverAddresses.add(new InetSocketAddress(resolvedAddress.getHostAddress(), address.getPort()));
                }  
            }
        }
        
        if (this.serverAddresses.isEmpty()) {
            throw new IllegalArgumentException(
                    "A HostProvider may not be empty!");
        }

.

So somehoiw the FQDN (hostname) of your Zookeeper host is not correct. So can you please check if you are specifying the IP Adderss/Hostname in your configs?

Also please verify the following command on your cluster nodes to verify if the FQDN (fully qualified Hostname) is correctly setup? Also please chekc that every host is resolving each other using their hostnmae.

# hostname -f

# cat /etc/hosts

.

[#8]

Hi @Jay Kumar SenSharma,

I fixed the host error, and now when I execute the command you gave me by adding (') in the end, it works well without returning any output.

However, If I execute it as you wrote it, it returns the result given below :

Connecting to XXXXXXXXXXXXXX:2181
2018-10-26 09:29:29,163 - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.6-187--1, built on 09/19/2018 10:36 GMT
2018-10-26 09:29:29,165 - INFO  [main:Environment@100] - Client environment:host.name=XXXXXXXXXXXXXX
2018-10-26 09:29:29,165 - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_112
2018-10-26 09:29:29,167 - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2018-10-26 09:29:29,167 - INFO  [main:Environment@100] - Client environment:java.home=/usr/jdk64/jdk1.8.0_112/jre
2018-10-26 09:29:29,167 - INFO  [main:Environment@100] - Client environment:java.class.path=/usr/hdp/current/zookeeper-client/bin/../build/classes:/usr/hdp/current/
zookeeper-client/bin/../build/lib/*.jar:/usr/hdp/current/zookeeper-client/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/current/zookeeper-client/bin/../lib/slf4j-api-
1.6.1.jar:/usr/hdp/current/zookeeper-client/bin/../lib/netty-3.10.5.Final.jar:/usr/hdp/current/zookeeper-client/bin/../lib/log4j-1.2.16.jar:/usr/hdp/current/zookeep
er-client/bin/../lib/jline-0.9.94.jar:/usr/hdp/current/zookeeper-client/bin/../zookeeper-3.4.6.3.0.1.0-187.jar:/usr/hdp/current/zookeeper-client/bin/../src/java/lib
/*.jar:/usr/hdp/current/zookeeper-client/bin/../conf::/usr/share/zookeeper/*
2018-10-26 09:29:29,167 - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2018-10-26 09:29:29,167 - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2018-10-26 09:29:29,167 - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2018-10-26 09:29:29,168 - INFO  [main:Environment@100] - Client environment:os.name=Linux
2018-10-26 09:29:29,168 - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2018-10-26 09:29:29,168 - INFO  [main:Environment@100] - Client environment:os.version=4.15.0-1023-gcp
2018-10-26 09:29:29,168 - INFO  [main:Environment@100] - Client environment:user.name=root
2018-10-26 09:29:29,168 - INFO  [main:Environment@100] - Client environment:user.home=/root
2018-10-26 09:29:29,168 - INFO  [main:Environment@100] - Client environment:user.dir=/root
2018-10-26 09:29:29,169 - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=XXXXXXXXXXXXXX
:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@67424e82
2018-10-26 09:29:29,190 - INFO  [main-SendThread(XXXXXXXXXXXXXX:2181):ClientCnxn$SendThread@1019] - Opening socke
t connection to server XXXXXXXXXXXXXX/XX.XXX.X.X:2181. Will not attempt to authenticate using SASL (unknown error
)
2018-10-26 09:29:29,256 - INFO  [main-SendThread(XXXXXXXXXXXXXX:2181):ClientCnxn$SendThread@864] - Socket connect
ion established, initiating session, client: /XX.XXX.X.X:37962, server: XXXXXXXXXXXXXX/XX.XXX.X.X:2181
2018-10-26 09:29:29,263 - INFO  [main-SendThread(XXXXXXXXXXXXXX:2181):ClientCnxn$SendThread@1279] - Session estab
lishment complete on server XXXXXXXXXXXXXX/XX.XXX.X.X:2181, sessionid = 0x166ab69786a01a1, negotiated timeout = 3
0000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[]

Can please detect anything wrong ?

Thanks in advance 🙂

[#9]

@Jay Kumar SenSharma,

I remarked a weird behavior from hiveserver2. When I use the command manually, the HS2 starts and change its status to 'running' (when I do it from the ambari interface it doesn't run).

However, once I start hive from my terminal, the HS2 service stops again.

Do you have any idea please ?

Thank you 🙂