Hive beeline connectivity issue

[#1]

beeline> !connect jdbc:hive2://Hive-server2:10000

Connecting to jdbc:hive2://Hive-server2:10000

Enter username for jdbc:hive2://Hive-server2:10000: username

Enter password for jdbc:hive2://Hive-server2:10000: *********

17/09/29 15:46:32 [main]: WARN jdbc.HiveConnection: Failed to connect to Hive-server2:10000

Error: Could not open client transport with JDBC Uri: jdbc:hive2://Hive-server2:10000: java.net.ConnectException: Connection refused (state=08S01,code=0)

0: jdbc:hive2://Hive-server2:10000 (closed)>

I checked few other posts on the community forums, but didn't really help me.

Please let me know what am I missing here guys..Thanks in Advance

[#2]

@Bharath Nagamalla

One of the reason may be that from the host where you are running the beeline, "Hive-server2:10000" host/port might not be accessible Or the port 10000 might not be opened by the HS2 server.

So please check the following:

1. From the host where you are running the Beeline , are you able to access the host & Port of HS2 ? To isolate the Firewall or N/W issue.

# telnet  Hive-server2  10000
(OR)
# nc  -v  Hive-server2  10000

2. Please check on the HiveServer2 host if the port 10000 is opened ? Please run the following command on the Hive Server 2 host to see if the port is opened ?

# netstat -tnlpa | grep 10000

3. If the port 10000 is not opened then please check the Hive Server2 log and share if you find any error there.

.

Additionally if this is a Kerberized Cluster then you might need to also pass the Principal in the connection URL. Please refer to the following link for more detail on the same: https://community.hortonworks.com/articles/4103/hiveserver2-jdbc-connection-url-examples.html

[#3]

@Jay SenSharmaBelow is the response:

[hadoop-admin@edge-node hive]$ telnet hive-server2 10000 telnet: hive-server2: Name or service not known

[hadoop-admin@edge-node hive]$ nc -v hive-server2 10000 nc: getaddrinfo: Name or service not known

Ran the following command on the Hive Server 2 host to see if the port is opened ?

root@Name-node:~ # netstat -tnlpa|grep 10000

Itdidn't return anything

Also, ran the Hive Debug command :I found this from the output

FATAL thrift.ThriftCLIService: Error starting HiveServer2: could not start ThriftHttpCLIService

[#4]

@Bharath N

[hadoop-admin@edge-node hive]$ telnet hive-server2 10000 telnet: hive-server2: Name or service not known

Above message, means that your HiveServer2 Hostname (FQDN) is not set correctly Or might not be mapped correctly on the Beeline "/etc/hosts" file.

So first of all please check the FQDN of your HiveServer2 host.

1. Login to HiveServer2 host using SSH and check if it has correct FQDN/Hostname

# hostname -f
# cat /etc/hosts

.

2. On the machine where you are running Beeline, please check if it has correct Hostname maping for the hiveserver2 inside it's "/etc/hosts" file. it should resolved the hiveserver2 host name correctly.

# cat /etc/hosts

.

3. Also we see that the port 10000 is not opened on the HiveServer2 host

root@Name-node:~ # netstat -tnlpa|grep 10000

So please check the Hive configuration to find out if it is using the correct port 10000? And also check the hiveServer2 logs to find out if it started successfully or not?

.

Please refer to the following links to findout if you have configured the Hostname/FQDN properly:

https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.2.0/bk_ambari-installation-ppc/content/edit_the...

https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.2.0/bk_ambari-installation-ppc/content/set_the_...

https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.2.0/bk_ambari-installation-ppc/content/edit_the...

.

Also please make sure that the "iptables" is disabled on the HiveServer2 host. So that it's port can be accessed from a remote host.

# service iptables status
# service iptables stop

(OR)

# systemctl disable firewalld
# service firewalld stop

.

Also permanently disable SELinux set, If it is not already done.

SELINUX=disabled

in/etc/selinux/configThis ensures that SELinux does not turn itself on after you reboot the machine .

.

[#5]

@Jay SenSharma I don't see anything under /var/log/hive -its empty

I ran the jps command on node where hive server is running

root@Hive-server:~ # jps 55267 QuorumPeerMain

186337 RunJar

38279 JournalNode

43241 Jps

183016 RunJar

184015 NameNode

9038 HistoryServer

36721 ActivityAnalyzerFacade

192339 Bootstrap

6547 HistoryServer

189617 RunJar

9460 SparkSubmit

39412 DFSZKFailoverController

8089 LivyServer

13246 HMaster

7006 SparkSubmit

Also, I checked the "etc/hosts" file on both the hive-server node and node from where beeline command is executing./...its exactly the same.

From Ambari Ui, I see the port# is 10000.

screen-shot-2017-10-02-at-115713-am.png

I am not sure what am I missing here.

Please let me know how to open a port, if that is what causing the issue here?

[#6]

@Bharath N

Please check if HiveServer2 is running or not? Please check the Ambari UI.

# ps -ef | grep hiveserver2

Also please check the Hive Logs.

Please also check the port (10000) and other ports opened by the HiveServer2 process. Like following:

# netstat -tnlpa | grep `cat /var/run/hive/hive-server.pid`
tcp  0  0 0.0.0.0:10000  0.0.0.0:*  LISTEN  1755/java   

.

If you do not see the port opened yet then there must be some ERROR / WARNING mentioned in the HiveServer2 logs. In that case please check the hive logs , Please check and share if any.

less /var/log/hive/hiveserver2.log

.

[#7]

@Jay SenSharma

1. # ps -ef | grep hiveserver2

I see the process running, Also Ambari UI has the hiveserver2 running

screen-shot-2017-10-02-at-21921-pm.png

2. # netstat -tnlpa | grep `cat /var/run/hive/hive-server.pid`

I see a list, but I don't see any process with 10000.Instead I see couple of them with 10001

3. less /var/log/hive/hiveserver2.log

> 2017-10-02 00:00:36,108 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doPost(145)) - Could not validate cookie sent, will try to generate a new cookie

2017-10-02 00:00:36,108 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doPost(204)) - Cookie added for clientUserName anonymous

2017-10-02 00:00:36,108 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(316)) - Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8

2017-10-02 00:00:36,109 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: metastore.ObjectStore (ObjectStore.java:initializeHelper(377)) - ObjectStore, initialize called

2017-10-02 00:00:36,111 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:<init>(139)) - Using direct SQL, underlying DB is OTHER

2017-10-02 00:00:36,111 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: metastore.ObjectStore (ObjectStore.java:setConf(291)) - Initialized ObjectStore

2017-10-02 00:00:36,111 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: metastore.HiveMetaStore (HiveMetaStore.java:init(533)) - Begin calculating metadata count metrics.

2017-10-02 00:00:36,113 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: metastore.HiveMetaStore (HiveMetaStore.java:init(535)) - Finished metadata count metrics: 18 databases, 1009 tables, 33 partitions.

2017-10-02 00:00:36,113 WARN [HiveServer2-HttpHandler-Pool: Thread-82257]: metrics2.CodahaleMetrics (CodahaleMetrics.java:addGauge(299)) - A Gauge with name [init_total_count_dbs] already exists. The old gauge will be overwritten, but this is not recommended

2017-10-02 00:00:36,113 WARN [HiveServer2-HttpHandler-Pool: Thread-82257]: metrics2.CodahaleMetrics (CodahaleMetrics.java:addGauge(299)) - A Gauge with name [init_total_count_tables] already exists. The old gauge will be overwritten, but this is not recommended

2017-10-02 00:00:36,113 WARN [HiveServer2-HttpHandler-Pool: Thread-82257]: metrics2.CodahaleMetrics (CodahaleMetrics.java:addGauge(299)) - A Gauge with name [init_total_count_partitions] already exists. The old gauge will be overwritten, but this is not recommended

2017-10-02 00:00:36,125 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: session.SessionState (SessionState.java:createPath(677)) - Created local directory: /tmp/de11c83f-e087-406e-b715-dd6ba7148cfe_resources

2017-10-02 00:00:36,128 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: session.SessionState (SessionState.java:createPath(677)) - Created HDFS directory: /tmp/hive/anonymous/de11c83f-e087-406e-b715-dd6ba7148cfe

2017-10-02 00:00:36,129 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: session.SessionState (SessionState.java:createPath(677)) - Created local directory: /tmp/hive/de11c83f-e087-406e-b715-dd6ba7148cfe

2017-10-02 00:00:36,130 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: session.SessionState (SessionState.java:createPath(677)) - Created HDFS directory: /tmp/hive/anonymous/de11c83f-e087-406e-b715-dd6ba7148cfe/_tmp_space.db

2017-10-02 00:00:36,130 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: session.HiveSessionImpl (HiveSessionImpl.java:setOperationLogSessionDir(265)) - Operation log session directory is created: /tmp/hive/operation_logs/de11c83f-e087-406e-b715-dd6ba7148cfe

2017-10-02 00:00:36,189 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: session.HiveSessionImpl (HiveSessionImpl.java:acquireAfterOpLock(333)) - We are setting the hadoop caller context to de11c83f-e087-406e-b715-dd6ba7148cfe for thread HiveServer2-HttpHandler-Pool: Thread-82257

2017-10-02 00:00:36,190 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: session.HiveSessionImpl (HiveSessionImpl.java:releaseBeforeOpLock(357)) - We are resetting the hadoop caller context for thread HiveServer2-HttpHandler-Pool: Thread-82257

2017-10-02 00:00:36,196 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: session.HiveSessionImpl (HiveSessionImpl.java:acquireAfterOpLock(333)) - We are setting the hadoop caller context to de11c83f-e087-406e-b715-dd6ba7148cfe for thread HiveServer2-HttpHandler-Pool: Thread-82257

2017-10-02 00:00:36,196 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: session.HiveSessionImpl (HiveSessionImpl.java:releaseBeforeOpLock(357)) - We are resetting the hadoop caller context for thread HiveServer2-HttpHandler-Pool: Thread-82257

2017-10-02 00:00:36,210 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: session.HiveSessionImpl (HiveSessionImpl.java:acquireAfterOpLock(333)) - We are setting the hadoop caller context to de11c83f-e087-406e-b715-dd6ba7148cfe for thread HiveServer2-HttpHandler-Pool: Thread-82257

2017-10-02 00:00:36,211 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: session.HiveSessionImpl (HiveSessionImpl.java:releaseBeforeOpLock(357)) - We are resetting the hadoop caller context for thread HiveServer2-HttpHandler-Pool: Thread-82257

2017-10-02 00:00:36,211 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: session.HiveSessionImpl (HiveSessionImpl.java:acquireAfterOpLock(333)) - We are setting the hadoop caller context to de11c83f-e087-406e-b715-dd6ba7148cfe for thread HiveServer2-HttpHandler-Pool: Thread-82257

2017-10-02 00:00:36,212 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: session.HiveSessionImpl (HiveSessionImpl.java:releaseBeforeOpLock(357)) - We are resetting the hadoop caller context for thread HiveServer2-HttpHandler-Pool: Thread-82257

2017-10-02 00:03:36,081 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doPost(145)) - Could not validate cookie sent, wi/var/log/hadoop/hive/hiveserver2.log

What I might be missing here?