Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Not able to connect to hiveserver2 to access hivetables with python

Highlighted

Re: Not able to connect to hiveserver2 to access hivetables with python

Guru
hive.server2.thrift.port is the one you should connect to, which is 10000, NOT 10002, please try with 10000 port and see how you go.

Cheers
Eric

Re: Not able to connect to hiveserver2 to access hivetables with python

Explorer

@EricL 

netstat -tlnp . (hive is running on 10001 ND 10002)  eventhough hive-site.xml conatains:

 

<property>
<name>hive.server2.thrift.port</name>
<value>10000</value>
</property>
<property>
<name>hive.server2.webui.port</name>
<value>10002</value>
</property>

 

 

proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 0 0.0.0.0:9083 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:8670 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:9026 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:9027 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:21000 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:9000 0.0.0.0:* LISTEN 2412/java-
tcp 0 0 0.0.0.0:9083 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:8670 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:9026 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:9027 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:21000 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:9000 0.0.0.0:* LISTEN 2412/java

tcp 0 0 0.0.0.0:10001 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:10002 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:50070 0.0.0.0:* LISTEN 2412/java
tcp 0 0 0.0.0.0:50070 0.0.0.0:* LISTEN 2412/java

 

 

With 10000 , I receive this error:

 

No handlers could be found for logger "thrift.transport.TSocket"
Traceback (most recent call last):
File "cloudera.py", line 22, in <module>
output = hiveconnection(host_name, port, user,password, database)
File "cloudera.py", line 14, in hiveconnection
database=database, auth='CUSTOM')
File "/usr/lib/python2.7/site-packages/pyhive/hive.py", line 192, in __init__
self._transport.open()
File "/usr/lib/python2.7/site-packages/thrift_sasl/__init__.py", line 61, in open
self._trans.open()
File "/usr/lib64/python2.7/site-packages/thrift/transport/TSocket.py", line 113, in open
raise TTransportException(TTransportException.NOT_OPEN, msg)
thrift.transport.TTransport.TTransportException: Could not connect to any of [('::1', 10000, 0, 0), ('127.0.0.1', 10000)]

 

and with 10001, error is:

 

  File "cloudera.py", line 22, in <module>
output = hiveconnection(host_name, port, user,password, database)
File "cloudera.py", line 14, in hiveconnection
database=database, auth='CUSTOM')
File "/usr/lib/python2.7/site-packages/pyhive/hive.py", line 192, in __init__
self._transport.open()
File "/usr/lib/python2.7/site-packages/thrift_sasl/__init__.py", line 80, in open
status, payload = self._recv_sasl_message()
File "/usr/lib/python2.7/site-packages/thrift_sasl/__init__.py", line 101, in _recv_sasl_message
payload = read_all_compat(self._trans, length)
File "/usr/lib/python2.7/site-packages/thrift_sasl/six.py", line 31, in <lambda>
read_all_compat = lambda trans, sz: trans.readAll(sz)
File "/usr/lib64/python2.7/site-packages/thrift/transport/TTransport.py", line 60, in readAll
chunk = self.read(sz - have)
File "/usr/lib64/python2.7/site-packages/thrift/transport/TSocket.py", line 132, in read
message='TSocket read 0 bytes')

 

Re: Not able to connect to hiveserver2 to access hivetables with python

Guru
Before you try to connect from PyHive, please confirm if you can do so through beeline and make sure HS2 is functioning OK. Again, HS2 port is 10000 based on your config, so you should connect to 10000 from client side.

Do you see any errors from HS2 log? Try beeline and what does it say?

Cheers

Re: Not able to connect to hiveserver2 to access hivetables with python

Explorer

@EricL 

 

I am facing the following problems when connecting to port 10000 and 10001 and Hive server2 logs goes as follows

 

 

!connect jdbc:hive2://localhost:10000 hive hive
Connecting to jdbc:hive2://localhost:10000
19/09/23 10:22:05 [main]: WARN jdbc.HiveConnection: Failed to connect to localhost:10000
Could not open connection to the HS2 server. Please check the server URI and if the URI is correct, then ask the administrator to check the server status.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=0)

 

HiveServer2 log

 

It does not even hit Hiveserver2 , hence there are no logs

 

 

beeline> !connect jdbc:hive2://localhost:10001 hive hive
Connecting to jdbc:hive2://localhost:10001
19/09/23 10:22:21 [main]: WARN jdbc.HiveConnection: Failed to connect to localhost:10001
Unknown HS2 problem when communicating with Thrift server.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10001: Invalid status 72 (state=08S01,code=0)

 

HiveServer2 log:

 

2019-09-23T10:24:08,767 WARN [HiveServer2-HttpHandler-Pool: Thread-59] http.HttpParser: Illegal character 0x1 in state=START for buffer HeapByteBuffer@63bc8470[p=1,l=25,c=8192,r=24]={\x01<<<\x00\x00\x00\x05PLAIN\x05\x00\x00\x00\n\x00hive\x00hive>>>\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2019-09-23T10:24:08,767 WARN [HiveServer2-HttpHandler-Pool: Thread-59] http.HttpParser: bad HTTP parsed: 400 Illegal character 0x1 for HttpChannelOverHttp@751d099b{r=0,c=false,a=IDLE,uri=null}