Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive server restart autonomously and TTransportException: TSocket read 0 bytes

Highlighted

Hive server restart autonomously and TTransportException: TSocket read 0 bytes

New Contributor

Dear all,

i am experiencing a weird behavior on Hive Server HA (HDP 3.1). I am contacting the Hive Server from a Python program exploiting PyHive 0.6.1 and Thrift 0.11.0 I am subsequently performing queries on Hive and every time the program stops on a different query throwing the following exception:

in hiveAsync cursor.execute(q, async=True) File ".../lib/python2.7/site-packages/pyhive/hive.py", line 364, in execute response = self._connection.client.ExecuteStatement(req) File ".../lib/python2.7/site-packages/TCLIService/TCLIService.py", line 280, in ExecuteStatement return self.recv_ExecuteStatement() File ".../lib/python2.7/site-packages/TCLIService/TCLIService.py", line 292, in recv_ExecuteStatement (fname, mtype, rseqid) = iprot.readMessageBegin() File ".../python2.7/site-packages/thrift/protocol/TBinaryProtocol.py", line 134, in readMessageBegin sz = self.readI32() File ".../lib/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py", line 217, in readI32 buff = self.trans.readAll(4) File ".../lib/python2.7/site-packages/thrift/transport/TTransport.py", line 60, in readAll chunk = self.read(sz - have) File ".../lib/python2.7/site-packages/thrift_sasl/__init__.py", line 166, in read self._read_frame() File ".../lib/python2.7/site-packages/thrift_sasl/__init__.py", line 170, in _read_frame header = self._trans.readAll(4) File ".../lib/python2.7/site-packages/thrift/transport/TTransport.py", line 60, in readAll chunk = self.read(sz - have) File ".../lib/python2.7/site-packages/thrift/transport/TSocket.py", line 132, in read message='TSocket read 0 bytes') TTransportException: TSocket read 0 bytes 

It is stopping always on a different query but it is always stopping somewhere. I found also that when the above happen the Hive Server is autonomously restarting. But in the hiveserver.log i didn't find any errors:

2019-02-15T15:03:07,588 INFO  [e74552d7-15b5-4199-8734-bb6ff7a830f1 HiveServer2-Handler-Pool: Thread-127]: lockmgr.DbTxnManager (:()) - Opened txnid:1226
2019-02-15T15:03:07,589 INFO  [e74552d7-15b5-4199-8734-bb6ff7a830f1 HiveServer2-Handler-Pool: Thread-127]: parse.CalcitePlanner (:()) - Starting Semantic Analysis
2019-02-15T15:03:07,712 INFO  [e74552d7-15b5-4199-8734-bb6ff7a830f1 HiveServer2-Handler-Pool: Thread-127]: parse.CalcitePlanner (:()) - Completed phase 1 of Semantic Analysis
2019-02-15T15:03:07,712 INFO  [e74552d7-15b5-4199-8734-bb6ff7a830f1 HiveServer2-Handler-Pool: Thread-127]: parse.CalcitePlanner (:()) - Get metadata for source tables
2019-02-15T15:03:07,712 INFO  [e74552d7-15b5-4199-8734-bb6ff7a830f1 HiveServer2-Handler-Pool: Thread-127]: parse.CalcitePlanner (:()) - Get metadata for subqueries
2019-02-15T15:03:07,712 INFO  [e74552d7-15b5-4199-8734-bb6ff7a830f1 HiveServer2-Handler-Pool: Thread-127]: parse.CalcitePlanner (:()) - Get metadata for destination tables
2019-02-15T15:03:07,726 WARN  [e74552d7-15b5-4199-8734-bb6ff7a830f1 HiveServer2-Handler-Pool: Thread-127]: parse.BaseSemanticAnalyzer (:()) - Dynamic partitioning is used; only validating 0 columns
2019-02-15T15:03:07,726 INFO  [e74552d7-15b5-4199-8734-bb6ff7a830f1 HiveServer2-Handler-Pool: Thread-127]: parse.CalcitePlanner (:()) - Completed getting MetaData in Semantic Analysis
2019-02-15T15:03:07,754 INFO  [e74552d7-15b5-4199-8734-bb6ff7a830f1 HiveServer2-Handler-Pool: Thread-127]: ql.Context (:()) - New scratch dir is hdfs://testHDP/tmp/hive/hive/e74552d7-15b5-4199-8734-bb6ff7a830f1/hive_2019-02-15_15-03-07_193_2046998341793917224-1
2019-02-15T15:04:41,859 INFO  [main]: conf.HiveConf (HiveConf.java:findConfigFile(187)) - Found configuration file file:/etc/hive/3.1.0.0-78/0/hive-site.xml
2019-02-15T15:04:42,850 INFO  [main]: server.HiveServer2 (HiveStringUtils.java:startupShutdownMessage(767)) - STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting HiveServer2
STARTUP_MSG:   host = hdptest1/192.168.227.201
STARTUP_MSG:   args = [--hiveconf, hive.aux.jars.path=file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar]
STARTUP_MSG:   version = 3.1.0.3.1.0.0-78 

Moreover, i tested the very same software exploiting an Hive Server (Not-HA) (HDP 2.6.0) and it is working perfectly.

The HDP3 and HDP2.6 have the very same configurations for Hive, they have been installed with a blueprint.

What could be the cause of the restart?

Thanks