Created on 08-30-2016 09:23 AM - edited 09-16-2022 03:37 AM
Since installing Hive 1.2, we are seeing several "Invalid OperationHandle" errors while running or retrieving results from hive queries, in both Hue 3.9 and 3.10.
On HiveServer2, we are seeing logs of this type correlating with the above errors:
thrift.ThriftCLIService (ThriftCLIService.java:deleteContext(118)) - Session disconnected without closing properly, close it now
and we suspect this is related to HIVE-13415 and HIVE-14227, i.e. that some connections used by thrift get closed and cause the destruction of the attached session on Hive side.
Does anybody have a fix or work-around for this behaviour?
Cheers!
Created 09-07-2016 05:44 AM
Created 09-07-2016 06:45 AM
We use Hue from master and Hive from another provider. Hue is run under apache mod_wsgi.
We started having this problem while running 3.9 after upgrading Hive to 1.2. Upgrading Hue to 3.10 didn't fix it.
Our current understading is that under some conditions a network connection between Hue and HiveServer2 used by thrift calls gets closed and this causes the corresponding Hive session to get dropped, as per behaviour introduced in HIVE-9601. This makes all related operations unavailable.
BTW, in the related HIVE-14227 story Hue is mentioned a couple of times.
Waiting for a solution or upgrade on the Hive side, we are investigating if anything in the setup or code of Hue can avoid or mitigate the problem.
We can show the problem with the following commands in hue shell:
from beeswax.server import dbms
import beeswax.api
import desktop.lib.thrift_util
from django.contrib.auth.models import User
server = dbms.get_query_server_config('beeswax')
user = User.objects.get(username='my_username')
db = dbms.get(user, server)
session = db.open_session(user)
history = db.execute_statement("select 1")
handle = history.get_handle()
result = db.fetch(handle, rows=1, start_over=True)
clients = desktop.lib.thrift_util._connection_pool.pooldict.values()[0].queue
client = clients[-1]
client.transport._trans.close()
result = db.fetch(handle, rows=1, start_over=True)
Any help or useful pointer would be really appreciated.
Cheers
Created 09-14-2016 09:15 AM
Created 09-19-2016 07:15 AM
When Hue is run from the supervisor/cherrypy, the problem does not occur. We can still reproduce it by stopping and restarting the service, which causes messages "Session disconnected without closing properly, close it now" to appear in HiveServer2 logs.
Probably the lifecycle of wsgi processes under apache causes transport connections to be closed every now and then, triggering the deletion of session data on HiveServer2's side.
As far as we know, workarounds are:
- avoid restarting Hue processes, e.g. by running Hue via the cherrypy web server;
- change the thrift transport from binary to http in both Hue and HiveServer2 (we haven't tested this yet).
Cheers
Created 10-02-2016 01:15 PM