Hue: Invalid OperationHandle errors

[#1]

Since installing Hive 1.2, we are seeing several "Invalid OperationHandle" errors while running or retrieving results from hive queries, in both Hue 3.9 and 3.10.
On HiveServer2, we are seeing logs of this type correlating with the above errors:

thrift.ThriftCLIService (ThriftCLIService.java:deleteContext(118)) - Session disconnected without closing properly, close it now

and we suspect this is related to HIVE-13415 and HIVE-14227, i.e. that some connections used by thrift get closed and cause the destruction of the attached session on Hive side.

Does anybody have a fix or work-around for this behaviour?

Cheers!

[#2]

Are you using the Hive from CDH or somewhere else? (we never saw this
problem so far)

[#3]

We use Hue from master and Hive from another provider. Hue is run under apache mod_wsgi.

We started having this problem while running 3.9 after upgrading Hive to 1.2. Upgrading Hue to 3.10 didn't fix it.

Our current understading is that under some conditions a network connection between Hue and HiveServer2 used by thrift calls gets closed and this causes the corresponding Hive session to get dropped, as per behaviour introduced in HIVE-9601. This makes all related operations unavailable.

BTW, in the related HIVE-14227 story Hue is mentioned a couple of times.

Waiting for a solution or upgrade on the Hive side, we are investigating if anything in the setup or code of Hue can avoid or mitigate the problem.

We can show the problem with the following commands in hue shell:

from beeswax.server import dbms
import beeswax.api
import desktop.lib.thrift_util
from django.contrib.auth.models import User
server = dbms.get_query_server_config('beeswax')
user = User.objects.get(username='my_username')
db = dbms.get(user, server)
session = db.open_session(user)
history = db.execute_statement("select 1")
handle = history.get_handle()
result = db.fetch(handle, rows=1, start_over=True)
clients = desktop.lib.thrift_util._connection_pool.pooldict.values()[0].queue
client = clients[-1]
client.transport._trans.close()
result = db.fetch(handle, rows=1, start_over=True)

Any help or useful pointer would be really appreciated.

Cheers

[#4]

Not giving not much help there, but maybe diff your Hive with the one
shipped from CDH as we did not see this problem so far in Hue.

[#5]

When Hue is run from the supervisor/cherrypy, the problem does not occur. We can still reproduce it by stopping and restarting the service, which causes messages "Session disconnected without closing properly, close it now" to appear in HiveServer2 logs.

Probably the lifecycle of wsgi processes under apache causes transport connections to be closed every now and then, triggering the deletion of session data on HiveServer2's side.

As far as we know, workarounds are:

 - avoid restarting Hue processes, e.g. by running Hue via the cherrypy web server;

 - change the thrift transport from binary to http in both Hue and HiveServer2 (we haven't tested this yet).

Cheers

[#6]

If you restart Hue, HiveServer2 will indeed close the user sessions as the
connections get dropped.