Created on 12-10-2017 08:52 AM - edited 09-16-2022 05:37 AM
Hi All,
We have recently configured an F5 proxy for our HiveServer2 services in order to support increasing scale in clients accessing HS2 service, following this procedure:
http://www.cloudera.com/documentation/other/reference-architecture/PDF/Impala-HA-with-F5-BIG-IP.pdf
We started with session timeout of 1 hour but we quickly found out that for long running Hive queries the proxy is killing the connection once the timeout is reached.
I assumed that Hive connections using beeline\PyHive\Cloudera ODBC\etc. are aware of the fact that Hive processes are usually long, hence, should implement a keep alive mechanism to keep the connection active until the Hive process finish.
To my surprise none of the Hive clients we are using, implements such a keep alive mechanism and only when we increased the proxy's session timeout to be longer than our longest Hive query, our long Hive processes stopped being killed by the proxy.
Digging a bit deeper to HS2 configuration I found the parameter hive.server2.idle.session.timeout which is set to 12 hours, and I understood why before using the proxy all our Hive processes worked perfectly.
Our network guys said that setting session timeout at the proxy level to be 12 hours is not best practice and that the clients accessing HS2 should implement keep alive mechanism.
Is there a better way addressing this keep alive issue? Or setting the proxy's session timeout to be bigger than the longest query is the way to go?
Best,
Eyal
Created 12-16-2017 07:23 PM
Created 12-17-2017 01:06 AM
Hi Eric,
In order to setup the proxy for HS2 using F5 I followed the exact steps described in the following Cloudera "Impala HA with F5 BIG-IP" manual:
http://www.cloudera.com/documentation/other/reference-architecture/PDF/Impala-HA-with-F5-BIG-IP.pdf
I known the above manual is for setting up Proxy for Impala but I don't see how for HS2 the steps should change.
If this is an offical Cloudera manual, how come only when increaseing the F5 proxy timeout itself to 12H (from 1H) both Impala & HS2 session running long queries (longer than 1H) stopped being killed?
Was the above procedure made against a specific F5 version?
Best,
Eyal
Created 01-07-2018 05:23 AM
Hi Eric,
We followed Cloudera's link to how to set up a LB using F5 and still had to increase the session timeout to 12 hours to stop long processes from failing, are we missing anything?
Do you know of other companies that implemented the LB using F5 and had similar issues?
Best,
Eyal
Created 12-17-2017 02:56 AM
Created 12-17-2017 03:00 AM