Knox Hive HA configuration does not work when knox switch between Hiveserver2 instances.

[#1]

I have configured knox to work with two hiveserver2 instances and during my testing I connected to hive through knox using beeline. Then stopped the current hiveserver2 and ran a query in beeline. I can see that knox did fail over to the second one, but my query through a 500 error. Looking into hiveserver2 log I saw the following error.

2018-01-17 14:14:38,611 WARN  servlet.ServletHandler (ServletHandler.java:doHandle(546)) - /cliservice

java.lang.IllegalArgumentException: Invalid sign, original = Qp+dxp0/PGiidNY7TWAHYo4nlJs= current = XaZLVoAJAF+88lAl9Z/9VYXWeg0=

        at org.apache.hive.service.CookieSigner.verifyAndExtract(CookieSigner.java:85)

        at org.apache.hive.service.cli.thrift.ThriftHttpServlet.getClientNameFromCookie(ThriftHttpServlet.java:253)

        at org.apache.hive.service.cli.thrift.ThriftHttpServlet.validateCookie(ThriftHttpServlet.java:309)

        at org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:142)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)

        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)

        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479)

        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225)

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)

        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)

        at org.eclipse.jetty.server.Server.handle(Server.java:349)

        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449)

        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:925)

        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:952)

        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)

        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)

        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)

        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

The following article talks about the same exact issue : https://community.hortonworks.com/content/supportkb/150288/error-opening-sessionorgapachethrifttran... But, That solution is not optimal as it defeats the purpose of have HA configuration if I have to bounce knox server everytime a hiveserver2 goes down. Also the Jira ticket mentioned in it does not exist.

Does anyone has a better solution or suggestion?

[#2 · Accepted solution]

Hello @Theyaa Matti this issue should be fixed in the current 0.14.0 Knox release.

In case you do not want to upgrade you can set the following properties in gateway-site.xml

"gateway.httpclient.connectionTimeout" and "gateway.httpclient.socketTimeout"

to appropriate value, say 300.