Knox Hive HA configuration does not work when knox switch between Hiveserver2 instances.

theyaa — Fri, 19 Jan 2018 00:10:33 GMT

I have configured knox to work with two hiveserver2 instances and during my testing I connected to hive through knox using beeline. Then stopped the current hiveserver2 and ran a query in beeline. I can see that knox did fail over to the second one, but my query through a 500 error. Looking into hiveserver2 log I saw the following error.

2018-01-17 14:14:38,611 WARN  servlet.ServletHandler (ServletHandler.java:doHandle(546)) - /cliservice

java.lang.IllegalArgumentException: Invalid sign, original = Qp+dxp0/PGiidNY7TWAHYo4nlJs= current = XaZLVoAJAF+88lAl9Z/9VYXWeg0=

        at org.apache.hive.service.CookieSigner.verifyAndExtract(CookieSigner.java:85)

        at org.apache.hive.service.cli.thrift.ThriftHttpServlet.getClientNameFromCookie(ThriftHttpServlet.java:253)

        at org.apache.hive.service.cli.thrift.ThriftHttpServlet.validateCookie(ThriftHttpServlet.java:309)

        at org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:142)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)

        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)

        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479)

        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225)

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)

        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)

        at org.eclipse.jetty.server.Server.handle(Server.java:349)

        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449)

        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:925)

        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:952)

        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)

        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)

        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)

        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

The following article talks about the same exact issue : https://community.hortonworks.com/content/supportkb/150288/error-opening-sessionorgapachethrifttransportttran.html But, That solution is not optimal as it defeats the purpose of have HA configuration if I have to bounce knox server everytime a hiveserver2 goes down. Also the Jira ticket mentioned in it does not exist.

Does anyone has a better solution or suggestion?

Re: Knox Hive HA configuration does not work when knox switch between Hiveserver2 instances.

smore — Fri, 19 Jan 2018 02:51:14 GMT

Hello @Theyaa Matti this issue should be fixed in the current 0.14.0 Knox release.

In case you do not want to upgrade you can set the following properties in gateway-site.xml

"gateway.httpclient.connectionTimeout" and "gateway.httpclient.socketTimeout"

to appropriate value, say 300.

question Knox Hive HA configuration does not work when knox switch between Hiveserver2 instances. in Support Questions

Knox Hive HA configuration does not work when knox switch between Hiveserver2 instances.

Re: Knox Hive HA configuration does not work when knox switch between Hiveserver2 instances.