question Re: Load balancing HiveServer2 Over Knox in Archives of Support Questions (Read Only)

Load balancing HiveServer2 Over Knox

smartninja723 — Wed, 05 Jul 2017 22:47:34 GMT

Hi,

Using https://knox.apache.org/books/knox-0-9-0/user-guide.html, I have configured Knox topology for Hive Server2 High Availability.

I also noticed Dynamic Service Discovery Through ZooKeeper in documentation.

I see that all the queries/connections happen though only one of the HiveServer2, now if this HS2 instances down, I notice that connections/queries happen through another instance of HS2.

My question is : In the busy cluster, when we have multiple HS2 servers installed, is it possible to load balance (possibly round robin) so that one server does not get overloaded? If yes, how?

Regards,

Re: Load balancing HiveServer2 Over Knox

manish1 — Wed, 05 Jul 2017 23:52:02 GMT

Your Network team can create a DNS Name and can add both the hiverserver2 IPs under this DNS in either round robin or load-balance mode. Then, you can use the DNS name and 10000 port in the Knox configuration to submit hive jobs. It should provide what you are looking for. Please let me know.

Re: Load balancing HiveServer2 Over Knox

smartninja723 — Thu, 06 Jul 2017 16:07:13 GMT

Thank you @Manish Gupta,

This is something we can try by having a proxy configured to have load balancing for HS2. Also would like to understand are they changes to be made for Zookeeper ?

Is there any other way by which, without using non HDP component or external network changes, we can achieve load balancing of HS2.

Regards,

Re: Load balancing HiveServer2 Over Knox

manish1 — Thu, 06 Jul 2017 21:11:05 GMT

@Smart Solutions

No changes would be required for ZooKeeper. Unfortunatley, load-balancing of Hive Server2 feature is not available in HDP currently. Hopefully in future release.

As always, if you find this post helpful, don't forget to "accept" answer.

Re: Load balancing HiveServer2 Over Knox

smartninja723 — Fri, 07 Jul 2017 17:01:48 GMT

Thanks @Manish Gupta for clarification. This is helpful.

Cheers.