Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

Load balancing HiveServer2 Over Knox

Expert Contributor

Hi,

Using https://knox.apache.org/books/knox-0-9-0/user-guide.html, I have configured Knox topology for Hive Server2 High Availability.

I also noticed Dynamic Service Discovery Through ZooKeeper in documentation.

I see that all the queries/connections happen though only one of the HiveServer2, now if this HS2 instances down, I notice that connections/queries happen through another instance of HS2.

My question is : In the busy cluster, when we have multiple HS2 servers installed, is it possible to load balance (possibly round robin) so that one server does not get overloaded? If yes, how?

Regards,

SS

1 ACCEPTED SOLUTION

Expert Contributor

Your Network team can create a DNS Name and can add both the hiverserver2 IPs under this DNS in either round robin or load-balance mode. Then, you can use the DNS name and 10000 port in the Knox configuration to submit hive jobs. It should provide what you are looking for. Please let me know.

View solution in original post

4 REPLIES 4

Expert Contributor

Your Network team can create a DNS Name and can add both the hiverserver2 IPs under this DNS in either round robin or load-balance mode. Then, you can use the DNS name and 10000 port in the Knox configuration to submit hive jobs. It should provide what you are looking for. Please let me know.

Expert Contributor

Thank you @Manish Gupta,

This is something we can try by having a proxy configured to have load balancing for HS2. Also would like to understand are they changes to be made for Zookeeper ?

Is there any other way by which, without using non HDP component or external network changes, we can achieve load balancing of HS2.

Regards,

Expert Contributor

@Smart Solutions

No changes would be required for ZooKeeper. Unfortunatley, load-balancing of Hive Server2 feature is not available in HDP currently. Hopefully in future release.

As always, if you find this post helpful, don't forget to "accept" answer.

Expert Contributor

Thanks @Manish Gupta for clarification. This is helpful.

Cheers.