Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Issue with Hive loadbalancer

Issue with Hive loadbalancer

Explorer

Hi Team,

 

Few weeks before I had a dedicated Hiveserver and configured another Hiveserver in slavenode. But the problem is the traffic was directed only to the dedicated Hiveserver and not the other. So I have configured HAProxy in masternode which now acts as a loadbalancer and configured 3 dedicated Hiveserver. Now the traffic is being routed to the all the server. But the hive query that is being executed from hue is getting expired withing 4 to 5 Min. When I give one of the dedicated Hiveserver IP and Port in the hive loadbalancer configuration the query is there for an hour. But when I give the loadbalancer IP and Port in hive loadbalancer configuration thats when the hive query is geting expired soon.

Is there some change that need to be done in haproxy config in order to make sure all the 3 dedicated Hiveserver can withstand query for a specified time. If so that will be much helpful.

 

 

Thanks in advance.

7 REPLIES 7

Re: Issue with Hive loadbalancer

Guru
Hi @corestack,

What's the timeout setting for LB and the timeout value for HS2? They should be in sync.

You also need to make sure that the LB uses stickiness config so that Hue's connection to HS2 won't jump from one to another, this will cause issues when running Hive query in Hue. One client should always goes to the same backend HS2.

Cheers
Eric

Re: Issue with Hive loadbalancer

Explorer

Hi Eric,

 

Among the 3 hive server the configuration that was given in clouderamanager are,

hive.server2.idle.operation.timeout 30Min
hive.server2.idle.session.timeout 1Hr

Is there any other timeout value that should be checked?

 

I found that all 3 server has Unexpected Exits Monitoring Period 5Min & Pause Duration Monitoring Period 5Min, is this has something to do with the hive query expiration?

 

In HAProxy config,

 

listen beeline :10001
      mode tcp

      option tcplog
      balance roundrobin
      stick-table type ip size 200k expire 120m
      stick on src

 

      server hiveserver2a x.x.x.x:10000 cookie hiveserver1 check
      server hiveserver2b x.x.x.x:10000 cookie hiveserver2 check
      server hiveserver2c x.x.x.x:10000 cookie hiveserver3 check

 

Is there someting else that should be added?

 

Regards,

C

Highlighted

Re: Issue with Hive loadbalancer

Explorer
Hi Team, Is there any solution for my issue. I'm still facing issue with my load balancer. Please suggest some solution. As this is very priority for my work. Regards, C

Re: Issue with Hive loadbalancer

Guru
Hi @corestack,

Sorry for the late reply, what's the settings under below section in HAProxy?

defaults
....
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000

Especially the "timeout client" and "timeout server", they should match with the session timeout value in HS2.

Cheers
Eric

Re: Issue with Hive loadbalancer

Explorer

Hi,

 

The timeout config that I have given in HAproxy config file are default value.

timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000

 

But for hive, the config for Idle Session Timeout  is 1hr.

 

So now I have to change the config as

timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1h
timeout server 1h
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000

 

Is this correct? Should we give as 1h or 1hr?

 

Thanks,

C

Re: Issue with Hive loadbalancer

Guru
Yes, should be 1h, see below possible formats for timeout:

- us : microseconds. 1 microsecond = 1/1000000 second
- ms : milliseconds. 1 millisecond = 1/1000 second. This is the default.
- s : seconds. 1s = 1000ms
- m : minutes. 1m = 60s = 60000ms
- h : hours. 1h = 60m = 3600s = 3600000ms
- d : days. 1d = 24h = 1440m = 86400s = 86400000ms

Please give it a try and see it if helps.

Cheers
Eric

Re: Issue with Hive loadbalancer

Explorer

Hi,

 

Thanks a lot. This is very helpful.

 

But, in HAProxy config there are 3 server that are given. I tested with 10 user, the query are withstanding for the specified time 1h. But all the 10 user are getting routed to one hive server and not the other 2.

My assumption was roundrobin. When the first user is getting routed to the first hive server, then the second user to another server. For this is there any config to be given.

I want to load to be splitted with specified time 1h.

 

 

Thanks,

C