Unable to connect to Impala using Cloudera ODBC Driver with HAproxy and Elastic Load balancer

[#1]

Hi All,

We need your support on issues we are facing currently.

We are trying to connect to Impala using Cloudera ODBC driver with HA Proxy and Elastic Load Balancer. it is failing with below error.

FAILED!

[Cloudera][DriverSupport] (1110) Unexpected response received from server. Please ensure the server host and port specified for the connection are correct and confirm if SSL should be enabled for the connection.

Cluster Details -

CDH Version 6.1

Clouder ODBC Driver 2.6

Impala daemons run on a machine dn1,dn2, master2

ELB points to only master2 daemon for now.

HAProxy points to only master2 daemon for now.

Cluster is kerberos enabled.

Let us assume

ELB Name - elb-test-odbc.com

HAProxy name  - haproxy-name.com

Below combination works -

we put ELB Name in HOST and impala daemon name in HOST FQDN(under kerberos) name it works.

Below combination DOES NOT works -

we put ELB Name in HOST and haproxy-name.com in HOST FQDN(under kerberos) name it does not work.

We actaully want to achieve the resilency so that we are not depenedent on single impala daemon.

Can someone please let us know how can we make it happen. any help on this would be greatly appreciated.

Regards

Ajay chaudhary

[#2]

What value are you noticing for  principal in the impalad-webui varz page?

https://<impalad-hostname>:25000/varz

Did you add the ELB or Haproxy details in the CM > Impala > Configuration > 

Impala Daemons Load Balancer ?

[#3]

Hi,

Thank you for your response. We have not set the load balancer property value in Impala configurations.

here is the details -

<tr >
<td> <samp>internal_principals_whitelist (string)</samp></td>
<td>(Advanced) Comma-separated list of additional usernames authorized to access Impala&apos;s internal APIs. Defaults to &apos;hdfs&apos; which is the system user that in certain deployments must access catalog server APIs.</td>
<td><samp>hdfs</samp></td>
<td><samp>hdfs</samp></td>
</tr>

<tr >
<td> <samp>be_principal (string)</samp></td>
<td>Kerberos principal for backend network connections only,overriding --principal if set. Must not be set if --principal is not set.</td>
<td><samp></samp></td>
<td><samp></samp></td>
</tr>

<tr class="active">
<td> <samp>principal (string)</samp></td>
<td>Kerberos principal. If set, both client and backend network connections will use Kerberos encryption and authentication. Kerberos will not be used for internal or external connections if this is not set.</td>
<td><samp></samp></td>
<td><samp>impala/master2-impala-20.yodlee.com@YODLEEINSIGHTS.COM</samp></td>
</tr>

Regards

Ajay chaudhary

[#4]

Please note this cluster is setup on AWS EC2 instances. ELB is created on AWS which forwards the request coming on port 25003 to EC2 machine which host HAProxy and HAProxy is setup on EC2 machine which does not host Impala daemons so it forwards the request to another EC2 machine which host Impala daemon.

Regards

Ajay 

[#5]

<td> <samp>principal (string)</samp></td>
<td><samp>impala/master2-impala-20.yodlee.com@YODLEEINSIGHTS.COM</samp></td>
Impalad will expect the client to use this SPN while client tries to connect to it. This is why it failed when you put host fqdn as haproxy

When you enable haproxy in CM > Impala > Configuration > Impalad Load Balancer name then CM will prepare a merged keytab consisting of SPN of the loadbalancer and also changes this principal field in impalad configuration to haproxy spn, After which you will be able to connect to impalad.

[#6]

Hi,

We are unable to connect to Impala daemon through Impala-shell and Hue when we update the load balancer property in Cloudera manager.

Not sure if we have any bug around this. 

if you can, Can you please pass on the steps as how to merge the existing keytab with the proxy’s keytab.

Regards

Ajay chaudhary

[#7]

What exact command you used for impala-shell?

Can you try 

impala-shell -i <haproxy-host> and let us know if it works.

Also do you have any overrides in

hue.ini,

hue_safety_valve.ini
hue_safety_valve_server.ini

If yes what values are under [impala] tag? 

Also Is there any reason behind having ELB and also Haproxy?