Impala connection question

[#1]

Hi,

I have a hadoop with one mater and two datanode, accessing hdfs can hit the master url, have impala on those two nodes as well, when connect through jdbc i cannot connect with the mater url, have to connect directly through individual node, is this by design or i missed something?

Thanks

Shannon

[#2 · Accepted solution]

Yes, this is by design. The master roles for Impala preform functions for all Impala daemons, like caching metadata from HMS and HDFS block locations, maintaining a list of available Impala daemons, etc. But they do not manage connections. Each individual Impala daemon will managed the connections made to it and act as the coordinator for those connections.

For production, I recommend putting a load balancer in front of your Impala daemons to spread that connection load across all. Otherwise, having all users connect to a single one will exhaust the memory for that Impala daemon quickly. Another option I have seen is assigning blocks of Impala daemons to specific users groups.

[#3]

Thanks.

[#4]

A related question, for running impala queries in HUE, i am hitting master, does it randomly hit one of the boxes?

[#5]

Do you mean that HUE is installed on a master node? The HUE configs will contain an [impala] section and that will contain the Impala Daemon used by HUE. It will only connect to that ImpalaD.