Impala relies on hivemetastore for persisting table metadata. HMS inturn uses db to persist metadata. So if the db's filesystem is out of space, request from hms to persist metadata will be denied and yes if the space on the db is not freed up then request from hms will be failed and eventually impala also fails the query ... View more

First of all. I am sorry for getting back late on this question. One of the key factor of kerberos authentication is its reliability on DNS reverse resolution. Quote from MIT KDC - https://web.mit.edu/kerberos/krb5-1.4/krb5-1.4.4/doc/krb5-admin/Getting-DNS-Information-Correct.html ---- [1] Getting DNS Information Correct Several aspects of Kerberos rely on name service. In order for Kerberos to provide its high level of security, it is less forgiving of name service problems than some other parts of your network. It is important that your Domain Name System (DNS) entries and your hosts have the correct information. ---- Lets say the virtual ip as haproxy.com And loadbalancers are running on below nodes haproxy1.com - 10.0.0.1 haproxy2.com - 10.0.0.2 haproxy3.com - 10.0.0.3 impalad running on nodes impalad1 - 10.0.0.4 impalad2 - 10.0.0.5 impalad3 - 10.0.0.6 ==== # Forward resolution configs for DNS haproxy1.com IN A 10.0.0.1 haproxy.com IN CNAME haproxy1.com ==== Now haproxy.com resolves to ip 10.0.0.1 reverse resolution of ip 10.0.0.1 will result in answer haproxy1.com. This breaks the expectation of kerberos [1] authentication, so service ticket request will fail when you run impala-shell -i haproxy.com [2] So our aim is to achieve DNS resolution like this. haproxy.com -> 10.0.0.1 10.0.0.1 -> haproxy.com We can now alter the reverse resolution of DNS to achieve this Reverse zone configuration: ==== inverse-query-haproxy1.com IN PTR haproxy1.com 10.0.0.1 IN CNAME inverse-query-haproxy1.com ==== With these above set of configs we can achieve forward and reverse resolution as expected in [2] Caution Note: If you run CM agents on one of the proxy machines, i.e. its a part of the cluster, its identity will have to change permanently to the VIP name, because reverse DNS will now never show the original hostname, which could cause other services to have issues unless listening_hostname is configured to use the VIP name. Ideally the haproxy machine should not be added as part of the cluster in CM hosts control, to avoid this from happening - it should be a standalone box. ... View more

@bushnoh This is normal. Once you setup loadbalancer infront of impalad, the impalad will expose itself through the service principal name(SPN) of the loadbalancer to the external client. If you check the varz page of individual impalad, you can notice following parameters https://<impalad-hostname>:25000/varz principal ==> LB SPN be_principal ==> IMPALAD SPN This shows that impalad expects LB's SPN for clients communication whereas for internal communication[within impalad's] it uses its own SPN. be_principal --> Backend principal for internal communication. hence it is required to contact the impalad with LB's SPN. ... View more