Even though the VM looks fine it is probably a resource contraint on the VM that is causing this issue.     The Namenode writes each edit to its own local directory and all of the JN edits directories.  It simply sounds like the VM isn't keeping up or getting the job done in time.   Examine the contents of the JN edits directory on each and you will find that the VM does on contain all of the necessary edits.  You can manually copy the edits_* files to the VM nodes to get it back in sync and see if it happens again.  I do recommend using the same hardware for all three Master nodes that would run each JN and ZK instance.  Otherwise, you will often be found on just barely maintaining the quorum to stay running.   dfs.namenode.shared.edits.dir dfs.journalnode.edits.dir ... View more

Give it a try with the -k switch. This lets the shell know that it should authenticate with Kerberos and this is off by default. The logs also tell me that the server side was expected Kerberos auth but it failed that. I don't know why it works to the LB. Is the LB doing pass through authentication? I haven't worked with an LB in front of Impala, so I don't know if this is normal. I would hope that I could still query each one if needed. I am curious as I am looking to add a LB for Impala soon. ... View more

Miss the last line on your last message. If the client, and the user, are in a third domain then you need to add that rule and add the new realm to the trusted realms list. ... View more

See the below link. It has a good chart to help understand the rules. The ones you have in place cover a Kerberos principal with 3 parts. These are the host and service principals, hdfs/hostname.realm.com@realm.com, for both realms. You need something more along the lines of [1:$1]. This will strip of the domain name. The remain username will be left and what is used to access resources across the cluster. https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_sg_kerbprin_to_sn.html ... View more

Do all of these resolve correctly from all three nodes? use this cmd to verify: python -c "import socket; print socket.getfqdn(); print socket.gethostbyname(socket.getfqdn())" name_node-m0.my_domain name_node-m1.my_domain hadoop-01.my_domain ... View more

Kerberos and LDAP for Impala are different. MS AD provide both but configuring Hadoop and Impala for Kerberos uses that for auth and not LDAP. Based on what you said and the message from using the base command I'd say you are using Kerberos. Below is the command you need to use. impala-shell -k -i FQDN --ssl ... View more

Something is wrong with the NN process. Restart it and tail the log to see what exception pops up. Also ensure that the process is listening on port 8020. ... View more