The error usually happens when you try to connect to ssl enabled hs2 with plaintext connection.    a.Which version of CDH/HDP are you using? b. Can you check in HS2 logs exactly during the timestamp the error "Unable to connect to server: Invalid status 21" was reported on client. The error you notice on server side will give further clues c. Do you have SSL enabled on HS2 ? ... View more

Per https://serverfault.com/questions/647569/reverse-ssh-connection-time-out-during-banner-exchange this error may indicate network issues. If you see a socket is established using netstat on both the server and the client, there may be a firewall or packet inspection device that is preventing the SSH connection from being established   Are you able to manually ssh from oozie server machine to the remote <host> as <user>  ... View more

Hi san_t_o Thanks for adding more context.   When cleaning the directories indicated, the libraries are copied automatically or it is necessary to copy them manually? --> /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/* and also on /var/lib/ambari-agent/tmp/ I was testing in my local cluster. Apologies, I meant to clear /var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir/ and sorry for the typo in my previous comment.   Even before clearing off these directories or altering the location it would be best to review with strace once. It traces all system level calls and reviewing the last call prior to failure could give us more clues. To install strace - you can run yum -y install strace   export HADOOP_LIBEXEC_DIR=/usr/hdp/3.0.1.0-187/hadoop/libexec strace -f -s 2000 -o problematic_node /usr/hdp/3.0.1.0-187/hadoop-yarn/bin/yarn --debug --config /usr/hdp/3.0.1.0-187/hadoop/conf --daemon start nodemanager export HADOOP_LIBEXEC_DIR=/usr/hdp/3.0.1.0-187/hadoop/libexec strace -f -s 2000 -o good_node /usr/hdp/3.0.1.0-187/hadoop-yarn/bin/yarn --debug --config /usr/hdp/3.0.1.0-187/hadoop/conf --daemon start nodemanager   The file problematic_node and good_node would have the traces and can you attach/paste them here. ... View more

Please share the full exception from beeline, if full exception is not available try with --verbose on the beeline command ... View more

Hi @san_t_o I wanted to validate once if the mounts and permission are same. Those look exactly same except the additional "sunit=512,swidth=512" for /var mount but that cant be an issue. At this point its unclear what exactly is denied permission. What is the selinux status? Is it disabled on both working and nonworking node. Please run below command on both working and non-working node getenforce   If its same in both nodes, Can you clear the entries under this directory /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/* and also on /var/lib/ambari-agent/tmp/ If that doesn't work, Can you try pointing JAVA_LIBRARY_PATH in yarn-env.sh to a different directory   How exactly are you starting nodemanager? Is it by running commands manually? If yes can you try running the command with strace -f -s 2000 <command> [Strace allows to capture all syscalls and we can get more debug info] ... View more

Could you please share the result of below commands from one problematic and a working node 1. namei -l /var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir 2. mount ... View more