Can you share the full stack trace of the exception to understand further ... View more

At present Navigator doesn't support gathering lineage from NiFi - however within nifi there is lineage of flowfile. You can get the steps from this link https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.4.1.1/getting-started-with-apache-nifi/content/lineage-graph.html ... View more

49213 open("/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir/libleveldbjni-64-1-6110205147654050510.8", O_RDWR|O_CREAT|O_EXCL, 0666) = -1 EACCES (Permission denied) During this step, the script is trying to open and get file descriptor for this directory and it was denied access  /var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir/libleveldbjni-64-1-6110205147654050510.8 So far we have inspected its parent directories and haven't seen any issues with. Can we get details of this directory too ls -ln /var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir/libleveldbjni-64-1-6110205147654050510.8 stat /var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir/libleveldbjni-64-1-6110205147654050510.8 id yarn   ... View more

 The error usually happens when you try to connect to ssl enabled hs2 with plaintext connection.    a.Which version of CDH/HDP are you using? b. Can you check in HS2 logs exactly during the timestamp the error "Unable to connect to server: Invalid status 21" was reported on client. The error you notice on server side will give further clues c. Do you have SSL enabled on HS2 ? ... View more

Per https://serverfault.com/questions/647569/reverse-ssh-connection-time-out-during-banner-exchange this error may indicate network issues. If you see a socket is established using netstat on both the server and the client, there may be a firewall or packet inspection device that is preventing the SSH connection from being established   Are you able to manually ssh from oozie server machine to the remote <host> as <user>  ... View more

Hi san_t_o Thanks for adding more context.   When cleaning the directories indicated, the libraries are copied automatically or it is necessary to copy them manually? --> /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/* and also on /var/lib/ambari-agent/tmp/ I was testing in my local cluster. Apologies, I meant to clear /var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir/ and sorry for the typo in my previous comment.   Even before clearing off these directories or altering the location it would be best to review with strace once. It traces all system level calls and reviewing the last call prior to failure could give us more clues. To install strace - you can run yum -y install strace   export HADOOP_LIBEXEC_DIR=/usr/hdp/3.0.1.0-187/hadoop/libexec strace -f -s 2000 -o problematic_node /usr/hdp/3.0.1.0-187/hadoop-yarn/bin/yarn --debug --config /usr/hdp/3.0.1.0-187/hadoop/conf --daemon start nodemanager export HADOOP_LIBEXEC_DIR=/usr/hdp/3.0.1.0-187/hadoop/libexec strace -f -s 2000 -o good_node /usr/hdp/3.0.1.0-187/hadoop-yarn/bin/yarn --debug --config /usr/hdp/3.0.1.0-187/hadoop/conf --daemon start nodemanager   The file problematic_node and good_node would have the traces and can you attach/paste them here. ... View more