It worked 1/3 test. it works sometimes, but not always: I checked all nodemanager directories, they all looked same owner:grp: ls -ld /hadoop/yarn/local drwxr-xr-x 6 yarn hadoop 78 Oct 6 12:56 /hadoop/yarn/local ls -ld /appl/hadoop/yarn/local drwxrwxrwx 5 yarn hadoop 57 Oct 6 12:56 /appl/hadoop/yarn/local ls -ld /usr/db/hadoop/yarn/local drwxr-xr-x 5 yarn hadoop 57 Oct 6 12:56 /usr/db/hadoop/yarn/local ls -ld /usr/local/opt/hadoop/yarn/local drwxr-xr-x 5 yarn hadoop 57 Oct 6 12:58 /usr/local/opt/hadoop/yarn/local ... View more

I noticed /appl/hadoop/yarn on different nodes had different owner/grp. I made them all to yarn:hadoop before: [yarn@vlmazgrpmaster ~]$ ls -ld /appl/hadoop/yarn drwxrwxrwx 4 root hadoop 30 Aug 24 14:10 /appl/hadoop/yarn [e1009156@vlmazgrpdata1 ~]$ ls -ld /appl/hadoop/yarn drwxrwxrwx 4 root root 30 Aug 24 14:36 /appl/hadoop/yarn [e1009156@vlmazgrpdata2 ~]$ ls -ld /appl/hadoop/yarn drwxrwxrwx 4 root hadoop 30 Aug 24 14:35 /appl/hadoop/yarn now: drwxrwxrwx 4 yarn hadoop 30 Aug 24 14:35 /appl/hadoop/yarn ... View more

all 3 nodes have following lines in 2 files, how do I check umask value set for the yarn user? i logged as sudo su - yarn, and noted umask is 022 /etc/bashrc /etc/profile # By default, we want umask to get set. This sets it for non-login shell. # Current threshold for system reserved uid/gids is 200 # You could check uidgid reservation validity in # /usr/share/doc/setup-*/uidgid file if [ $UID -gt 199 ] && [ "`/usr/bin/id -gn`" = "`/usr/bin/id -un`" ]; then umask 002 else umask 022 fi ... View more

umask on Linux is 0022 ... View more

Hi Geoffrey, Please find attatched the RM log from today. rm-log.txt ... View more

I also noticed the permission I added keeps changing to original value for below folder. So i get inconsistent response. [appadmin@mygrpdata1 yarn]$ ls -al /appl/hadoop/yarn/local/ total 0 drwxrwxrwx 5 yarn hadoop 57 Oct 5 16:13 . drwxrwxrwx 4 root root 30 Aug 24 14:36 .. drwxr-xr-x 3 yarn hadoop 16 Aug 30 13:46 filecache drwx------ 3 yarn hadoop 44 Oct 5 16:13 nmPrivate drwxr-xr-x 5 yarn hadoop 57 Oct 5 09:24 usercache ... View more

Thanks so much Jay! My problem is fixed. After fixing JAVA_HOME path, I stopped ambari-server and edited ambari-server.property file to reflect the new java_home path, and restarted. After that I was able to start datanode etc. Thanks again for helping me today!!! ... View more

Thank you, Jay! That's exactly is the problem. I can execute the java -version as root, but not as "hdfs". I don't know how to fix it. I issued 777 permission, still not worked. [root@vlmazgrpdata2 ~]# ls -l /home/winnie/apps/jdk1.8.0_144/bin/java -rwxr-xr-x 1 root root 7734 Jul 22 00:07 /home/winnie/apps/jdk1.8.0_144/bin/java [root@vlmazgrpdata2 ~]# chmod 777 /home/winnie/apps/jdk1.8.0_144/bin/java [root@vlmazgrpdata2 ~]# ls -l /home/winnie/apps/jdk1.8.0_144/bin/java -rwxrwxrwx 1 root root 7734 Jul 22 00:07 /home/winnie/apps/jdk1.8.0_144/bin/java [root@vlmazgrpdata2 ~]# su hdfs [hdfs@vlmazgrpdata2 root]$ /home/winnie/apps/jdk1.8.0_144//bin/java -version bash: /home/winnie/apps/jdk1.8.0_144//bin/java: Permission denied ... View more

How do I check the permission for "exec"? ... View more