@venkatsambath    I think and the same, we have analized its parents directories and apparently they are fine.   # ls -ln /var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir/libleveldbjni-64-1-6110205147654050510.8 ls: cannot access /var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir/libleveldbjni-64-1-6110205147654050510.8: No such file or directory  # id yarn uid=1075(yarn) gid=1051(hadoop) groups=1051(hadoop)   I have managed to start the node from the command line, but I have detected that the command that is executed from Ambari, sets the permissions of the path "hadoop_java_io_tmpdir" as owner to the user "hdfs:hadoop", however I do not identify why the user yarn does not have permissions writing and execution, the permissions are 1777 and the yarn user is member of hadoop group.   Regards     ... View more

@venkatsambath,   Here you have the strace outputs:   https://drive.google.com/open?id=1aktK1QqOqY0ub5-qd35XSy5bUUj2hyIt   https://drive.google.com/open?id=1cebTRxbcjdkXFtdQ2adEDMBaJlZg3SPP   Regards. ... View more

hi @venkatsambath, I attach the strace output.   problematic_node good_node   I found this lines: Problematic Node 49213 stat("/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir/libleveldbjni-64-1-6110205147654050510.8", 0x7f19c4ef1800) = -1 ENOENT (No such file or directory) 49213 open("/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir/libleveldbjni-64-1-6110205147654050510.8", O_RDWR|O_CREAT|O_EXCL, 0666) = -1 EACCES (Permission denied) Good Node: 19290 stat("/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir/libleveldbjni-64-1-280379409949290123.8", 0x7fbf276ef800) = -1 ENOENT (No such file or directory) 19290 open("/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir/libleveldbjni-64-1-280379409949290123.8", O_RDWR|O_CREAT|O_EXCL, 0666) = 354   The permissions to "hadoop_java_io_tmpdir" in the problematic nodo : # stat hadoop_java_io_tmpdir/ File: ‘hadoop_java_io_tmpdir/’ Size: 8192 Blocks: 24 IO Block: 4096 directory Device: fd02h/64770d Inode: 29362223 Links: 39 Access: (1777/drwxrwxrwt) Uid: ( 1073/ hdfs) Gid: ( 1051/ hadoop) Access: 2020-03-09 12:29:40.811107572 -0500 Modify: 2020-03-09 12:29:38.659084671 -0500 Change: 2020-03-09 12:29:38.659084671 -0500   I'll be waiting for your comments about.        Regards. ... View more

Hi @venkatsambath.   Personally I consier the same, mount options is not the problem, becausse with root user, the NM service start.   About selinux is disabled in both; old nodes and new nodes. Old Node # getenforce Disabled New Node # getenforce Disabled I can't find out which directories the permission is denied. I have recreated the directory structure however the error is the same.   I am trying to start the NM service from Ambari and also from the command line with the user YANR. I have executed with the --debug option and the error is:   # export HADOOP_LIBEXEC_DIR=/usr/hdp/3.0.1.0-187/hadoop/libexec && /usr/hdp/3.0.1.0-187/hadoop-yarn/bin/yarn --debug --config /usr/hdp/3.0.1.0-187/hadoop/conf --daemon start nodemanager DEBUG: hadoop_parse_args: processing --config DEBUG: hadoop_parse_args: processing --daemon DEBUG: hadoop_parse_args: processing nodemanager DEBUG: hadoop_parse: asking caller to skip 5 DEBUG: HADOOP_CONF_DIR=/usr/hdp/3.0.1.0-187/hadoop/conf DEBUG: shellprofiles: /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-aliyun.sh /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-archive-logs.sh /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-archives.sh /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-aws.sh /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-azure-datalake.sh /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-azure.sh /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-distcp.sh /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-extras.sh /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-gridmix.sh /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-hdfs.sh /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-httpfs.sh /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-kafka.sh /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-kms.sh /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-mapreduce.sh /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-openstack.sh /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-rumen.sh /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-s3guard.sh /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-streaming.sh /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-yarn.sh DEBUG: Profiles: importing /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-aliyun.sh DEBUG: Profiles: importing /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-archive-logs.sh DEBUG: Profiles: importing /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-archives.sh DEBUG: Profiles: importing /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-aws.sh DEBUG: Profiles: importing /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-azure-datalake.sh DEBUG: Profiles: importing /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-azure.sh DEBUG: Profiles: importing /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-distcp.sh DEBUG: Profiles: importing /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-extras.sh DEBUG: Profiles: importing /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-gridmix.sh DEBUG: Profiles: importing /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-hdfs.sh DEBUG: HADOOP_SHELL_PROFILES accepted hdfs DEBUG: Profiles: importing /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-httpfs.sh DEBUG: Profiles: importing /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-kafka.sh DEBUG: Profiles: importing /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-kms.sh DEBUG: Profiles: importing /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-mapreduce.sh DEBUG: HADOOP_SHELL_PROFILES accepted mapred DEBUG: Profiles: importing /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-openstack.sh DEBUG: Profiles: importing /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-rumen.sh DEBUG: Profiles: importing /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-s3guard.sh DEBUG: Profiles: importing /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-streaming.sh DEBUG: Profiles: importing /usr/hdp/3.0.1.0-187/hadoop/libexec/shellprofile.d/hadoop-yarn.sh DEBUG: HADOOP_SHELL_PROFILES accepted yarn DEBUG: Initialize CLASSPATH DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /usr/hdp/3.0.1.0-187/hadoop/build/native DEBUG: Append colonpath(JAVA_LIBRARY_PATH): /usr/hdp/3.0.1.0-187/hadoop/lib/native DEBUG: Initial CLASSPATH=/usr/hdp/3.0.1.0-187/hadoop/lib/* DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/hadoop/.//* DEBUG: Profiles: hdfs classpath DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/hadoop-hdfs/./ DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/hadoop-hdfs/lib/* DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/hadoop-hdfs/.//* DEBUG: Profiles: mapred classpath DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/hadoop-mapreduce/lib/* DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//* DEBUG: Profiles: yarn classpath DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/hadoop-yarn/./ DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/hadoop-yarn/lib/* DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/hadoop-yarn/.//* DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/hadoop-yarn/.//timelineservice/* DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/hadoop-yarn/.//timelineservice/lib/* DEBUG: Appending YARN_NODEMANAGER_OPTS onto HADOOP_OPTS DEBUG: No secure classname defined. DEBUG: Profiles: yarn finalize DEBUG: HADOOP_OPTS accepted -Dyarn.log.dir=/var/log/hadoop-yarn/yarn DEBUG: HADOOP_OPTS accepted -Dyarn.log.file=hadoop-root-nodemanager-pgdacl-hdpdat13.log DEBUG: HADOOP_OPTS accepted -Dyarn.home.dir=/usr/hdp/3.0.1.0-187/hadoop-yarn DEBUG: HADOOP_OPTS accepted -Dyarn.root.logger=INFO,console DEBUG: Prepend CLASSPATH: /usr/hdp/3.0.1.0-187/hadoop/conf DEBUG: Dupe CLASSPATH: /usr/hdp/3.0.1.0-187/tez/conf DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/conf_llap DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/doc DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/hadoop-shim-0.9.1.3.0.1.0-187.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/hadoop-shim-2.8-0.9.1.3.0.1.0-187.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/man DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/tez-api-0.9.1.3.0.1.0-187.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/tez-common-0.9.1.3.0.1.0-187.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/tez-dag-0.9.1.3.0.1.0-187.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/tez-examples-0.9.1.3.0.1.0-187.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/tez-history-parser-0.9.1.3.0.1.0-187.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/tez-javadoc-tools-0.9.1.3.0.1.0-187.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/tez-job-analyzer-0.9.1.3.0.1.0-187.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/tez-mapreduce-0.9.1.3.0.1.0-187.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/tez-protobuf-history-plugin-0.9.1.3.0.1.0-187.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/tez-runtime-internals-0.9.1.3.0.1.0-187.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/tez-runtime-library-0.9.1.3.0.1.0-187.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/tez-tests-0.9.1.3.0.1.0-187.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/tez-yarn-timeline-cache-plugin-0.9.1.3.0.1.0-187.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/tez-yarn-timeline-history-0.9.1.3.0.1.0-187.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/tez-yarn-timeline-history-with-acls-0.9.1.3.0.1.0-187.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/tez-yarn-timeline-history-with-fs-0.9.1.3.0.1.0-187.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/ui DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/async-http-client-1.9.40.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/commons-cli-1.2.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/commons-codec-1.4.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/commons-collections-3.2.2.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/commons-collections4-4.1.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/commons-io-2.4.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/commons-lang-2.6.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/commons-math3-3.1.1.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/gcs-connector-1.9.0.3.0.1.0-187-shaded.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/guava-11.0.2.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/hadoop-aws-3.1.1.3.0.1.0-187.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/hadoop-azure-3.1.1.3.0.1.0-187.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/hadoop-azure-datalake-3.1.1.3.0.1.0-187.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/hadoop-hdfs-client-3.1.1.3.0.1.0-187.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/hadoop-mapreduce-client-common-3.1.1.3.0.1.0-187.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/hadoop-mapreduce-client-core-3.1.1.3.0.1.0-187.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/hadoop-yarn-server-timeline-pluginstorage-3.1.1.3.0.1.0-187.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/jersey-client-1.19.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/jersey-json-1.19.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/jettison-1.3.4.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/jetty-server-9.3.22.v20171030.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/jetty-util-9.3.22.v20171030.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/jsr305-3.0.0.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/metrics-core-3.1.0.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/protobuf-java-2.5.0.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/RoaringBitmap-0.4.9.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/servlet-api-2.5.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/slf4j-api-1.7.10.jar DEBUG: Append CLASSPATH: /usr/hdp/3.0.1.0-187/tez/lib/tez.tar.gz DEBUG: Dupe CLASSPATH: /usr/hdp/3.0.1.0-187/tez/conf DEBUG: HADOOP_OPTS accepted -Djava.library.path=:/usr/hdp/3.0.1.0-187/hadoop/lib/native/Linux-amd64-64:/usr/hdp/3.0.1.0-187/hadoop/lib/native/Linux-amd64-64:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir:/usr/hdp/3.0.1.0-187/hadoop/lib/native DEBUG: HADOOP_OPTS accepted -Xmx4096m DEBUG: HADOOP_OPTS declined -Xmx5120m DEBUG: HADOOP_OPTS accepted -Dhadoop.log.dir=/var/log/hadoop-yarn/yarn DEBUG: HADOOP_OPTS accepted -Dhadoop.log.file=hadoop-root-nodemanager-pgdacl-hdpdat13.log DEBUG: HADOOP_OPTS accepted -Dhadoop.home.dir=/usr/hdp/3.0.1.0-187/hadoop DEBUG: HADOOP_OPTS accepted -Dhadoop.id.str=root DEBUG: HADOOP_OPTS accepted -Dhadoop.root.logger=INFO,RFA DEBUG: HADOOP_OPTS accepted -Dhadoop.policy.file=hadoop-policy.xml DEBUG: HADOOP_OPTS accepted -Dhadoop.security.logger=INFO,NullAppender ERROR: Cannot set priority of nodemanager process 9633   When cleaning the directories indicated, the libraries are copied automatically or it is necessary to copy them manually?   /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/* and also on /var/lib/ambari-agent/tmp/   What would be the recommended procedure to change the directory  JAVA_LIBRARY_PATH? considering that I have NM in production and processing applications.       ... View more

Hi @venkatsambath,   I share the outputs: Problematic Node: # namei -l /var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir f: /var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir dr-xr-xr-x root root / drwxr-xr-x root root var drwxr-xr-x root root lib drwxr-xr-x root root ambari-agent drwxrwxrwt root root tmp drwxrwxrwt hdfs hadoop hadoop_java_io_tmpdir   # mount sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,size=131760652k,nr_inodes=32940163,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/mapper/system-root on / type xfs (rw,relatime,attr2,inode64,sunit=512,swidth=512,noquota) systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=22,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=78969) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime) /dev/sdf1 on /grid/2 type ext3 (rw,relatime,stripe=64,data=ordered) /dev/sdg1 on /grid/3 type ext3 (rw,relatime,stripe=64,data=ordered) /dev/sdc1 on /grid/11 type ext3 (rw,relatime,stripe=64,data=ordered) /dev/sde1 on /grid/1 type ext3 (rw,relatime,stripe=64,data=ordered) /dev/sdm1 on /grid/9 type ext3 (rw,relatime,stripe=64,data=ordered) /dev/sda2 on /boot type xfs (rw,relatime,attr2,inode64,sunit=512,swidth=512,noquota) /dev/sdd1 on /grid/12 type ext3 (rw,relatime,stripe=64,data=ordered) /dev/sdb1 on /grid/10 type ext3 (rw,relatime,stripe=64,data=ordered) /dev/sdl1 on /grid/8 type ext3 (rw,relatime,stripe=64,data=ordered) /dev/sdk1 on /grid/7 type ext3 (rw,relatime,stripe=64,data=ordered) /dev/sdi1 on /grid/5 type ext3 (rw,relatime,stripe=64,data=ordered) /dev/sdj1 on /grid/6 type ext3 (rw,relatime,stripe=64,data=ordered) /dev/sdh1 on /grid/4 type ext3 (rw,relatime,stripe=64,data=ordered) /dev/sdn1 on /grid/0 type ext3 (rw,relatime,stripe=64,data=ordered) /dev/sda1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro) /dev/mapper/system-tmp on /tmp type xfs (rw,relatime,attr2,inode64,sunit=512,swidth=512,noquota) /dev/mapper/system-LVusuarios on /usuarios type xfs (rw,relatime,attr2,inode64,sunit=512,swidth=512,noquota) /dev/mapper/system-opt on /opt type xfs (rw,relatime,attr2,inode64,sunit=512,swidth=512,noquota) /dev/mapper/system-var on /var type xfs (rw,relatime,attr2,inode64,sunit=512,swidth=512,noquota) /dev/mapper/system-home on /home type xfs (rw,relatime,attr2,inode64,sunit=512,swidth=512,noquota) /dev/mapper/system-usr_hdp on /usr/hdp type xfs (rw,relatime,attr2,inode64,sunit=512,swidth=512,noquota) /dev/mapper/system-var_log on /var/log type xfs (rw,relatime,attr2,inode64,sunit=512,swidth=512,noquota) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime) tmpfs on /run/user/1051 type tmpfs (rw,nosuid,nodev,relatime,size=26354564k,mode=700,uid=1051,gid=1051) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=26354564k,mode=700) Working Node:  # namei -l /var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir f: /var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir dr-xr-xr-x root root / drwxr-xr-x root root var drwxr-xr-x root root lib drwxr-xr-x root root ambari-agent drwxrwxrwt root root tmp drwxrwxrwt hdfs hadoop hadoop_java_io_tmpdir # mount sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,size=96883504k,nr_inodes=24220876,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/mapper/system-root on / type xfs (rw,relatime,attr2,inode64,noquota) systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=32,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16126) debugfs on /sys/kernel/debug type debugfs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) /dev/sdf1 on /grid/2 type ext3 (rw,relatime,data=ordered) /dev/sdj1 on /grid/6 type ext3 (rw,relatime,data=ordered) /dev/sdl1 on /grid/8 type ext3 (rw,relatime,data=ordered) /dev/sdd1 on /grid/0 type ext3 (rw,relatime,data=ordered) /dev/sdh1 on /grid/4 type ext3 (rw,relatime,data=ordered) /dev/sde1 on /grid/1 type ext3 (rw,relatime,data=ordered) /dev/sdg1 on /grid/3 type ext3 (rw,relatime,data=ordered) /dev/sdk1 on /grid/7 type ext3 (rw,relatime,data=ordered) /dev/sdi1 on /grid/5 type ext3 (rw,relatime,data=ordered) /dev/mapper/system-opt on /opt type xfs (rw,relatime,attr2,inode64,noquota) /dev/mapper/system-home on /home type xfs (rw,relatime,attr2,inode64,noquota) /dev/mapper/system-var on /var type xfs (rw,relatime,attr2,inode64,noquota) /dev/mapper/system-tmp on /tmp type xfs (rw,relatime,attr2,inode64,noquota) /dev/mapper/system-var--log on /var/log type xfs (rw,relatime,attr2,inode64,noquota) /dev/sda1 on /boot type xfs (rw,relatime,attr2,inode64,noquota) /dev/mapper/hadoop-hdp on /usr/hdp type xfs (rw,relatime,attr2,inode64,noquota) binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=19379104k,mode=700) tmpfs on /run/user/1062 type tmpfs (rw,nosuid,nodev,relatime,size=19379104k,mode=700,uid=1062,gid=1062)  Any sugestion? ... View more

Hi,   The directory configured for "-Djava.io.tmpdir" has exec options.   With root user NodeManager runs but with yarn user from Ambari not runs.   Regards.   ... View more

@jsensharma    Thanks for your answers. I have checked the appropriate permissions and apparently everything is fine. I share the command outputs.  1). The output is the same with the old nodes the problem is with four new nodes installed with root user.   # ps -ef | grep NodeManager /usr/jdk64/jdk1.8.0_112/bin/java -Dproc_nodemanager -Dhdp.version=3.0.1.0-187 -Djava.net.preferIPv4Stack=true -Dhdp.version=3.0.1.0-187 -Dyarn.id.str= -Dyarn.policy.file=hadoop-policy.xml -Djava.io.tmpdir=/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir -Djavax.security.auth.useSubjectCredsOnly=false -Djava.security.auth.login.config=/etc/hadoop/3.0.1.0-187/0/yarn_nm_jaas.conf -Dsun.security.krb5.rcache=none -Dnm.audit.logger=INFO,NMAUDIT -Dyarn.log.dir=/var/log/hadoop-yarn/yarn -Dyarn.log.file=hadoop-yarn-nodemanager-pgdacl-hdpdat13.log -Dyarn.home.dir=/usr/hdp/3.0.1.0-187/hadoop-yarn -Dyarn.root.logger=INFO,console -Djava.library.path=:/usr/hdp/3.0.1.0-187/hadoop/lib/native/Linux-amd64-64:/usr/hdp/3.0.1.0-187/hadoop/lib/native/Linux-amd64-64:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir:/usr/hdp/3.0.1.0-187/hadoop/lib/native -Xmx4096m -Dhadoop.log.dir=/var/log/hadoop-yarn/yarn -Dhadoop.log.file=hadoop-yarn-nodemanager-NewHost.log -Dhadoop.home.dir=/usr/hdp/3.0.1.0-187/hadoop -Dhadoop.id.str=yarn -Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.yarn.server.nodemanager.NodeManager     2) Permissions for temp directories  # grep 'JAVA_LIBRARY_PATH' /etc/hadoop/conf/yarn-env.sh export JAVA_LIBRARY_PATH="${JAVA_LIBRARY_PATH}:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir" # ls -ld /var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir drwxrwxrwt+ 36 hdfs hadoop 8192 Mar 4 09:55 /var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir # ls -ld /var/lib/ambari-agent/tmp/ drwxrwxrwt+ 6 root root 4096 Mar 3 10:49 /var/lib/ambari-agent/tmp/  3) Owner for content in "hadoop_java_io_tmpdir" ls -lart /var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir/libleveldbjni* -rwxrwxr-x+ 1 yarn hadoop 752803 Mar 3 12:33 /var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir/libleveldbjni-64-1-1195312416327962249.8 I have compared the directories permissions with an old node and apparently they are similar. Suddenly any additional suggestions? ... View more

Hi,    I have reviewed the execution permissions of the temporary directories. /tmp /var/tmp /var/lib/ambari-agent/tmp/   The java.library.path files were also copied.   The NodeManager service runs with the root user but with the user yarn I have the following message:   java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no leveldbjni64-1.8 in java.library.path, no leveldbjni-1.8 in java.library.path, no leveldbjni in java.library.path, Permission denied]   Maybe, another sugestion?   ... View more