question Ambari Service Start not resolving $USER in Archives of Support Questions (Read Only)

Ambari Service Start not resolving $USER

vijaysl — Tue, 14 Feb 2017 02:06:45 GMT

Hi,

We are using HDP 2.5.0.0 on RHEL7. We had the cluster up and running for about 6 months now. We had to reboot the cluster couple of days back and noticed that the service status did not get correctly reflected on Ambari. If I do ps -ef on the box, I see that the process is running. Upon digging further I find that the $USER variable in the scripts is not getting resolved. As a result the pid files are created in the wrong directory with incorrect name. e.g /var/run/hadoop-mapreduce/mapred--historyserver.pid instead of /var/run/hadoop-mapreduce/mapred/mapred-mapred-history-server.pid. Any pointers in troubleshooting this? TIA

resource_management.core.exceptions.Fail: Execution of 'ambari-sudo.sh su mapred -l -s /bin/bash -c 'ls /var/run/hadoop-mapreduce/mapred/mapred-mapred-historyserver.pid && ps -p `cat /var/run/hadoop-mapreduce/mapred/mapred-mapred-historyserver.pid`'' returned 2. ls: cannot access /var/run/hadoop-mapreduce/mapred/mapred-mapred-historyserver.pid: No such file or directory

Re: Ambari Service Start not resolving $USER

yvora — Tue, 14 Feb 2017 08:14:47 GMT

@Vijay Lakshman, how did you restart the mapred history server after machine reboot ? Did you start it through ambari ?

Can you also check the value of HADOOP_MAPRED_PID_DIR var in hadoop-env.sh ? ideally it should be as below.

export HADOOP_MAPRED_PID_DIR=/var/run/hadoop-mapreduce/$USER

In order to fix this issue, stop the mapred-history server and make sure /var/run/hadoop-mapreduce/mapred--historyserver.pid and /var/run/hadoop-mapreduce/mapred/mapred-mapred-history-server.pid is deleted. Then start mapred history server through ambari.

Re: Ambari Service Start not resolving $USER

vijaysl — Tue, 14 Feb 2017 08:25:22 GMT

Thanks. I am starting all the services by clicking on Start All in Ambari. I am facing the same issue with all the services.

Yes, the environment variable is using the default and not overriden.

e.g for timeline server

drwxr-xr-x 2 yarn hadoop 40 Feb 13 18:45 yarn

-rw-r--r-- 1 yarn hadoop 6 Feb 13 18:48 yarn--resourcemanager.pid

-rw-r--r-- 1 yarn hadoop 5 Feb 13 18:47 yarn--timelineserver.pid

$ ls yarn

Re: Ambari Service Start not resolving $USER

yvora — Wed, 15 Feb 2017 08:41:26 GMT

@Vijay Lakshman, looks like the USER env variable got missing from the machines. Can you please check on your hosts whether $USER is set for all the users ( such as hdfs, yarn, mapred etc ). You can also use "printenv" to print all env var.

[root@xxx]# sudo su hdfs

 bash-4.2$ echo $USER

 hdfs

Re: Ambari Service Start not resolving $USER

vijaysl — Thu, 16 Feb 2017 00:34:37 GMT

Thanks that's helpful. We upgraded to RHEL7.3. Looks like this is not supported by HDP 2.5.3 yet.

Re: Ambari Service Start not resolving $USER

yvora — Thu, 16 Feb 2017 05:00:00 GMT

Glad to know I was able to help.

Re: Ambari Service Start not resolving $USER

vijaysl — Fri, 17 Feb 2017 02:31:39 GMT

It helped us troubleshoot the issue but we could not figure out why this is happening. We went back to reinstalling a new cluster on RHEL 7.2 and that is working fine. We still need to figure why RHEL 7.3 is causing this issue. Will post an update if we ever figure out.