Error in Ambari Metrics

[#1]

When I am trying to restart the Ambari -Metrics in my cluster, It is giving me the attached error. Please suggest a workaround here.

ambari-metrics_log.txt

[#2]

Can anyone please help on this?

[#3]

@Pritam Konar

As we see the following error:

shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/metrics_monitor.py", line 78, in <module>
    AmsMonitor().execute()

So can you please try this and then see if it works for you?

Edit the AGENT_WORKING_DIR property value inside the "/var/lib/ambari-agent/bin/ambari-agent" to something else and then try again. The default location will be something like following on all hosts.

Default setting

# grep 'AGENT_WORKING_DIR' /var/lib/ambari-agent/bin/ambari-agent
AGENT_WORKING_DIR=/var/lib/ambari-agent

For example try changing it to something like "/tmp/ambari-agent " as following:

Example After change

# grep 'AGENT_WORKING_DIR' /var/lib/ambari-agent/bin/ambari-agent
AGENT_WORKING_DIR=/tmp/ambari-agent 

Then restart Amabri Agent on all hosts.

# ambari-agent restart

[#4]

@Pritam Konar

Is this issue resolved? Are you still facing this issue?

[#5]

I tried with the above mentioned steps and tried to restart the ambari-metrics. But it is again giving the attached error:-

ambari-metrics_log.txt

[#6]

@Pritam Konar

We see the new Error in the logs now.

resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/sbin/ambari-metrics-monitor --config /etc/ambari-metrics-monitor/conf start' returned 255. psutil build directory is not empty, continuing...

.

So please try to rebuild the "psutil" on your own to see if it works for you on the Metrics Monitor Hosts. Then try again.

# cd /usr/lib/python2.6/site-packages/resource_monitoring 
# python psutil/build.py

Reference:

https://community.hortonworks.com/questions/235118/centos7-hdp31-metrics-monitor-start-importerror-c...

.

[#7]

Please find the output from the above command. The python script is throwing some syntactical error in the script:-

[#8]

@Pritam Konar

The Line 40 of "psutil/build.py" is just a Print Statement.

Please comment that line using # and then try again using.

.

Looks like you are using Python3 instead of python 2.7

# python --version

[#9]

@Jay Kumar SenSharma Based on your last comment, I commented out the line 40 in build.py. Still I am getting error for line 53 which was also a 'print' statement. When i commented out that line also, i am agian getting error for line 54. Please find the screenshot below:-

[#10]

@Jay Kumar SenSharma

Please find the latest log after commenting out the lines in build.py script. In ambari-metrics log, we can see the below error:-

raise Fail("Pid file {0} doesn't exist after starting of the component.".format(pid_file)) resource_management.core.exceptions.Fail: Pid file /var/run/ambari-metrics-collector//hbase-ams-master.pid doesn't exist after starting of the component.

I am attaching the full log of ambari-metrics:-

Ambari_Metrics_log.txt