Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Error in Ambari Metrics

avatar

When I am trying to restart the Ambari -Metrics in my cluster, It is giving me the attached error. Please suggest a workaround here.


ambari-metrics_log.txt

9 REPLIES 9

avatar

Can anyone please help on this?

avatar
Master Mentor

@Pritam Konar

As we see the following error:

shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/metrics_monitor.py", line 78, in <module>
    AmsMonitor().execute()


So can you please try this and then see if it works for you?

Edit the AGENT_WORKING_DIR property value inside the "/var/lib/ambari-agent/bin/ambari-agent" to something else and then try again. The default location will be something like following on all hosts.

Default setting

# grep 'AGENT_WORKING_DIR' /var/lib/ambari-agent/bin/ambari-agent
AGENT_WORKING_DIR=/var/lib/ambari-agent


For example try changing it to something like "/tmp/ambari-agent " as following:

Example After change

# grep 'AGENT_WORKING_DIR' /var/lib/ambari-agent/bin/ambari-agent
AGENT_WORKING_DIR=/tmp/ambari-agent 


Then restart Amabri Agent on all hosts.

# ambari-agent restart

avatar
Master Mentor

@Pritam Konar

Is this issue resolved? Are you still facing this issue?

avatar

I tried with the above mentioned steps and tried to restart the ambari-metrics. But it is again giving the attached error:-


ambari-metrics_log.txt

avatar
Master Mentor

@Pritam Konar

We see the new Error in the logs now.

resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/sbin/ambari-metrics-monitor --config /etc/ambari-metrics-monitor/conf start' returned 255. psutil build directory is not empty, continuing...

.


So please try to rebuild the "psutil" on your own to see if it works for you on the Metrics Monitor Hosts. Then try again.

# cd /usr/lib/python2.6/site-packages/resource_monitoring 
# python psutil/build.py

Reference:

https://community.hortonworks.com/questions/235118/centos7-hdp31-metrics-monitor-start-importerror-c...


.

avatar

Please find the output from the above command. The python script is throwing some syntactical error in the script:-


109723-ambari-metrics.png



avatar
Master Mentor

@Pritam Konar

The Line 40 of "psutil/build.py" is just a Print Statement.

Please comment that line using # and then try again using.

.

Looks like you are using Python3 instead of python 2.7

# python --version

avatar

@Jay Kumar SenSharma Based on your last comment, I commented out the line 40 in build.py. Still I am getting error for line 53 which was also a 'print' statement. When i commented out that line also, i am agian getting error for line 54. Please find the screenshot below:-


109718-ambari-screenshot1.png


109719-ambari-error23.png


109733-python-version.png

avatar

@Jay Kumar SenSharma


Please find the latest log after commenting out the lines in build.py script. In ambari-metrics log, we can see the below error:-


raise Fail("Pid file {0} doesn't exist after starting of the component.".format(pid_file)) resource_management.core.exceptions.Fail: Pid file /var/run/ambari-metrics-collector//hbase-ams-master.pid doesn't exist after starting of the component.


I am attaching the full log of ambari-metrics:-


Ambari_Metrics_log.txt