Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Error in Ambari Metrics

Highlighted

Error in Ambari Metrics

When I am trying to restart the Ambari -Metrics in my cluster, It is giving me the attached error. Please suggest a workaround here.


ambari-metrics_log.txt

9 REPLIES 9

Re: Error in Ambari Metrics

Can anyone please help on this?

Re: Error in Ambari Metrics

Super Mentor

@Pritam Konar

As we see the following error:

shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/metrics_monitor.py", line 78, in <module>
    AmsMonitor().execute()


So can you please try this and then see if it works for you?

Edit the AGENT_WORKING_DIR property value inside the "/var/lib/ambari-agent/bin/ambari-agent" to something else and then try again. The default location will be something like following on all hosts.

Default setting

# grep 'AGENT_WORKING_DIR' /var/lib/ambari-agent/bin/ambari-agent
AGENT_WORKING_DIR=/var/lib/ambari-agent


For example try changing it to something like "/tmp/ambari-agent " as following:

Example After change

# grep 'AGENT_WORKING_DIR' /var/lib/ambari-agent/bin/ambari-agent
AGENT_WORKING_DIR=/tmp/ambari-agent 


Then restart Amabri Agent on all hosts.

# ambari-agent restart

Re: Error in Ambari Metrics

Super Mentor

@Pritam Konar

Is this issue resolved? Are you still facing this issue?

Re: Error in Ambari Metrics

I tried with the above mentioned steps and tried to restart the ambari-metrics. But it is again giving the attached error:-


ambari-metrics_log.txt

Re: Error in Ambari Metrics

Super Mentor

@Pritam Konar

We see the new Error in the logs now.

resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/sbin/ambari-metrics-monitor --config /etc/ambari-metrics-monitor/conf start' returned 255. psutil build directory is not empty, continuing...

.


So please try to rebuild the "psutil" on your own to see if it works for you on the Metrics Monitor Hosts. Then try again.

# cd /usr/lib/python2.6/site-packages/resource_monitoring 
# python psutil/build.py

Reference:

https://community.hortonworks.com/questions/235118/centos7-hdp31-metrics-monitor-start-importerror-c...


.

Re: Error in Ambari Metrics

Please find the output from the above command. The python script is throwing some syntactical error in the script:-


109723-ambari-metrics.png



Re: Error in Ambari Metrics

Super Mentor

@Pritam Konar

The Line 40 of "psutil/build.py" is just a Print Statement.

Please comment that line using # and then try again using.

.

Looks like you are using Python3 instead of python 2.7

# python --version

Re: Error in Ambari Metrics

@Jay Kumar SenSharma Based on your last comment, I commented out the line 40 in build.py. Still I am getting error for line 53 which was also a 'print' statement. When i commented out that line also, i am agian getting error for line 54. Please find the screenshot below:-


109718-ambari-screenshot1.png


109719-ambari-error23.png


109733-python-version.png

Re: Error in Ambari Metrics

@Jay Kumar SenSharma


Please find the latest log after commenting out the lines in build.py script. In ambari-metrics log, we can see the below error:-


raise Fail("Pid file {0} doesn't exist after starting of the component.".format(pid_file)) resource_management.core.exceptions.Fail: Pid file /var/run/ambari-metrics-collector//hbase-ams-master.pid doesn't exist after starting of the component.


I am attaching the full log of ambari-metrics:-


Ambari_Metrics_log.txt