Support Questions
Find answers, ask questions, and share your expertise

Re: ambari monitor failed to start

One other thing , The data for metrics is unavailable for only one master host. For the other host the graphs are visible

Re: ambari monitor failed to start

Super Mentor

@Krupal Jagtap

You can use the Ambari API approach to delete the AMS monitor component.

Example:

# curl  -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://hdfcluster1.example.com:8080/api/v1/clusters/TestCluster/hosts/hdfcluster3.example.com/host_c...

.


Here "hdfcluster3.example.com" is the host from where we want to delete the Metrics Monitor.
"hdfcluster1.example.com:8080" is ambari server host & port.
"TestCluster" is the cluster name.

Please replace these values accordingly.

Once it is deleted .. it is better to clean the "/etc/ambari-metrics-monitor" directory from that host. And it is also better to take a backup of the mentioned directory and then move it.

# mv /etc/ambari-metrics-monitor  /etc/ambari-metrics-monitor_BKP
# mv /usr/lib/python2.6/site-packages/resource_monitoring /usr/lib/python2.6/site-packages/resource_monitoring_BKP

.

Then freshly reinstall the monitor from ambari UI on this host.

Re: ambari monitor failed to start

HI

I tried the curl command for removing it

it returned the following error

{ "status" : 500, "message" : "org.apache.ambari.server.controller.spi.SystemException: An inter nal system exception occurred: Host Component cannot be removed, clusterName=Qc, serviceName=AMBARI_METRICS, componentName=METRICS_MONITOR, hostname= itxcqchdp01.catmdev.com, request={ clusterName=CardtronicsQc, serviceName=AMBARI _METRICS, componentName=METRICS_MONITOR, hostname=itxcqchdp01.catmdev.com, desir edState=null, state=null, desiredStackId=null, staleConfig=null, adminState=null }"

The problem seems to be on the main host itself

Re: ambari monitor failed to start

Super Mentor

@Krupal Jagtap

We will have to look at the complete "ambari-server.log" to understand what might be the issue.

However looks like it might be due to Database Inconsistency.

Take a fresh Amabri DB dump (as a backup).

Is it possible to just stop the "Ambari Metrics" service itself and then Delete the whole "Ambari Metrics Service" from ambari UI and then reinstall it to quickly attempt to find the fix. (If you do not have much AMS data then deleting this service and installing it might be a better option)


Re: ambari monitor failed to start

Yep of course. I do not have much ams data and I think to delete it completely and reinstall it will be the better option. Not sure how to do it though

Re: ambari monitor failed to start

Super Mentor

@Krupal Jagtap

Login to Ambari UI and the navigate to Amabri Dashboard left panel

Ambari Metrics --> "Service Actions" (Drop down menu) --> Stop
Ambari Metrics --> "Service Actions" (Drop down menu) --> Delete Service

.

Re: ambari monitor failed to start

Jay Kumar SenSharma there is no option of delete even after service is stopped, Just restart and move

Re: ambari monitor failed to start

Jay Kumar SenSharma ♦ Hi jay . So I removed the service and added it again through teh ambari web ui. Still one of the hosts does not generate any metrics. other host displays the graphs and everything. What might be the problem here?

Re: ambari monitor failed to start

the ambari-metric-monitor has following log

2018-04-05 00:07:04,749 [INFO] controller.py:110 - Adding event to cache, : {u'metrics': [], u'collect_every': u'15'} 2018-04-05 00:07:04,749 [INFO] main.py:65 - Starting Server RPC Thread: /usr/lib/python2.6/site-packages/resource_monitoring/main.py start 2018-04-05 00:07:04,749 [INFO] controller.py:57 - Running Controller thread: Thread-1 2018-04-05 00:07:04,750 [INFO] emitter.py:45 - Running Emitter thread: Thread-2 2018-04-05 00:07:04,750 [INFO] emitter.py:65 - Nothing to emit, resume waiting. 2018-04-05 00:08:04,752 [WARNING] emitter.py:74 - Error sending metrics to server. 'NoneType' object has no attribute 'strip' 2018-04-05 00:08:04,752 [WARNING] emitter.py:80 - Retrying after 5 ...

Re: ambari monitor failed to start

And if tried from hdfs user , It does not return any output