Support Questions
Find answers, ask questions, and share your expertise

Ambari Metrics Error HDF 3.0

Expert Contributor

I just installed HDF 3.0. I see the following error in NiFi bulletin board:

12:49:49 UTC
ERROR
3b80ba0f-a6c0-48db-b721-4dbc04cef28e

sandbox-hdf.hortonworks.com:19090
AmbariReportingTask[id=3b80ba0f-a6c0-48db-b721-4dbc04cef28e] Error running task AmbariReportingTask[id=3b80ba0f-a6c0-48db-b721-4dbc04cef28e] due to javax.ws.rs.ProcessingException: java.net.ConnectException: Connection refused (Connection refused

I added the port forwarding rule in virtual box and then tried to navigate to the following URL:

http://localhost:6188/ws/v1/timeline/metrics

Where I get the site cant be reached message. Does any one have any idea how to resolve this?

Also what dependencies does Ambari Metrics have? which I need to make sure are up and running.

I have made no changed to the sandbox and everything is set to default settings.

I also notice in the dashboard I see no data available for most of the metrics. I started the HDFS service, I see only a few metrics displayed like "under replicated bloacks", "HDFS disk usage", name node rpc etc but not all of them.

@Pierre Villard I am trying to follow your tutorial on monitoring nifi. Unfortunately I can not move forward unless i resolve this.

9 REPLIES 9

Re: Ambari Metrics Error HDF 3.0

Super Mentor

@Arsalan Siddiqi

- AMS service is installed properly on the host?

- On the host where the AMS is running do we see that the AMS collector port 6188 is opened?

# netstat -tnlpa | grep 6188

.

If the port is not opened then there might be some AMS startup issues, which we can find in the following log files: (If you notice any error there then please share.

# ls -l /var/log/ambari-metrics-collector/ambari-metrics-collector.log

.

Ambari metrics collector packages comes from Ambari repository that you can find inside the "/etc/yum/repos.d/ambari,repo" file. In order to find its dependency you can try:

# repoquery --requires --resolve ambari-metrics-collector
python-0:2.7.5-48.el7.x86_64
snappy-0:1.1.0-3.el7.i686
snappy-0:1.1.0-3.el7.x86_64

.


Regarding "No Datapoint" issue :
There can be many reasons behind "no data available" on dashboard for different widgets. For example if the HDFS components have not started the SINK service. In order to find if that is the case then try the following grep on the DataNode or NameNode log:

# cd /var/log/hadoop/hdfs/
# find . -name "*.*" | xargs grep -i "Sink timeline started"

# cd /var/log/hadoop-yarn/yarn
# find . -name "*.*" | xargs grep -i "Sink timeline started"


OR

May be the "ambari-metrics-hadoop-sink-with-common" jar is not loaded by the components like NameNode/DataNode ...etc. That we can find out using the following command, following command should list the jar is loaded by these components ir not? If not then try restarting the components.

# lsof -p `cat /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid `  | grep metrics
# lsof -p `cat /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid`  | grep metrics


- Also please check the file like "/etc/hadoop/conf/hadoop-metrics2.properties" to see if it has correct AMS host port entry?

# cat /etc/hadoop/conf/hadoop-metrics2.properties

.

Re: Ambari Metrics Error HDF 3.0

Expert Contributor

@Jay SenSharma thanks for the reply. I dont see anything when i run the first command. Attached is the log file:hdflog.txt

it seems to be a zookeeper issue. Any idea?

Re: Ambari Metrics Error HDF 3.0

Super Mentor

@Arsalan Siddiqi

Please check the value of "timeline.metrics.service.webapp.address" is set to "0.0.0.0:6188" or not?

Also if the issue still persist then it is better to try Cleaning up the AMS Data (Specially "hbase.zookeeper.property.dataDir") as described in the following doc and then try restart the AMS. https://cwiki.apache.org/confluence/display/AMBARI/Cleaning+up+Ambari+Metrics+System+Data

- Additionally please try to increase the AMS heap size to at least 1024M

Re: Ambari Metrics Error HDF 3.0

Expert Contributor

@Jay SenSharma thanks for the reply. the value is set correctly. Is ambari dependent on any other thing? ambari infra? like do i need to start any other service aswell....

I followed the instructions on the link above and cleared all the data.. Now i no longer seem to get the error in the NiFi bulletin but i do not see any metrics on the Nifi page in ambari. it still displays no data available . I did run a flow to see if it gets updated.

Re: Ambari Metrics Error HDF 3.0

Super Mentor

@Arsalan Siddiqi

Regarding "Is ambari dependent on any other thing?"

>>>> By default ambari uses embedded HBase which is started as soon as we start the AMS. Only if we use AMS in distributed mode then it has the dependency on the external HBase. Also Embedded AMS by default writes on the local file system. It can be seen in the property "hbase.rootdir" defined in ams-site (Default value: "file:///hadoop/var/lib/ambari-metrics-collector/hbase") . But if we use HDFS for the storege then it depends on HDFS as well.

- Have you restarted NiFi service once the AMS started working without the zookeeper error.

Re: Ambari Metrics Error HDF 3.0

Expert Contributor

@Jay SenSharma i did restart nifi so far displays nothing. attached is the new log file. ambari-metrics-collectorlog.tar.gz

Also i still cant reach http://localhost:6188/ws/v1/timeline/metrics from my browser. Although now When use the following command : # netstat -tnlpa | grep 6188

I do get a bunch of items.

After cleaning the data I see the metrics do work and I do see some graphs in the widgets like :

CPU usage, Memory Usage, Disk IO on the main page for ambari but unfortunately I still dont see any data for NiFi. it still says no data available.

I have the default settings for the reporting task in Nifi set to

${ambari.metrics.collector.url}

${ambari.application.id}

${hostname(true)} I also checked

lsof -p `cat /var/run/nifi/nifi.pid ` | grep metrics

and i do have a bunch of jar files.

Any idea why cant i see the NiFi metrics and why cant i hit the URL.

Re: Ambari Metrics Error HDF 3.0

Expert Contributor

@Bryan Bende any idea?

Re: Ambari Metrics Error HDF 3.0

Do you see any metrics if you go to Grafana rather than the widgets in Ambari UI?

Re: Ambari Metrics Error HDF 3.0

I am wondering if this was ever resolved? I am having the exact issue as @Arsalan Siddiqi

I am trying to setup AmbariReportingTask and get Connection refused (Connection refused) for metric collector url:

http://localhost:6188/ws/v1/timeline/metrics

Thank you.