Support Questions
Find answers, ask questions, and share your expertise

Can't collection metric on Ambari 2.5 and HDP 2.6 and Isilon 8.0.5

Hi thanks for the support, I wanted to report a problem that I can not solve on Ambari 2.5 cluster, I installed Hadoop 2.6.

Installation is fine, but I can not collect metric to the ISILON cluster with version 8.0.5

How can I solve this problem ? I attach you some photos.

metric1.jpg

metric2.jpg

11 REPLIES 11

Re: Can't collection metric on Ambari 2.5 and HDP 2.6 and Isilon 8.0.5

10 Aug 2017 12:38:50,926 ERROR [ambari-metrics-retrieval-service-thread-12822] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem...
10 Aug 2017 12:38:50,968 INFO [qtp-ambari-agent-32682] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361530968}
10 Aug 2017 12:38:56,111 INFO [qtp-ambari-agent-32698] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361536110}
10 Aug 2017 12:38:57,503 ERROR [ambari-metrics-retrieval-service-thread-12818] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem...
10 Aug 2017 12:38:57,503 ERROR [ambari-metrics-retrieval-service-thread-12814] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx
10 Aug 2017 12:39:01,235 INFO [qtp-ambari-agent-32698] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361541234}
10 Aug 2017 12:39:04,127 ERROR [ambari-metrics-retrieval-service-thread-12816] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem...
10 Aug 2017 12:39:04,127 ERROR [ambari-metrics-retrieval-service-thread-12823] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx
10 Aug 2017 12:39:06,365 INFO [qtp-ambari-agent-32682] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361546364}
10 Aug 2017 12:39:10,799 ERROR [ambari-metrics-retrieval-service-thread-12824] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem...
10 Aug 2017 12:39:10,799 ERROR [ambari-metrics-retrieval-service-thread-12842] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx
10 Aug 2017 12:39:11,491 INFO [qtp-ambari-agent-32698] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361551490}
10 Aug 2017 12:39:16,672 INFO [qtp-ambari-agent-32682] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361556671}
10 Aug 2017 12:39:17,547 ERROR [ambari-metrics-retrieval-service-thread-12835] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem...
10 Aug 2017 12:39:17,547 ERROR [ambari-metrics-retrieval-service-thread-12844] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx
10 Aug 2017 12:39:21,878 INFO [qtp-ambari-agent-32682] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361561877}
10 Aug 2017 12:39:24,205 ERROR [ambari-metrics-retrieval-service-thread-12846] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx
10 Aug 2017 12:39:24,205 ERROR [ambari-metrics-retrieval-service-thread-12843] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem...
10 Aug 2017 12:39:27,025 INFO [qtp-ambari-agent-32718] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361567024}
10 Aug 2017 12:39:30,865 ERROR [ambari-metrics-retrieval-service-thread-12840] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx
10 Aug 2017 12:39:30,865 ERROR [ambari-metrics-retrieval-service-thread-12818] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem...
10 Aug 2017 12:39:32,166 INFO [qtp-ambari-agent-32682] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361572165}
10 Aug 2017 12:39:37,306 INFO [qtp-ambari-agent-32714] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361577305}
10 Aug 2017 12:39:37,572 ERROR [ambari-metrics-retrieval-service-thread-12845] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx
10 Aug 2017 12:39:37,572 ERROR [ambari-metrics-retrieval-service-thread-12835] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem...
10 Aug 2017 12:39:42,498 INFO [qtp-ambari-agent-32698] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361582497}
10 Aug 2017 12:39:44,457 ERROR [ambari-metrics-retrieval-service-thread-12824] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx
10 Aug 2017 12:39:44,457 ERROR [ambari-metrics-retrieval-service-thread-12836] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem...
10 Aug 2017 12:39:47,623 INFO [qtp-ambari-agent-32698] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361587622}
10 Aug 2017 12:39:51,146 ERROR [ambari-metrics-retrieval-service-thread-12844] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx
10 Aug 2017 12:39:51,147 ERROR [ambari-metrics-retrieval-service-thread-12841] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem...
10 Aug 2017 12:39:52,750 INFO [qtp-ambari-agent-32698] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361592749}
10 Aug 2017 12:39:57,820 ERROR [ambari-metrics-retrieval-service-thread-12817] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem...
10 Aug 2017 12:39:57,821 ERROR [ambari-metrics-retrieval-service-thread-12822] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx
10 Aug 2017 12:39:57,943 INFO [qtp-ambari-agent-32714] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361597942}
10 Aug 2017 12:40:03,148 INFO [qtp-ambari-agent-32714] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361603146}
10 Aug 2017 12:40:04,431 ERROR [ambari-metrics-retrieval-service-thread-12816] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx
10 Aug 2017 12:40:04,432 ERROR [ambari-metrics-retrieval-service-thread-12842] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem...
10 Aug 2017 12:40:08,308 INFO [qtp-ambari-agent-32698] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361608307}
10 Aug 2017 12:40:11,033 ERROR [ambari-metrics-retrieval-service-thread-12823] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem...
10 Aug 2017 12:40:11,033 ERROR [ambari-metrics-retrieval-service-thread-12814] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx
10 Aug 2017 12:40:13,525 INFO [qtp-ambari-agent-32718] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361613524}
10 Aug 2017 12:40:17,628 ERROR [ambari-metrics-retrieval-service-thread-12846] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx
10 Aug 2017 12:40:17,628 ERROR [ambari-metrics-retrieval-service-thread-12824] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem...
10 Aug 2017 12:40:18,712 INFO [qtp-ambari-agent-32718] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361618711}
10 Aug 2017 12:40:23,867 INFO [qtp-ambari-agent-32718] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361623866}
10 Aug 2017 12:40:24,303 ERROR [ambari-metrics-retrieval-service-thread-12840] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx
10 Aug 2017 12:40:24,304 ERROR [ambari-metrics-retrieval-service-thread-12818] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem...
10 Aug 2017 12:40:28,995 INFO [qtp-ambari-agent-32682] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361628994}
10 Aug 2017 12:40:30,926 ERROR [ambari-metrics-retrieval-service-thread-12835] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem...
10 Aug 2017 12:40:30,926 ERROR [ambari-metrics-retrieval-service-thread-12844] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx
10 Aug 2017 12:40:34,136 INFO [qtp-ambari-agent-32718] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361634135}
10 Aug 2017 12:40:37,581 ERROR [ambari-metrics-retrieval-service-thread-12843] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem...
10 Aug 2017 12:40:37,581 ERROR [ambari-metrics-retrieval-service-thread-12816] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx
10 Aug 2017 12:40:39,259 INFO [qtp-ambari-agent-32682] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361639259}
10 Aug 2017 12:40:44,202 ERROR [ambari-metrics-retrieval-service-thread-12841] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx
10 Aug 2017 12:40:44,202 ERROR [ambari-metrics-retrieval-service-thread-12823] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem...
10 Aug 2017 12:40:44,406 INFO [qtp-ambari-agent-32682] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361644405}
10 Aug 2017 12:40:50,205 INFO [qtp-ambari-agent-32718] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361650204}
10 Aug 2017 12:40:50,840 ERROR [ambari-metrics-retrieval-service-thread-12846] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem...
10 Aug 2017 12:40:50,841 ERROR [ambari-metrics-retrieval-service-thread-12822] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx
10 Aug 2017 12:40:55,362 INFO [qtp-ambari-agent-32682] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361655361}
10 Aug 2017 12:40:57,582 ERROR [ambari-metrics-retrieval-service-thread-12836] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx
10 Aug 2017 12:40:57,582 ERROR [ambari-metrics-retrieval-service-thread-12814] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem...
10 Aug 2017 12:41:00,534 INFO [qtp-ambari-agent-32725] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361660533}
10 Aug 2017 12:41:04,287 ERROR [ambari-metrics-retrieval-service-thread-12845] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx
10 Aug 2017 12:41:04,287 ERROR [ambari-metrics-retrieval-service-thread-12840] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem...
10 Aug 2017 12:41:05,680 INFO [qtp-ambari-agent-32723] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361665679}
10 Aug 2017 12:41:10,815 INFO [qtp-ambari-agent-32725] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361670814}
10 Aug 2017 12:41:11,012 ERROR [ambari-metrics-retrieval-service-thread-12817] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx
10 Aug 2017 12:41:11,013 ERROR [ambari-metrics-retrieval-service-thread-12816] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem...
10 Aug 2017 12:41:15,947 INFO [qtp-ambari-agent-32698] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361675946}
10 Aug 2017 12:41:17,641 ERROR [ambari-metrics-retrieval-service-thread-12841] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx
10 Aug 2017 12:41:17,641 ERROR [ambari-metrics-retrieval-service-thread-12842] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem...
10 Aug 2017 12:41:21,158 INFO [qtp-ambari-agent-32723] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361681156}
10 Aug 2017 12:41:24,388 ERROR [ambari-metrics-retrieval-service-thread-12846] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem...
10 Aug 2017 12:41:24,388 ERROR [ambari-metrics-retrieval-service-thread-12835] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx
10 Aug 2017 12:41:26,303 INFO [qtp-ambari-agent-32714] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361686302}
10 Aug 2017 12:41:31,033 ERROR [ambari-metrics-retrieval-service-thread-12843] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem...
10 Aug 2017 12:41:31,034 ERROR [ambari-metrics-retrieval-service-thread-12824] URLStreamProvider:245 - Received HTTP 403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx
10 Aug 2017 12:41:31,482 INFO [qtp-ambari-agent-32724] HeartBeatHandler:289 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1502361691481}

Re: Can't collection metric on Ambari 2.5 and HDP 2.6 and Isilon 8.0.5

Super Mentor

@Giovanni Ruotolo

As most of those errors shows HTTP Response 403 while accessing 8082 port (You do not have Kerberos enabled according to screenshot). So it might be some Proxy issue. Can you check if you are able to do a simple curl call to the mentioned URL and get the response?

403 response from URL: http://isilon-hdfs-poc.discoverylab.bdt.local:8082/

.

Example: (From ambari server)

wget "http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem::tag.HAState"

OR

wget "http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem"

.

If it fails then it's some issue with the proxy or authentication. Can you please share the output of the above command?

- Also please check if that port is correct ?

Re: Can't collection metric on Ambari 2.5 and HDP 2.6 and Isilon 8.0.5

Super Mentor

@Giovanni Ruotolo

Above was for the failing JMX calls. But we see the the AMS Sink widgets are also not showing data, that may be because the AMS version might not be correct or the Components like NameNode/DataNode might not have started the sink services properly.

1. Please check the NameNode/DataNdoe log it you see any metrics collection related connection refused error? If yes then please check if from the NameNode/DataNode hosts if the AMS host port are correctly mentioned following file and the AMS host & port mentioned in this file are actually reachable?

# grep 'timeline.collector.host'  /etc/hadoop/conf/hadoop-metrics2.properties
datanode.sink.timeline.collector.hosts=amb25102.example.com,amb25104.example.com
.
.

.

2. Also please check if the correct version of Sink JAR is loaded on those Nodes/components?

# lsof -p `cat /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid`  | grep sink
java    1632 hdfs  mem       REG              253,1   3125254   8472765 /usr/lib/ambari-metrics-hadoop-sink/ambari-metrics-hadoop-sink-with-common-2.5.1.0.159.jar

.

3. Also please check if the AMS collector version is same and matches the Ambari version through out the cluster.

# rpm -qa | grep ambari

.

Above commands are CentOS/RHEL specific so please find the equivilant command for your OS.

Re: Can't collection metric on Ambari 2.5 and HDP 2.6 and Isilon 8.0.5

Hi Jay

I see that in Grafana i can't collect data..

26435-hny3c.png

Re: Can't collection metric on Ambari 2.5 and HDP 2.6 and Isilon 8.0.5

Super Mentor

@Giovanni Ruotolo

Grafana Pulls data from "Ambari Metrics Collector" So until unless the AMS is healthy and running file ... we wont be able to see any data in the grafana.

.

Re: Can't collection metric on Ambari 2.5 and HDP 2.6 and Isilon 8.0.5

Hi Jay

I saw that I can not open any links on the HDFS protocol

23523-o8gqd.png

And

23524-rf4o2.png

Wer i can see this issue ?

Thanks so much

Re: Can't collection metric on Ambari 2.5 and HDP 2.6 and Isilon 8.0.5

I've followed your instructions, even eliminating internal proxies, but I continue to make the same mistake.
Could you give me some additional indication?
Thanks so much

Re: Can't collection metric on Ambari 2.5 and HDP 2.6 and Isilon 8.0.5

Hi i'm try to wget

[root@ambari ~]# wget "http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem"
--2017-08-10 13:54:54-- http://isilon-hdfs-poc.discoverylab.bdt.local:8082/jmx?get=Hadoop:service=NameNode,name=FSNamesystem
Resolving isilon-hdfs-poc.discoverylab.bdt.local (isilon-hdfs-poc.discoverylab.bdt.local)... 10.56.245.203
Connecting to isilon-hdfs-poc.discoverylab.bdt.local (isilon-hdfs-poc.discoverylab.bdt.local)|10.56.245.203|:8082... connected.
HTTP request sent, awaiting response... 403 Forbidden
2017-08-10 13:54:54 ERROR 403: Forbidden.

Step 1 on Ambari server

[root@ambari ~]# grep 'timeline.collector.host' /etc/hadoop/conf/hadoop-metrics2.properties
grep: /etc/hadoop/conf/hadoop-metrics2.properties: No such file or directory
[root@ambari ~]# lsof -p `cat /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid` | grep sink
cat: /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid: No such file or directory
lsof: no process ID specified
lsof 4.87
latest revision: ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/
latest FAQ: ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/FAQ
latest man page: ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/lsof_man
usage: [-?abhKlnNoOPRtUvVX] [+|-c c] [+|-d s] [+D D] [+|-f[gG]] [+|-e s]
[-F [f]] [-g [s]] [-i [i]] [+|-L [l]] [+m [m]] [+|-M] [-o [o]] [-p s]
[+|-r [t]] [-s [p:s]] [-S [t]] [-T [t]] [-u s] [+|-w] [-x [fl]] [--] [names]
Use the ``-h'' option to get more help information.
[root@ambari ~]# rpm -qa | grep ambari
ambari-server-2.5.0.3-7.x86_64
[root@ambari ~]#

Step 2 on Head01

[root@head01 ~]# grep 'timeline.collector.host' /etc/hadoop/conf/hadoop-metrics2.properties
datanode.sink.timeline.collector.hosts=head01.discoverylab.bdt.local
namenode.sink.timeline.collector.hosts=head01.discoverylab.bdt.local
resourcemanager.sink.timeline.collector.hosts=head01.discoverylab.bdt.local
nodemanager.sink.timeline.collector.hosts=head01.discoverylab.bdt.local
jobhistoryserver.sink.timeline.collector.hosts=head01.discoverylab.bdt.local
journalnode.sink.timeline.collector.hosts=head01.discoverylab.bdt.local
maptask.sink.timeline.collector.hosts=head01.discoverylab.bdt.local
reducetask.sink.timeline.collector.hosts=head01.discoverylab.bdt.local
applicationhistoryserver.sink.timeline.collector.hosts=head01.discoverylab.bdt.local
[root@head01 ~]# lsof -p `cat /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid` | grep sink
cat: /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid: No such file or directory
lsof: no process ID specified
lsof 4.87
latest revision: ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/
latest FAQ: ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/FAQ
latest man page: ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/lsof_man
usage: [-?abhKlnNoOPRtUvVX] [+|-c c] [+|-d s] [+D D] [+|-f[gG]] [+|-e s]
[-F [f]] [-g [s]] [-i [i]] [+|-L [l]] [+m [m]] [+|-M] [-o [o]] [-p s]
[+|-r [t]] [-s [p:s]] [-S [t]] [-T [t]] [-u s] [+|-w] [-x [fl]] [--] [names]
Use the ``-h'' option to get more help information.
[root@head01 ~]# rpm -qa | grep ambari
ambari-metrics-hadoop-sink-2.5.0.3-7.x86_64
ambari-agent-2.5.0.3-7.x86_64
ambari-infra-solr-2.5.0.3-7.noarch
ambari-infra-solr-client-2.5.0.3-7.noarch
ambari-metrics-collector-2.5.0.3-7.x86_64
ambari-metrics-monitor-2.5.0.3-7.x86_64
[root@head01 ~]#

Re: Can't collection metric on Ambari 2.5 and HDP 2.6 and Isilon 8.0.5

For the Isilon metrics in Grafana its actually better to monitor them using the isilon statisics API...

I write this up in a blog.

https://community.emc.com/people/NickyRuggs/blog/2017/06/17/isilon-telemetry-for-the-hadoop-admin