javax.ws.rs.WebApplicationException running ambari-metrics-collector

[#1]

I've installed HDP 2.6 and I've a problem when running ambari-metrics-collector. I start it from ambari, and it starts correctly with no alert, but no metrics are collected. The only log with some error is `ambari-metrics-collector.log` and it displays the following exception:

2017-03-22 10:26:41,226 WARN org.apache.hadoop.yarn.webapp.GenericExceptionHandler: INTERNAL_SERVER_ERROR
  javax.ws.rs.WebApplicationException

     at 
org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TimelineWebServices.getEntity(TimelineWebServices.java:209)
     at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at java.lang.reflect.Method.invoke(Method.java:606)
     at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)

     at 
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)

     at 
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
     at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
     at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
     at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
     at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
     at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
     at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
     at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
     at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
     at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
     at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
     at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
     at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:895)
     at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:843)
     at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:804)
     at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
     at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
     at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
     at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
     at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
     at org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57)
     at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
     at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
     at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
     at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1294)
     at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
     at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
     at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
     at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
     at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
     at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
     at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
     at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
     at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
     at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
     at org.mortbay.jetty.Server.handle(Server.java:326)
     at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
     at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
     at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
     at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

And there is the full log: http://pastebin.com/NEcJZrR8

Thanks in advance for any help.

[#2]

@Adrián Gil

The Pastebin url is asking for existing account hence did not check it.Will check it shortly.

However as you mentioned that you are using HDP2.6 so have you upgraded the Ambari Metrics Collector packages as well?

Can you please share the output of the following command? If we have the correct same version installed for AMS on the cluster as ambari version.

$ rpm -qa | grep ambari-*

.

[#3]

The comand execution is giving no output:

pcitbu@master ~ $ rpm -qa | grep ambari-*
pcitbu@master ~ $ 

I deployed HDP 2.6 directly as it was the latest version available in Ambari, so I did not perform any upgrade at all

[#4]

@Adrián Gil

From where did you get the HDP 2.6 repo?

With HDP 2.6 i guess you should use Ambari 2.5, So please try upgrading ambari to at least 2.5.0 because with ambari repo you will get ambari-metrics packages as well.

[#5]

@Jay SenSharma

I compiled ambari from source creating .deb packages, then installed and configured server and agents and deployed HDP from the repositories.

After that, I downloaded some packages which ambari was not installing by default and manually installed them with the following code

sudo wget http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.4.2.0/ambari-2.4.2.0-ubuntu14.tar... -O ambari.tar.gz
sudo tar -xvzf ambari.tar.gz
sudo dpkg --install AMBARI-2.4.2.0/ubuntu14/2.4.2.0-136/pool/main/a/ambari-infra-solr/ambari-infra-solr_2.4.2.0-136_amd64.deb
sudo dpkg --install AMBARI-2.4.2.0/ubuntu14/2.4.2.0-136/pool/main/a/ambari-infra-solr-client/ambari-infra-solr-client_2.4.2.0-136_amd64.deb
sudo dpkg --install AMBARI-2.4.2.0/ubuntu14/2.4.2.0-136/pool/main/a/ambari-metrics-assembly/ambari-metrics-assembly_2.4.2.0-136_amd64.deb
 

Once done, I made some configuration changes so that everything worked fine (and installing hue to work it with) and I get no errors in any service and everything is working fine, even on logs, except for this problem with metrics.

[#6]

@Adrián Gil

As you mentioned that you are building ambari , hence you will need to build the "ambari-metrics" prject as well.

Please see: https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development

# cd ambari
# mvn versions:set -DnewVersion=${AMBARI_VERSION}
# pushd ambari-metrics
# mvn versions:set -DnewVersion=${AMBARI_VERSION}
# popd  
# mvn -B clean install package rpm:rpm -DskipTests -Pcentos5 -Dpython.ver="python26" -Preplaceurl

. If your build is successful then you should see the ambari-metrics-*.rmp as well.

[#7]

@Jay SenSharma

OK. I'll try to build my metrics and try to run it again. I will notice you any results.

Thanks.

PS: As I'm using Ubuntu, the only necesary change will be using in the last command:

mvn -B clean install package jdeb:jdeb -DskipTests -Dpython.ver="python >= 2.6" -Preplaceurl

[#8]

@Jay SenSharma

I've tried it but it's not working fine. When I execute the mvn command, I'm getting the following error and fails to compile:

[INFO] Exclude: contrib/ambari-scom/metrics-sink/target/**
[INFO] Exclude: **/velocity.log*
[INFO] Exclude: ambari-metrics/ambari-metrics-grafana/conf/unix/ams-grafana.ini
[INFO] Exclude: ambari-metrics/target/**
[INFO] Exclude: ambari-metrics/ambari-metrics-host-monitoring/src/main/python/psutil/**
[INFO] Exclude: ambari-metrics/target/rpm/ambari-metrics/SPECS/ambari-metrics.spec
[INFO] Exclude: ambari-metrics/ambari-metrics-timelineservice/src/test/resources/lib/org/apache/phoenix/phoenix-core-tests/4.2.0/phoenix-core-tests-4.2.0.pom
[INFO] Exclude: ambari-metrics/ambari-metrics-timelineservice/src/test/resources/lib/org/apache/phoenix/phoenix-core-tests/maven-metadata-local.xml
[INFO] Exclude: ambari-metrics/*/target/**
[INFO] Exclude: ambari-metrics/**/.*/**
[INFO] 5004 resources included (use -debug for more details)
Warning:  org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not recognized.
Compiler warnings:
  WARNING:  'org.apache.xerces.jaxp.SAXParserImpl: Property 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.'
Warning:  org.apache.xerces.parsers.SAXParser: Feature 'http://javax.xml.XMLConstants/feature/secure-processing' is not recognized.
Warning:  org.apache.xerces.parsers.SAXParser: Property 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.
Warning:  org.apache.xerces.parsers.SAXParser: Property 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not recognized.

Also if I try to compile only the metrics with:

cd ambari-metrics
sudo mvn clean package -Dbuild-deb -DskipTests

It's failing with the output:

[INFO] utility ............................................ FAILURE [  1.907 s]
[INFO] ambari-metrics ..................................... SKIPPED
[INFO] Ambari Metrics Common .............................. SKIPPED
[INFO] Ambari Metrics Hadoop Sink ......................... SKIPPED
[INFO] Ambari Metrics Flume Sink .......................... SKIPPED
[INFO] Ambari Metrics Kafka Sink .......................... SKIPPED
[INFO] Ambari Metrics Storm Sink .......................... SKIPPED
[INFO] Ambari Metrics Storm Sink (Legacy) ................. SKIPPED
[INFO] Ambari Metrics Collector ........................... SKIPPED
[INFO] Ambari Metrics Monitor ............................. SKIPPED
[INFO] Ambari Metrics Grafana ............................. SKIPPED
[INFO] Ambari Metrics Assembly ............................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.192 s
[INFO] Finished at: 2017-04-03T09:31:21+02:00
[INFO] Final Memory: 27M/608M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.4.1:single (make-assembly) on project utility: No formats specified in the execution parameters or the assembly descriptor. -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

Thanks.

[#9]

@Adrián Gil

I think the link for the log should be : http://pastebin.com/hiEBjGcC/1 ( i removed the 'edit' string from url)

In your log i see that you are getting the following error: "Unable to locate package ambari-metrics-assembly"

2016-12-22 14:04:34,948 - Installing package ambari-metrics-assembly ('/usr/bin/apt-get -q -o Dpkg::Options::=--force-confdef --allow-unauthenticated --assume-yes install ambari-metrics-assembly')
2016-12-22 14:04:35,184 - Execution of '/usr/bin/apt-get -q -o -Dpkg::Options::=--force-confdef --allow-unauthenticated --assume-yes install ambari-metrics-assembly' returned 100. Reading package lists...

Unable to locate package ambari-metrics-assembly
2016-12-22 14:04:35,184 - Failed to install package ambari-metrics-assembly. Executing '/usr/bin/apt-get update -qq'

- Which means your ambari repository is not correct. Are you pointing to correct ambari.repo? And is that repo reachable?

.

Also your log timestamp seems to be too old. (2016-12-22) ntpd service is one of the requirement for HDP installation for time syncing.

-

Also the following link is for Ambari 2.4.2 (not for 2.5) but the post installation / upgrade ambari steps would remain same

http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-upgrade/content/upgrade_ambari_met... something like following:

# apt-get install ambari-metrics-assembly 
# apt-get install ambari-metrics-collector
# apt-get install ambari-metrics-grafana

.

[#10]

No sorry. I confused when copying the link and that's an older log. I update with the correct link, and thanks for the advise.