Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

yarn queue manager error: Couldn't connect to the cluster RA040 I/O error while requesting Ambari....

avatar
Contributor

when i try to acess yarn queue manager it is showing below error

error.txt

Please let me know how to solve this issue.

Couldn't connect to the cluster

RA040 I/O error while requesting Ambari

TRACE

 org.apache.ambari.view.utils.ambari.AmbariApiException: RA040 I/O error while requesting Ambari
  at org.apache.ambari.view.utils.ambari.AmbariApi.requestClusterAPI(AmbariApi.java:106)
  at org.apache.ambari.view.utils.ambari.AmbariApi.requestClusterAPI(AmbariApi.java:77)
  at org.apache.ambari.view.capacityscheduler.ConfigurationService.readFromCluster(ConfigurationService.java:340)
  at org.apache.ambari.view.capacityscheduler.ConfigurationService.readClusterInfo(ConfigurationService.java:156)
  at sun.reflect.GeneratedMethodAccessor2590.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:497)
  at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
  at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
  at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
  at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
  at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
  at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
  at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
  at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
  at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
  at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
  at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
  at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
  at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
  at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
  at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
  at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
  at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
  at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
  at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
  at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
  at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
  at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
  at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
  at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
  at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
  at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1507)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
  at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
  at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
  at org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAuthorizationFilter.java:287)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
  at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
  at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
  at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
  at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
  at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
  at org.apache.ambari.server.security.authentication.AmbariDelegatingAuthenticationFilter.doFilter(AmbariDelegatingAuthenticationFilter.java:132)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
  at org.apache.ambari.server.security.authorization.AmbariUserAuthorizationFilter.doFilter(AmbariUserAuthorizationFilter.java:91)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
  at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
  at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
  at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
  at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
  at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
  at org.apache.ambari.server.api.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:72)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
  at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
  at org.apache.ambari.server.view.AmbariViewsMDCLoggingFilter.doFilter(AmbariViewsMDCLoggingFilter.java:54)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
  at org.apache.ambari.server.view.ViewThrottleFilter.doFilter(ViewThrottleFilter.java:161)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
  at org.apache.ambari.server.security.AbstractSecurityHeaderFilter.doFilter(AbstractSecurityHeaderFilter.java:125)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
  at org.apache.ambari.server.security.AbstractSecurityHeaderFilter.doFilter(AbstractSecurityHeaderFilter.java:125)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
  at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
  at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
  at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:427)
  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
  at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:212)
  at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:201)
  at org.apache.ambari.server.controller.AmbariHandlerList.handle(AmbariHandlerList.java:150)
  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
  at org.eclipse.jetty.server.Server.handle(Server.java:370)
  at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
  at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:973)
  at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1035)
  at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:641)
  at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:231)
  at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
  at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
  at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
  at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
  at java.lang.Thread.run(Thread.java:745)
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
  at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
  at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949)
  at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
  at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
  at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1506)
  at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
  at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
  at sun.security.ssl.Handshaker.process_record(Handshaker.java:914)
  at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062)
  at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
  at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
  at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
  at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
  at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
  at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1512)
  at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
  at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
  at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
  at org.apache.ambari.server.controller.internal.URLStreamProvider.processURL(URLStreamProvider.java:218)
  at org.apache.ambari.server.view.ViewAmbariStreamProvider.getInputStream(ViewAmbariStreamProvider.java:123)
  at org.apache.ambari.server.view.ViewAmbariStreamProvider.readFrom(ViewAmbariStreamProvider.java:85)
  at org.apache.ambari.view.utils.ambari.AmbariApi.requestClusterAPI(AmbariApi.java:103)
  ... 102 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
  at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
  at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
  at sun.security.validator.Validator.validate(Validator.java:260)
  at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
  at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
  at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
  at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1488)
  ... 119 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
  at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:146)
  at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:131)
  at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
  at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
  ... 125 more
1 ACCEPTED SOLUTION

avatar
Master Collaborator
To resolve the issue, import the Ambari certificates to the Ambari truststore. To import the Ambari certificates, do the following: 

STEP 1: 
Get certificate from ambari-server 
echo | openssl s_client -showcerts -connect <AMBARI_HOst>:<AMBARI_HTTPs_PORT> 2>&1 | sed --quiet '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /tmp/ambari_certificate.cr 

STEP 2: 
Get path of ambari trustore and truststore password from Ambari properties 
cat /etc/ambari-server/conf/ambari.properties |grep truststore 

As per your ambari.properties below is the path and password :-
ssl.trustStore.password=refer from ambari.property file
ssl.trustStore.path=/etc/ambari-server/conf/ambari-server-truststore

STEP 3: 
keytool -importcert -file /tmp/ambari_certificate.crt -keystore <keystore-path> 

STEP 4: 
ambari-server restart

View solution in original post

4 REPLIES 4

avatar

@kanna k

Try to set below parameters, hope it should work:

add to ambari.properties:

client.threadpool.size.max = 50

and then ambari-server restart

avatar
Master Mentor

@kanna k

Looks like you have configured HTTPS for you ambari server and accessing ambari server using HTTPS. And you do not have ambari truststore configured with the ambari certificate imported to it. Hence you are getting the following error:

Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

.

In order to resolve this issue: You case you will need to Ambari certificates to the Ambari truststore.

1. Setup Ambari Truststore as described in: https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.1.5/bk_ambari-security/content/set_up_truststor...

2. Get the Ambari certificate as following:

# echo | openssl s_client -showcerts -connect <AMBARI_HOST>:<AMBARI_HTTPS_PORT> 2>&1 | sed --quiet '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /tmp/ambari_certificate.crt

3. Suppose if your ambari truststore file name is "truststore.jks" then import the ambari_certificate to it as following:

# $JAVA_HOME/bin/keytool -importcert -file /tmp/ambari_certificate.crt -keystore truststore.jks

4. Restart Ambari Server.

5. Access the CS view again.

.

avatar
Master Mentor

avatar
Master Collaborator
To resolve the issue, import the Ambari certificates to the Ambari truststore. To import the Ambari certificates, do the following: 

STEP 1: 
Get certificate from ambari-server 
echo | openssl s_client -showcerts -connect <AMBARI_HOst>:<AMBARI_HTTPs_PORT> 2>&1 | sed --quiet '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /tmp/ambari_certificate.cr 

STEP 2: 
Get path of ambari trustore and truststore password from Ambari properties 
cat /etc/ambari-server/conf/ambari.properties |grep truststore 

As per your ambari.properties below is the path and password :-
ssl.trustStore.password=refer from ambari.property file
ssl.trustStore.path=/etc/ambari-server/conf/ambari-server-truststore

STEP 3: 
keytool -importcert -file /tmp/ambari_certificate.crt -keystore <keystore-path> 

STEP 4: 
ambari-server restart