API call to Cloudera Manager failed for Cloudera Express

Da — Fri, 16 Sep 2022 15:52:50 GMT

Recently upgraded Director from 6.1 to 6.2.1 and noticed a lot more spam in the logs with the following trace:

[2019-07-22 17:17:27.632 +0000] ERROR [task-thread-10] - - - - - com.cloudera.api.ext.ClouderaManagerClientProxy: API call to Cloudera Manager failed, gathering information
javax.ws.rs.NotFoundException: HTTP 404 Not Found
	at sun.reflect.GeneratedConstructorAccessor246.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.cxf.jaxrs.client.AbstractClient.convertToWebApplicationException(AbstractClient.java:544)
	at org.apache.cxf.jaxrs.client.ClientProxyImpl.checkResponse(ClientProxyImpl.java:326)
	at org.apache.cxf.jaxrs.client.ClientProxyImpl.handleResponse(ClientProxyImpl.java:891)
	at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:797)
	at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:238)
	at com.sun.proxy.$Proxy246.readLicense(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor494.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.cloudera.api.ext.ClouderaManagerClientProxy.invoke(ClouderaManagerClientProxy.java:117)
	at com.sun.proxy.$Proxy246.readLicense(Unknown Source)
	at com.cloudera.launchpad.bootstrap.deployment.util.DeploymentRefresher.checkEnterpriseTrial(DeploymentRefresher.java:358)
	at com.cloudera.launchpad.bootstrap.deployment.util.DeploymentRefresher.refreshCmDetails(DeploymentRefresher.java:298)
	at com.cloudera.launchpad.bootstrap.deployment.util.DeploymentRefresher.refreshDeploymentEntity(DeploymentRefresher.java:216)
	at com.cloudera.launchpad.task.RefreshDeployments.doRefresh(RefreshDeployments.java:118)
	at com.cloudera.launchpad.task.RefreshModelsTask.refreshDeployments(RefreshModelsTask.java:42)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Turning on API Debugging for one of the CMs gives me the following:

2019-07-22 17:16:32,783 DEBUG 994488843@scm-web-4270:com.cloudera.api.ApiExceptionMapper: Exception caught in API invocation.
java.util.NoSuchElementException: This installation is currently running Cloudera Express.
	at com.cloudera.api.dao.impl.LicenseManagerDaoImpl.readLicense(LicenseManagerDaoImpl.java:54)
	at com.cloudera.api.v1.impl.ClouderaManagerResourceImpl.readLicense(ClouderaManagerResourceImpl.java:52)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180)
	at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
	at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:192)
	at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:116)
	at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:256)
	at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:116)
	at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:256)
	at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:116)
	at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:100)
	at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57)
	at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93)
	at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272)
	at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
	at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239)
	at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:248)
	at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:222)
	at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:153)
	at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:167)
	at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:286)
	at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:211)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:262)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
	at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:109)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:101)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at com.cloudera.api.ApiBasicAuthFilter.doFilter(ApiBasicAuthFilter.java:44)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.handler.StatisticsHandler.handle(StatisticsHandler.java:53)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2019-07-22 17:16:32,783 DEBUG 994488843@scm-web-4270:com.cloudera.api.ApiInvoker: API Error 404 [/api/v14/cm/license]: ApiErrorMessage{This installation is currently running Cloudera Express.}
2019-07-22 17:16:32,783 DEBUG 994488843@scm-web-4270:com.cloudera.api.ApiInvoker: API Error 404 [/api/v14/cm/license]: ApiErrorMessage{This installation is currently running Cloudera Express.}
2019-07-22 17:16:32,783 DEBUG 994488843@scm-web-4270:com.cloudera.api.ApiInvoker: API Error 404 [/api/v14/cm/license]: ApiErrorMessage{This installation is currently running Cloudera Express.}
2019-07-22 17:16:32,783 DEBUG 994488843@scm-web-4270:com.cloudera.api.LoggingOutInterceptor: API response:
---------- id: 7
Response code: 404

So it looks like this version of Director doesn't respect the Deployment Template containing "License Type: Cloudera Express" when running this task or at least doesn't handle the (expected/unexpected?) 404 nicely!

Just creating this forum post in case anybody else sees the same spam (which with a lot of deployments can be annoying).

Cheers!

Re: API call to Cloudera Manager failed for Cloudera Express

Mike Wilson — Mon, 22 Jul 2019 20:16:19 GMT

What version of CM are you using?

Re: API call to Cloudera Manager failed for Cloudera Express

Da — Mon, 22 Jul 2019 20:20:27 GMT

Oops looks like I didn't include that! I'm using CM v5.16.2.

Re: API call to Cloudera Manager failed for Cloudera Express

Da — Mon, 22 Jul 2019 20:34:10 GMT

For reference the same issue is seen with a CM 6.1 deployment I just tested with.

Re: API call to Cloudera Manager failed for Cloudera Express

Da — Wed, 24 Jul 2019 16:00:57 GMT

@Mike Wilson is there anyway to disable this logging or call?

The logs really spam this as every health check for every deployment runs this:

grep com.cloudera.api.ext.ClouderaManagerClientProxy /var/log/cloudera-director-server/application.log* | wc -l
13063

Probably need to configure logback.xml but I'm not trying to silence a real error.

Re: API call to Cloudera Manager failed for Cloudera Express

Bill Havanki — Thu, 29 Aug 2019 14:56:19 GMT

Hi Da,

You're correct, you can edit the Altus Director server's logback.xml file to have it not emit log messages for that Java class. It would look like this.

 <logger name="com.cloudera.api.ext.ClouderaManagerClientProxy" level="INFO" />

More details on editing logback.xml are here: https://www.cloudera.com/documentation/director/latest/topics/director_troubleshoot.html#director-intro__d72814e37

While those debug lines are noisy, they can be helpful when problems arise. So, in the future, if you are troubleshooting problems with Altus Director and Cloudera Manager, flipping that logger back on might reveal some useful information.

Re: API call to Cloudera Manager failed for Cloudera Express

Da — Thu, 26 Sep 2019 21:51:29 GMT

Thanks for this info (and link) and duly noted on troubleshooting in the future!

question API call to Cloudera Manager failed for Cloudera Express in Support Questions

API call to Cloudera Manager failed for Cloudera Express

Re: API call to Cloudera Manager failed for Cloudera Express

Re: API call to Cloudera Manager failed for Cloudera Express

Re: API call to Cloudera Manager failed for Cloudera Express

Re: API call to Cloudera Manager failed for Cloudera Express

Re: API call to Cloudera Manager failed for Cloudera Express

Re: API call to Cloudera Manager failed for Cloudera Express