Member since
10-02-2017
116
Posts
3
Kudos Received
8
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
913 | 07-18-2020 12:04 PM | |
1560 | 09-11-2019 01:14 PM | |
2280 | 08-16-2019 08:17 AM | |
5915 | 08-15-2019 12:23 PM | |
3853 | 05-14-2019 08:48 AM |
08-15-2019
12:23 PM
OK - so it seems that the yum repo for cloudera manager was defined / pinned to 6.2 in the cluster definition - so of course Director did what it was being asked to do. Updated to point to 6.3. repository: "https://archive.cloudera.com/cm6/6.3/redhat7/yum/" repositoryKeyUrl: "https://archive.cloudera.com/cm6/6.3/redhat7/yum/RPM-GPG-KEY-cloudera"
... View more
08-15-2019
09:54 AM
Any suggestions for why director is deploying conflicting versions of the agent and parcels? Seems like a fundamental issue.
... View more
08-15-2019
05:56 AM
We use templates to deploy our clusters. I've discovered why the parcel activation is failing. Version 6.2 agents are being installed while version 6.3 parcels are being distributed / activated. This same template worked fine when 6 was originally released. In the template we have: cluster {
products {
CDH: 6 We are not defining a parcel URL, which I assume means we are using the default based on the version of director, which is 6.3.0. I've tried defining more specific version of CDH above (6.2, 6.3) but director continues to deploy 6.2 agents and 6.3 parcels.
... View more
08-14-2019
05:49 PM
I tried another deployment. Here are some entires that standout in the manager server log - at least to me: 2019-08-15 00:09:29,753 WARN scm-web-93:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,754 INFO scm-web-93:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.158 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:29,755 WARN scm-web-177:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,757 INFO scm-web-177:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.246 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:29,771 WARN scm-web-282:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,774 INFO scm-web-282:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.244 over PT0.003S. Rate: 25.352 MB/s 2019-08-15 00:09:29,785 WARN scm-web-161:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,785 INFO scm-web-161:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.191 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:29,819 WARN scm-web-282:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,819 INFO scm-web-282:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.215 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:29,827 WARN scm-web-93:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,827 INFO scm-web-93:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.226 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:29,868 WARN scm-web-91:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,869 INFO scm-web-91:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.208 over PT0.001S. Rate: 76.056 MB/s 2019-08-15 00:09:29,874 WARN scm-web-282:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,874 INFO scm-web-282:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.250 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:29,874 WARN scm-web-290:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,875 INFO scm-web-290:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.241 over PT0.001S. Rate: 76.056 MB/s 2019-08-15 00:09:29,905 WARN scm-web-282:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,905 INFO scm-web-282:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.243 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:29,914 WARN scm-web-170:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,915 INFO scm-web-170:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.137 over PT0.001S. Rate: 76.056 MB/s 2019-08-15 00:09:29,920 WARN scm-web-157:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,925 INFO scm-web-157:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.185 over PT0.001S. Rate: 76.056 MB/s 2019-08-15 00:09:29,928 WARN scm-web-177:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,929 INFO scm-web-177:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.138 over PT0.001S. Rate: 76.056 MB/s 2019-08-15 00:09:29,993 WARN scm-web-282:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,993 INFO scm-web-282:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.165 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:29,993 WARN scm-web-157:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,994 INFO scm-web-157:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.149 over PT0.001S. Rate: 76.056 MB/s 2019-08-15 00:09:30,018 WARN scm-web-157:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:30,018 INFO scm-web-157:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.150 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:30,030 WARN scm-web-94:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:30,030 INFO scm-web-94:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.188 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:30,034 INFO scm-web-89:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 172.24.1.165 2019-08-15 00:09:30,048 WARN scm-web-157:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:30,048 INFO scm-web-157:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.247 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:30,065 WARN scm-web-94:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent Then there are several INFO entries which seem to indicate parcels are being distributed / activated successfully until we hit the following error, after which Altus Director fails the deployment: /clusters/ez2/parcels/products/CDH/versions/6.3.0-1.cdh6.3.0.p0.1279813/commands/activate 2019-08-15 00:12:19,738 INFO scm-web-290:com.cloudera.parcel.components.ParcelManagerImpl: Activating parcel CDH:6.3.0-1.cdh6.3.0.p0.1279813 on cluster ez2 2019-08-15 00:12:19,756 INFO scm-web-290:com.cloudera.enterprise.AbstractWrappedEntityManager: Rolling back transaction that wasn't marked for rollback-only. java.lang.Exception: Non-thrown exception for stack trace. at com.cloudera.enterprise.AbstractWrappedEntityManager.close(AbstractWrappedEntityManager.java:161) at com.cloudera.cmf.persist.CmfEntityManager.close(CmfEntityManager.java:367) at com.cloudera.server.web.cmf.AuthFilterEntityManager.close(AuthFilterEntityManager.java:191) at com.cloudera.api.dao.impl.ManagerDaoBase.runInNewTransaction(ManagerDaoBase.java:209) at com.cloudera.api.dao.impl.ManagerDaoBase.access$100(ManagerDaoBase.java:82) at com.cloudera.api.dao.impl.ManagerDaoBase$TransactionCallable.call(ManagerDaoBase.java:239) at com.cloudera.server.common.RetryWrapper.executeWithRetry(RetryWrapper.java:32) at com.cloudera.server.common.RetryUtils.executeWithRetryHelper(RetryUtils.java:210) at com.cloudera.server.common.RetryUtils.executeWithRetry(RetryUtils.java:131) at com.cloudera.api.dao.impl.ManagerDaoBase.runInNewTransactionWithRetry(ManagerDaoBase.java:169) at com.cloudera.api.dao.impl.ManagerDaoBase.invoke(ManagerDaoBase.java:272) at com.sun.proxy.$Proxy205.activate(Unknown Source) at com.cloudera.api.v3.impl.ParcelResourceImpl.activateCommand(ParcelResourceImpl.java:70) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180) at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:189) at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:117) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:261) at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:117) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:261) at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:117) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:261) at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:117) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:261) at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:117) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:99) at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96) at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:263) at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234) at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208) at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160) at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:189) at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:299) at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:218) at javax.servlet.http.HttpServlet.service(HttpServlet.java:665) at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:274) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:867) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623) at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:200) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at com.cloudera.api.ApiBasicAuthFilter.doFilter(ApiBasicAuthFilter.java:86) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:502) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at com.cloudera.server.common.BoundedQueuedThreadPool$2.run(BoundedQueuedThreadPool.java:94) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.lang.Thread.run(Thread.java:748) 2019-08-15 00:12:19,759 INFO scm-web-290:com.cloudera.api.ApiExceptionMapper: Exception caught in API invocation. Msg:The version change by activating the parcel would cause an error java.lang.IllegalArgumentException: The version change by activating the parcel would cause an error
... View more
08-14-2019
04:49 PM
The log entry from Altus Director I provided mirrors the error I saw on the manager node while tailing the logs there. Parcel distribution / activation seemed to be going fine until it hit the error I posted. Unfortunately I've terminated the deployment as this was our 4th attempt. I was thinking of pinning CDH version in the cluster bootstrap to 6.2 instead of 6 so that it doesn't grab the latest (6.3) parcels. If I hit the same issue, I will rerun my original config and provide more logs from the manager node.
... View more
08-14-2019
04:10 PM
Altus Director 6.3 Bootstrap of CDH6.3 clusters are failing due to failed activation of parcels. We've successfully deployed earlier versions of CDH 6.x. On all nodes the agent logs show the following error for the same parcel: [14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel INFO Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'spark', '-K', 'UMASK=022', '--home', '/var/lib/spark', '--comment', 'Spark', '--shell', '/sbin/nologin', 'spark']
[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel INFO Ensuring correct file permissions for new parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813.
[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/desktop/desktop.db hue hue
[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel INFO Executing command ['chown', 'hue:hue', u'/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/desktop/desktop.db']
[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/desktop/desktop.db 0660
[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel INFO Executing command ['chmod', '0660', u'/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/desktop/desktop.db']
[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/desktop hue hue
[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel INFO Executing command ['chown', 'hue:hue', u'/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/desktop']
[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/desktop 0755
[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel INFO Executing command ['chmod', '0755', u'/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/desktop']
[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hadoop-yarn/bin/container-executor root yarn
[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel INFO Executing command ['chown', 'root:yarn', u'/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hadoop-yarn/bin/container-executor']
[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hadoop-yarn/bin/container-executor 6050
[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel INFO Executing command ['chmod', '6050', u'/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hadoop-yarn/bin/container-executor']
[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel ERROR Error while attempting to modify permissions of file '/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hadoop-0.20-mapreduce/sbin/Linux/task-controller'.
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/parcel.py", line 520, in ensure_permissions
file = cmf.util.validate_and_open_fd(path, self.get_parcel_home(parcel))
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/util/__init__.py", line 358, in validate_and_open_fd
fd = os.open(path, flags)
OSError: [Errno 2] No such file or directory: '/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hadoop-0.20-mapreduce/sbin/Linux/task-controller' Altus director server log shows: 2019-08-14 22:45:35.535 +0000] ERROR [p-b96f4ed12bad-DefaultBootstrapClusterJob] 3812e6e2-ba96-42c5-b53e-4cd2e99a54de POST /api/d6.2/environments/ez2/deployments/ez2/clusters com.cloudera.launchpad.bootstrap.cluster.ActivateParcel - com.cloudera.launchpad.pipeline.util.PipelineRunner: Attempt to execute job failed
com.cloudera.api.ext.ClouderaManagerException: API call to Cloudera Manager failed. Method=ParcelResource.activateCommand. Response Status Code: 400. Message: { "message" : "The version change by activating the parcel would cause an error"
... View more
Labels:
- Labels:
-
Cloudera Manager
07-18-2019
09:25 AM
I'm in the middle of another deployment where I see random failures reported by director when attempting to connect to port 22 of cluster instances during bootstrap. I'm seeing this for deployments where I haven't enabled publicip - so it's no just for publicip enabled deployments. Each time, it's only 1 or 2 of the instances and each time I can connect to the instance from the director server just fine, even during the retry window when director still claims it cannot connect. Not sure why director thinks it can't connect when I clearly can from the same server. I've confirmed the forward/reverse DNS resolution is working. I basically have to deploy several times before I hit the sweet spot and the deployment completes. This type of inconsistent behavior has been systemic with Azure deployments. I've experienced none of this with AWS deployments. I'll attempt to collect more logs.
... View more
07-18-2019
05:43 AM
Thanks for your reply. Bootstrap did fail in this case, which surprised me given the fact that during the retry window I could ssh to the instance's private IP from Director. Also, Director had no problem connecting to the other 2 master nodes that I enabled public IP on. The log entries I provided are repeated over and over until the retries are exhausted.
... View more