Error in service installing

[#1]

I'm installing Cloudera Manager 5.02 and encounter errors when installing services.

#########Error Info#############

A server error has occurred. Send the following information to Cloudera.

Path: http://192.168.28.40:7180/cmf/clusters/1/express-add-services/review

Version: Cloudera Express 5.0.2 (#297 built by jenkins on 20140606-2221 git: 80907df78ba6b50c21a598f0caff8b00685d5961)

java.lang.IllegalArgumentException:Host 093bced7-b29f-4716-8a60-8f209ac4d8d6 may not join service 'oozie': Host software version incompatible with service version
at OperationsManagerImpl.java line 760
in com.cloudera.server.cmf.components.OperationsManagerImpl createRole()

 Stack Trace:

1. OperationsManagerImpl.java line 760
   in com.cloudera.server.cmf.components.OperationsManagerImpl createRole()
2. RulesCluster.java line 586
   in com.cloudera.server.cmf.cluster.RulesCluster createAndConfigureServices()
3. ExpressAddServicesWizardController.java line 585
   in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController buildCustomCluster()
4. ExpressAddServicesWizardController.java line 521
   in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController buildCluster()
5. ExpressAddServicesWizardController.java line 492
   in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController buildAccs()
6. ExpressAddServicesWizardController.java line 433
   in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController handleReview()
7. ExpressAddServicesWizardController.java line 405
   in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController renderReviewStep()
8. <generated> line -1
   in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController$$FastClassByCGLIB$$71adc282 invoke()
9. MethodProxy.java line 191
   in net.sf.cglib.proxy.MethodProxy invoke()
10. Cglib2AopProxy.java line 688
    in org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation invokeJoinpoint()
11. ReflectiveMethodInvocation.java line 150
    in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
12. MethodSecurityInterceptor.java line 61
    in org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor invoke()
13. ReflectiveMethodInvocation.java line 172
    in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
14. Cglib2AopProxy.java line 621
    in org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor intercept()
15. <generated> line -1
    in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController$$EnhancerByCGLIB$$26be3314 renderReviewStep()
16. NativeMethodAccessorImpl.java line -2
    in sun.reflect.NativeMethodAccessorImpl invoke0()
17. NativeMethodAccessorImpl.java line 57
    in sun.reflect.NativeMethodAccessorImpl invoke()
18. DelegatingMethodAccessorImpl.java line 43
    in sun.reflect.DelegatingMethodAccessorImpl invoke()
19. Method.java line 606
    in java.lang.reflect.Method invoke()
20. HandlerMethodInvoker.java line 176
    in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
21. AnnotationMethodHandlerAdapter.java line 436
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
22. AnnotationMethodHandlerAdapter.java line 424
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
23. DispatcherServlet.java line 790
    in org.springframework.web.servlet.DispatcherServlet doDispatch()
24. DispatcherServlet.java line 719
    in org.springframework.web.servlet.DispatcherServlet doService()
25. FrameworkServlet.java line 669
    in org.springframework.web.servlet.FrameworkServlet processRequest()
26. FrameworkServlet.java line 574
    in org.springframework.web.servlet.FrameworkServlet doGet()
27. HttpServlet.java line 575
    in javax.servlet.http.HttpServlet service()
28. HttpServlet.java line 668
    in javax.servlet.http.HttpServlet service()
29. ServletHolder.java line 511
    in org.mortbay.jetty.servlet.ServletHolder handle()
30. ServletHandler.java line 1221
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
31. UserAgentFilter.java line 78
    in org.mortbay.servlet.UserAgentFilter doFilter()
32. GzipFilter.java line 131
    in org.mortbay.servlet.GzipFilter doFilter()
33. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
34. JAMonServletFilter.java line 48
    in com.jamonapi.http.JAMonServletFilter doFilter()
35. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
36. JavaMelodyFacade.java line 109
    in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
37. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
38. FilterChainProxy.java line 311
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
39. FilterSecurityInterceptor.java line 116
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
40. FilterSecurityInterceptor.java line 83
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
41. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
42. ExceptionTranslationFilter.java line 113
    in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
43. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
44. SessionManagementFilter.java line 101
    in org.springframework.security.web.session.SessionManagementFilter doFilter()
45. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
46. AnonymousAuthenticationFilter.java line 113
    in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
47. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
48. RememberMeAuthenticationFilter.java line 146
    in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
49. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
50. SecurityContextHolderAwareRequestFilter.java line 54
    in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
51. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
52. RequestCacheAwareFilter.java line 45
    in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
53. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
54. AbstractAuthenticationProcessingFilter.java line 182
    in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
55. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
56. LogoutFilter.java line 105
    in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
57. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
58. SecurityContextPersistenceFilter.java line 87
    in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
59. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
60. ConcurrentSessionFilter.java line 125
    in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
61. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
62. FilterChainProxy.java line 173
    in org.springframework.security.web.FilterChainProxy doFilter()
63. DelegatingFilterProxy.java line 237
    in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
64. DelegatingFilterProxy.java line 167
    in org.springframework.web.filter.DelegatingFilterProxy doFilter()
65. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
66. CharacterEncodingFilter.java line 88
    in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
67. OncePerRequestFilter.java line 76
    in org.springframework.web.filter.OncePerRequestFilter doFilter()
68. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
69. ServletHandler.java line 399
    in org.mortbay.jetty.servlet.ServletHandler handle()
70. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
71. SessionHandler.java line 182
    in org.mortbay.jetty.servlet.SessionHandler handle()
72. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
73. ContextHandler.java line 766
    in org.mortbay.jetty.handler.ContextHandler handle()
74. WebAppContext.java line 450
    in org.mortbay.jetty.webapp.WebAppContext handle()
75. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
76. StatisticsHandler.java line 53
    in org.mortbay.jetty.handler.StatisticsHandler handle()
77. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
78. Server.java line 326
    in org.mortbay.jetty.Server handle()
79. HttpConnection.java line 542
    in org.mortbay.jetty.HttpConnection handleRequest()
80. HttpConnection.java line 928
    in org.mortbay.jetty.HttpConnection$RequestHandler headerComplete()
81. HttpParser.java line 549
    in org.mortbay.jetty.HttpParser parseNext()
82. HttpParser.java line 212
    in org.mortbay.jetty.HttpParser parseAvailable()
83. HttpConnection.java line 404
    in org.mortbay.jetty.HttpConnection handle()
84. SelectChannelEndPoint.java line 410
    in org.mortbay.io.nio.SelectChannelEndPoint run()
85. QueuedThreadPool.java line 582
    in org.mortbay.thread.QueuedThreadPool$PoolThread run()

[#2]

This usually happens when you have manually installed CDH, but picked the wrong CDH version in the CM UI.

Exit the wizard (click the Cloudera Manager logo), click the Hosts tab, and run the Host Inspector. It should say what versions of CDH are installed. If those are incorrect, fix them.

Then run the Add Cluster wizard again (if you see an empty cluster on the home page, you may need to delete that cluster first), and be sure to pick the correct major CDH version.

Thanks,
Darren

[#3]

Here is the result of Inspector.

Should I reinstall the CDH?

######################################

Component Version Release CDH Version

Parquet	1.2.5+cdh5.0.3+97	1.cdh5.0.3.p0.22	CDH 5
Impala	1.3.1+cdh5.0.3+0	1.cdh5.0.3.p0.24	CDH 5
YARN	2.3.0+cdh5.0.3+597	1.cdh5.0.3.p0.30	CDH 5
HDFS	2.3.0+cdh5.0.3+597	1.cdh5.0.3.p0.30	CDH 5
hue-common	3.5.0+cdh5.0.3+381	1.cdh5.0.3.p0.18	CDH 5
Sqoop2	1.99.3+cdh5.0.3+32	1.cdh5.0.3.p0.14	CDH 5
MapReduce 2	2.3.0+cdh5.0.3+597	1.cdh5.0.3.p0.30	CDH 5
HBase	0.96.1.1+cdh5.0.3+78	1.cdh5.0.3.p0.22	CDH 5
Sqoop	1.4.4+cdh5.0.3+50	1.cdh5.0.3.p0.14	CDH 5
Oozie	4.0.0+cdh5.0.3+185	1.cdh5.0.3.p0.17	CDH 5
Zookeeper	3.4.5+26	1.cdh4.7.0.p0.17	CDH 4
Hue	3.5.0+cdh5.0.3+381	1.cdh5.0.3.p0.18	CDH 5
Lily HBase Indexer	1.3+26	1.cdh4.5.0.p0.14	CDH 4
MapReduce 1	2.3.0+cdh5.0.3+597	1.cdh5.0.3.p0.30	CDH 5
Pig	0.12.0+cdh5.0.3+36	1.cdh5.0.3.p0.18	CDH 5
Crunch (CDH 5 only)	0.9.0+cdh5.0.3+27	1.cdh5.0.3.p0.20	CDH 5
Llama (CDH 5 only)	1.0.0+cdh5.0.3+0	1.cdh5.0.3.p0.23	CDH 5
HttpFS	2.3.0+cdh5.0.3+597	1.cdh5.0.3.p0.30	CDH 5
Hadoop	2.3.0+cdh5.0.3+597	1.cdh5.0.3.p0.30	CDH 5
Hive	0.12.0+cdh5.0.3+322	1.cdh5.0.3.p0.22	CDH 5
HCatalog	0.12.0+cdh5.0.3+322	1.cdh5.0.3.p0.22	CDH 5
spark	0.9.0+cdh5.0.3+38	1.cdh5.0.3.p0.21	CDH 5
Solr	4.4.0+181	1.cdh4.5.0.p0.14	CDH 4
Flume NG	1.4.0+98	1.cdh4.7.0.p0.17	CDH 4
Cloudera Manager Management Daemons	5.0.2	1.cm502.p0.297	Not applicable
Java 6	java version "1.6.0_31" Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)	Unavailable	Not applicable
Java 7	java version "1.7.0_45" Java(TM) SE Runtime Environment (build 1.7.0_45-b18) Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)	Unavailable	Not applicable
Cloudera Manager Agent	5.0.2	1.cm502.p0.297.el5	Not applicable

[#4]

I uninstall CM and CDH following the guide

http://cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Cloudera-Manager-Quick-Start/C...

Then I reinstall CM. However, it still doesn't work.

[#5]

Hi Sky,

Host inspector showed that you had inconsistent versions of CDH installed.

Log in to each machine and consult the package manager (such as yum for rhel-based systems) to ensure these were all correctly removed, or are the correct version. If you are using Parcels to install CDH (which is recommended), then you should have no CDH packages installed on your operating system.

Thanks,
Darren

[#6]

I uninstall CM and CDH again following the guide I posted before, and then reinstall CM.

However the error happened again.

Here is the result of host inspector.

The version of oozie is CDH5 now.

############################

component       Version Release CDH Version
Parquet 1.2.5+cdh5.1.0+130      1.cdh5.1.0.p0.26        CDH 5
Impala  1.4.0+cdh5.1.0+0        1.cdh5.1.0.p0.92        CDH 5
YARN    2.3.0+cdh5.1.0+795      1.cdh5.1.0.p0.58        CDH 5
HDFS    2.3.0+cdh5.1.0+795      1.cdh5.1.0.p0.58        CDH 5
hue-common      3.6.0+cdh5.1.0+86       1.cdh5.1.0.p0.36        CDH 5
Sqoop2  1.99.3+cdh5.1.0+26      1.cdh5.1.0.p0.22        CDH 5
MapReduce 2     2.3.0+cdh5.1.0+795      1.cdh5.1.0.p0.58        CDH 5
HBase   0.98.1+cdh5.1.0+64      1.cdh5.1.0.p0.34        CDH 5
Sqoop   1.4.4+cdh5.1.0+55       1.cdh5.1.0.p0.24        CDH 5
Oozie   4.0.0+cdh5.1.0+249      1.cdh5.1.0.p0.28        CDH 5
Zookeeper       3.4.5+26        1.cdh4.7.0.p0.17        CDH 4
Hue     3.6.0+cdh5.1.0+86       1.cdh5.1.0.p0.36        CDH 5
spark   1.0.0+cdh5.1.0+41       1.cdh5.1.0.p0.27        CDH 5
MapReduce 1     2.3.0+cdh5.1.0+795      1.cdh5.1.0.p0.58        CDH 5
Pig     0.12.0+cdh5.1.0+33      1.cdh5.1.0.p0.23        CDH 5
Crunch (CDH 5 only)     0.10.0+cdh5.1.0+14      1.cdh5.1.0.p0.25        CDH 5
Llama (CDH 5 only)      1.0.0+cdh5.1.0+0        1.cdh5.1.0.p0.25        CDH 5
HttpFS  2.3.0+cdh5.1.0+795      1.cdh5.1.0.p0.58        CDH 5
Hadoop  2.3.0+cdh5.1.0+795      1.cdh5.1.0.p0.58        CDH 5
Hive    0.12.0+cdh5.1.0+369     1.cdh5.1.0.p0.39        CDH 5
HCatalog        0.12.0+cdh5.1.0+369     1.cdh5.1.0.p0.39        CDH 5
sentry  1.3.0+cdh5.1.0+155      1.cdh5.1.0.p0.59        CDH 5
Lily HBase Indexer      1.5+cdh5.1.0+12 1.cdh5.1.0.p0.41        CDH 5
Solr    4.4.0+181       1.cdh4.5.0.p0.14        CDH 4
Flume NG        1.5.0+cdh5.1.0+10       1.cdh5.1.0.p0.26        CDH 5
Cloudera Manager Management Daemons     5.1.0   1.cm510.p0.75   Not applicable
Java 6  java version "1.6.0_31" Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)     Unavailable     Not applicable
Java 7  java version "1.7.0_55" Java(TM) SE Runtime Environment (build 1.7.0_55-b13) Java HotSpot(TM) 64-Bit Server VM (build 24.55-b03, mixed mode)    Unavailable     Not applicable

[#7]

Try checking the /var/log/cloudera-scm-server/cloudera-scm-server.log files for a WARN message that looks like this:

Host %s (derived version %s) not eligible for cluster %s

The %s will be replaced with the values generated from the validity check.

This should show us what verion was detected on the node and hopefully give us an idea of why the failure is occurring.

[#8]

Here is related WARN info in cloudera-scm-server.log.

###################

2014-07-18 17:48:35,166  WARN [1836711262@scm-web-14:service.HostUtils@158] Host DbHost{id=3, hostId=1792acd5-c10f-4ae0-b46a-eb1fc1662102, hostName=master} (derived version MIXED) not eligible for cluster DbCluster{id=2, name=cluster}
2014-07-18 17:48:36,054  WARN [1836711262@scm-web-14:service.HostUtils@158] Host DbHost{id=3, hostId=1792acd5-c10f-4ae0-b46a-eb1fc1662102, hostName=master} (derived version MIXED) not eligible for cluster DbCluster{id=2, name=cluster}
2014-07-18 17:56:13,450  WARN [1836711262@scm-web-14:service.HostUtils@158] Host DbHost{id=3, hostId=1792acd5-c10f-4ae0-b46a-eb1fc1662102, hostName=master} (derived version MIXED) not eligible for cluster DbCluster{id=2, name=cluster}
2014-07-18 17:57:23,098  WARN [768616541@scm-web-21:service.HostUtils@158] Host DbHost{id=3, hostId=1792acd5-c10f-4ae0-b46a-eb1fc1662102, hostName=master} (derived version MIXED) not eligible for cluster DbCluster{id=2, name=cluster}
2014-07-18 17:57:23,790  WARN [768616541@scm-web-21:service.HostUtils@158] Host DbHost{id=3, hostId=1792acd5-c10f-4ae0-b46a-eb1fc1662102, hostName=master} (derived version MIXED) not eligible for cluster DbCluster{id=2, name=cluster}
2014-07-18 17:57:28,236  WARN [768616541@scm-web-21:service.HostUtils@158] Host DbHost{id=3, hostId=1792acd5-c10f-4ae0-b46a-eb1fc1662102, hostName=master} (derived version MIXED) not eligible for cluster DbCluster{id=2, name=cluster}
2014-07-18 17:57:46,904  WARN [1836711262@scm-web-14:service.HostUtils@158] Host DbHost{id=3, hostId=1792acd5-c10f-4ae0-b46a-eb1fc1662102, hostName=master} (derived version MIXED) not eligible for cluster DbCluster{id=2, name=cluster}
2014-07-18 17:57:47,518  WARN [1836711262@scm-web-14:service.HostUtils@158] Host DbHost{id=3, hostId=1792acd5-c10f-4ae0-b46a-eb1fc1662102, hostName=master} (derived version MIXED) not eligible for cluster DbCluster{id=2, name=cluster}

[#9]

The host inspector pretty clearly indicated exactly which components were detected as CDH 4.

For each of these, make sure the old versions were correctly uninstalled. Whatever steps you are following to uninstall is not working, possibly because you are using the wrong steps, and possibly because your package manager is in a weird state not anticipated by the uninstall instructions.

Thanks,
Darren

[#10]

Is there any way I can check whether I uninstall the components correctly after I uninstall the Cloudera Manager?

For example the remaining files?

By the way, is it possible that it is the installer's problem? Because my network is under some constraints. Is it possible that the installer use CDH4 package or source when the CDH5 cannot be download?