Error during service installation

[#1]

First in order to test the database login connectivity during installation path A, the server name was missing, (i.e. blank:7432 instead of servername:7432) and needed to be added.  Once it oast the connection /. login id pwd verrification test, I clicked continue and received the following error:

Error

A server error has occurred. Send the following information to Cloudera.

Path: http://192.168.0.102:7180/cmf/clusters/1/express-add-services/review

Version: Cloudera Enterprise Data Hub Trial 5.0.0-beta-2 (#119 built by jenkins on 20140209-0301 git: 8acd3c5559ccf82bf374d49bbf00bce58dee286e) 

 
java.lang.NullPointerException:Service must not be null to generate role name
 at Preconditions.java line 208
 in com.google.common.base.Preconditions checkNotNull()

 Stack Trace:
1. Preconditions.java line 208
 in com.google.common.base.Preconditions checkNotNull()
2. DbRoleNameGenerator.java line 85
 in com.cloudera.server.cmf.DbRoleNameGenerator generate()
3. RulesCluster.java line 406
 in com.cloudera.server.cmf.cluster.RulesCluster assignRole()
4. ExpressAddServicesWizardController.java line 587
 in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController buildCustomCluster()
5. ExpressAddServicesWizardController.java line 521
 in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController buildCluster()
6. ExpressAddServicesWizardController.java line 492
 in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController buildAccs()
7. ExpressAddServicesWizardController.java line 433
 in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController handleReview()
8. ExpressAddServicesWizardController.java line 405
 in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController renderReviewStep()
9. <generated> line -1
 in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController$$FastClassByCGLIB$$71adc282 invoke()
10. MethodProxy.java line 191
 in net.sf.cglib.proxy.MethodProxy invoke()
11. Cglib2AopProxy.java line 688
 in org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation invokeJoinpoint()
12. ReflectiveMethodInvocation.java line 150
 in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
13. MethodSecurityInterceptor.java line 61
 in org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor invoke()
14. ReflectiveMethodInvocation.java line 172
 in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
15. Cglib2AopProxy.java line 621
 in org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor intercept()
16. <generated> line -1
 in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController$$EnhancerByCGLIB$$5048f4b renderReviewStep()
17. NativeMethodAccessorImpl.java line -2
 in sun.reflect.NativeMethodAccessorImpl invoke0()
18. NativeMethodAccessorImpl.java line 57
 in sun.reflect.NativeMethodAccessorImpl invoke()
19. DelegatingMethodAccessorImpl.java line 43
 in sun.reflect.DelegatingMethodAccessorImpl invoke()
20. Method.java line 606
 in java.lang.reflect.Method invoke()
21. HandlerMethodInvoker.java line 176
 in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
22. AnnotationMethodHandlerAdapter.java line 436
 in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
23. AnnotationMethodHandlerAdapter.java line 424
 in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
24. DispatcherServlet.java line 790
 in org.springframework.web.servlet.DispatcherServlet doDispatch()
25. DispatcherServlet.java line 719
 in org.springframework.web.servlet.DispatcherServlet doService()
26. FrameworkServlet.java line 669
 in org.springframework.web.servlet.FrameworkServlet processRequest()
27. FrameworkServlet.java line 574
 in org.springframework.web.servlet.FrameworkServlet doGet()
28. HttpServlet.java line 707
 in javax.servlet.http.HttpServlet service()
29. HttpServlet.java line 820
 in javax.servlet.http.HttpServlet service()
30. ServletHolder.java line 511
 in org.mortbay.jetty.servlet.ServletHolder handle()
31. ServletHandler.java line 1221
 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
32. UserAgentFilter.java line 78
 in org.mortbay.servlet.UserAgentFilter doFilter()
33. GzipFilter.java line 131
 in org.mortbay.servlet.GzipFilter doFilter()
34. ServletHandler.java line 1212
 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
35. JAMonServletFilter.java line 48
 in com.jamonapi.http.JAMonServletFilter doFilter()
36. ServletHandler.java line 1212
 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
37. JavaMelodyFacade.java line 109
 in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
38. ServletHandler.java line 1212
 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
39. FilterChainProxy.java line 311
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
40. FilterSecurityInterceptor.java line 116
 in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
41. FilterSecurityInterceptor.java line 83
 in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
42. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
43. ExceptionTranslationFilter.java line 113
 in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
44. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
45. SessionManagementFilter.java line 101
 in org.springframework.security.web.session.SessionManagementFilter doFilter()
46. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
47. AnonymousAuthenticationFilter.java line 113
 in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
48. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
49. RememberMeAuthenticationFilter.java line 146
 in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
50. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
51. SecurityContextHolderAwareRequestFilter.java line 54
 in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
52. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
53. RequestCacheAwareFilter.java line 45
 in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
54. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
55. AbstractAuthenticationProcessingFilter.java line 182
 in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
56. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
57. LogoutFilter.java line 105
 in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
58. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
59. SecurityContextPersistenceFilter.java line 87
 in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
60. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
61. ConcurrentSessionFilter.java line 125
 in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
62. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
63. FilterChainProxy.java line 173
 in org.springframework.security.web.FilterChainProxy doFilter()
64. DelegatingFilterProxy.java line 237
 in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
65. DelegatingFilterProxy.java line 167
 in org.springframework.web.filter.DelegatingFilterProxy doFilter()
66. ServletHandler.java line 1212
 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
67. CharacterEncodingFilter.java line 88
 in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
68. OncePerRequestFilter.java line 76
 in org.springframework.web.filter.OncePerRequestFilter doFilter()
69. ServletHandler.java line 1212
 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
70. ServletHandler.java line 399
 in org.mortbay.jetty.servlet.ServletHandler handle()
71. SecurityHandler.java line 216
 in org.mortbay.jetty.security.SecurityHandler handle()
72. SessionHandler.java line 182
 in org.mortbay.jetty.servlet.SessionHandler handle()
73. SecurityHandler.java line 216
 in org.mortbay.jetty.security.SecurityHandler handle()
74. ContextHandler.java line 766
 in org.mortbay.jetty.handler.ContextHandler handle()
75. WebAppContext.java line 450
 in org.mortbay.jetty.webapp.WebAppContext handle()
76. HandlerWrapper.java line 152
 in org.mortbay.jetty.handler.HandlerWrapper handle()
77. StatisticsHandler.java line 53
 in org.mortbay.jetty.handler.StatisticsHandler handle()
78. HandlerWrapper.java line 152
 in org.mortbay.jetty.handler.HandlerWrapper handle()
79. Server.java line 326
 in org.mortbay.jetty.Server handle()
80. HttpConnection.java line 542
 in org.mortbay.jetty.HttpConnection handleRequest()
81. HttpConnection.java line 928
 in org.mortbay.jetty.HttpConnection$RequestHandler headerComplete()
82. HttpParser.java line 549
 in org.mortbay.jetty.HttpParser parseNext()
83. HttpParser.java line 212
 in org.mortbay.jetty.HttpParser parseAvailable()
84. HttpConnection.java line 404
 in org.mortbay.jetty.HttpConnection handle()
85. SelectChannelEndPoint.java line 410
 in org.mortbay.io.nio.SelectChannelEndPoint run()
86. QueuedThreadPool.java line 582
 in org.mortbay.thread.QueuedThreadPool$PoolThread run()

[#2]

Can someone tell me what error code 544 is, how to resolve it and retry?  Is there a manual for error codes?

[#3]

What is command 544?  Is there a list of commands and pre-requisites for successful completion of same?

[#4]

Since you re-created a cluster on the same hosts, HDFS was already formatted from your previous run. Your overall workflow should have succeeded and you can ignore this error. If all your services started, then you're good to go.

544 is the command ID in the CM database, not an error code. It's basically for debugging.

[#5]

That's true except that Hive never got to recreate it's database, that's a later step that was not completed because of this error.  If we could skip the error and move through the resto of the procesess, (18 more beyond the HDFS format component), then that would be great...

With the current unprocessed initial setup, the hive user id and postresql was incomplete and now postresql is giving me an invalid user id/password which makes sense since the script never got to that point due to the above error...

[#6]

Inspect hosts for correctness
 
 Run Again 
 

Validations

  Inspector ran on all 4 hosts. 
  Individual hosts resolved their own hostnames correctly. 
  No errors were found while looking for conflicting init scripts. 
  No errors were found while checking /etc/hosts. 
  All hosts resolved localhost to 127.0.0.1. 
  All hosts checked resolved each other's hostnames correctly and in a timely manner. 
  Host clocks are approximately in sync (within ten minutes). 
  Host time zones are consistent across the cluster. 
  No users or groups are missing. 
  No kernel versions that are known to be bad are running. 
  All hosts have /proc/sys/vm/swappiness set to 0. 
  No performance concerns with Transparent Huge Pages settings. 
   0 hosts are running CDH 4 and 4 hosts are running CDH5. 
  All checked hosts in each cluster are running the same version of components. 
  All managed hosts have consistent versions of Java. 
  All checked Cloudera Management Daemons versions are consistent with the server. 
  All checked Cloudera Management Agents versions are consistent with the server. 

Version Summary

Cluster 1 — CDH 5 

Hosts

hadoop0, hadoop1, hadoop2, hadoopmngr 

Component

Version

CDH Version

Bigtop-Tomcat (CDH 5 only) 0.7.0+cdh5.0.0+0 CDH5
Crunch (CDH 5 only) 0.9.0+cdh5.0.0+19 CDH5
Flume NG 1.4.0+cdh5.0.0+90 CDH5
MapReduce 1 2.2.0+cdh5.0.0+1610 CDH5
HDFS 2.2.0+cdh5.0.0+1610 CDH5
HttpFS 2.2.0+cdh5.0.0+1610 CDH5
MapReduce 2 2.2.0+cdh5.0.0+1610 CDH5
YARN 2.2.0+cdh5.0.0+1610 CDH5
Hadoop 2.2.0+cdh5.0.0+1610 CDH5
Lily HBase Indexer 1.3+cdh5.0.0+39 CDH5
HBase 0.96.1.1+cdh5.0.0+23 CDH5
HCatalog 0.12.0+cdh5.0.0+265 CDH5
Hive 0.12.0+cdh5.0.0+265 CDH5
Hue 3.5.0+cdh5.0.0+186 CDH5
Impala 1.2.3+cdh5.0.0+0 CDH5
Kite (CDH 5 only) 0.10.0+cdh5.0.0+69 CDH5
Llama (CDH 5 only) 1.0.0+cdh5.0.0+0 CDH5
Mahout 0.8+cdh5.0.0+28 CDH5
Oozie 4.0.0+cdh5.0.0+144 CDH5
Parquet 1.2.5+cdh5.0.0+29 CDH5
Pig 0.12.0+cdh5.0.0+20 CDH5
Solr 4.4.0+cdh5.0.0+163 CDH5
spark 0.9.0 CDH5
Sqoop 1.4.4+cdh5.0.0+40 CDH5
Sqoop2 1.99.3+cdh5.0.0+19 CDH5
Whirr 0.8.2+cdh5.0.0+20 CDH5
Zookeeper 3.4.5+cdh5.0.0+27 CDH5
Cloudera Manager Management Daemons 5.0.0-beta-2 Not applicable
Java 6 java version "1.6.0_31" Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)  Not applicable
Java 7 java version "1.7.0_25" Java(TM) SE Runtime Environment (build 1.7.0_25-b15) Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)  Not applicable
Cloudera Manager Agent 5.0.0-beta-2 Not applicable
 

[#7]

 Skipped. Will create database in later step

[#8]

 Basically DIO if you see what is happening, because the format HDFS does not complete, the hive database setup step is not executed either, then when we try to start hive, we get a postgresql access denied error.

 
 Progress
 

Command

Context

Status

Started at

Ended at

  First Run     Finished  Mar 31, 2014 9:35:56 PM EDT  Mar 31, 2014 9:36:55 PM EDT 

Failed to perform First Run of services.

Command Progress

Completed 3 of 20 steps.

 
Waiting for ZooKeeper Service to initialize
Finished waiting
Details
 
 
Starting ZooKeeper Service
Completed 1/1 steps successfully
Details
 
 
Checking if the name directories of the NameNode are empty. Formatting HDFS only if empty.
Command (563) has failed
Details
 
 
Starting HDFS Service
 
 
Creating HDFS /tmp directory
 
 
Creating MR2 job history directory
 
 
Creating NodeManager remote application log directory
 
 
Starting YARN (MR2 Included) Service
 
 
Creating Hive Metastore Database
 
 
Creating Hive Metastore Database Tables
 
 
Creating Hive user directory
 
 
Creating Hive warehouse directory
 
 
Starting Hive Service
 
 
Creating Oozie database
 
 
Installing Oozie ShareLib in HDFS
 
 
Starting Oozie Service
 
 
Creating Sqoop user directory
 
 
Starting Sqoop Service
 
 
Starting Hue Service
 
 
Deploying Client Configuration

[#9]

Assuming there's nothing valuable in your NameNode, try deleting your NameNode data directories and retrying your first run. There should be a retry button on the page that says First Run. You can find the namenode data directories on the configuration page in the wizard, or by clicking on HDFS and viewing the configuration.

It may also help to see the stdout and stderr log of the NameNode format command, which you can find by clicking on HDFS, then commands.

[#10]

I'll try that and reply.  Thanks!

[#11]

okay.. progress... I've now gotten past the HDFS format by removing /dfs/nn on the manager but now I have 0 datanodes started because of the following error on each node:

10:03:06.781 PM FATAL org.apache.hadoop.hdfs.server.datanode.DataNode Initialization failed for block pool Block pool <registering> (Datanode Uuid unassigned) service to hadoopmngr/192.168.0.102:8022java.io.IOException: Incompatible clusterIDs in /dfs/dn: namenode clusterID = cluster111; datanode clusterID = cluster6