Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Error during service installation

avatar
Rising Star

First in order to test the database login connectivity during installation path A, the server name was missing, (i.e. blank:7432 instead of servername:7432) and needed to be added.  Once it oast the connection /. login id pwd verrification test, I clicked continue and received the following error:

 


Error


A server error has occurred. Send the following information to Cloudera.

Path: http://192.168.0.102:7180/cmf/clusters/1/express-add-services/review

Version: Cloudera Enterprise Data Hub Trial 5.0.0-beta-2 (#119 built by jenkins on 20140209-0301 git: 8acd3c5559ccf82bf374d49bbf00bce58dee286e) 

 
java.lang.NullPointerException:Service must not be null to generate role name
 at Preconditions.java line 208
 in com.google.common.base.Preconditions checkNotNull()


 Stack Trace:
1. Preconditions.java line 208
 in com.google.common.base.Preconditions checkNotNull()
2. DbRoleNameGenerator.java line 85
 in com.cloudera.server.cmf.DbRoleNameGenerator generate()
3. RulesCluster.java line 406
 in com.cloudera.server.cmf.cluster.RulesCluster assignRole()
4. ExpressAddServicesWizardController.java line 587
 in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController buildCustomCluster()
5. ExpressAddServicesWizardController.java line 521
 in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController buildCluster()
6. ExpressAddServicesWizardController.java line 492
 in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController buildAccs()
7. ExpressAddServicesWizardController.java line 433
 in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController handleReview()
8. ExpressAddServicesWizardController.java line 405
 in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController renderReviewStep()
9. <generated> line -1
 in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController$$FastClassByCGLIB$$71adc282 invoke()
10. MethodProxy.java line 191
 in net.sf.cglib.proxy.MethodProxy invoke()
11. Cglib2AopProxy.java line 688
 in org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation invokeJoinpoint()
12. ReflectiveMethodInvocation.java line 150
 in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
13. MethodSecurityInterceptor.java line 61
 in org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor invoke()
14. ReflectiveMethodInvocation.java line 172
 in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
15. Cglib2AopProxy.java line 621
 in org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor intercept()
16. <generated> line -1
 in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController$$EnhancerByCGLIB$$5048f4b renderReviewStep()
17. NativeMethodAccessorImpl.java line -2
 in sun.reflect.NativeMethodAccessorImpl invoke0()
18. NativeMethodAccessorImpl.java line 57
 in sun.reflect.NativeMethodAccessorImpl invoke()
19. DelegatingMethodAccessorImpl.java line 43
 in sun.reflect.DelegatingMethodAccessorImpl invoke()
20. Method.java line 606
 in java.lang.reflect.Method invoke()
21. HandlerMethodInvoker.java line 176
 in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
22. AnnotationMethodHandlerAdapter.java line 436
 in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
23. AnnotationMethodHandlerAdapter.java line 424
 in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
24. DispatcherServlet.java line 790
 in org.springframework.web.servlet.DispatcherServlet doDispatch()
25. DispatcherServlet.java line 719
 in org.springframework.web.servlet.DispatcherServlet doService()
26. FrameworkServlet.java line 669
 in org.springframework.web.servlet.FrameworkServlet processRequest()
27. FrameworkServlet.java line 574
 in org.springframework.web.servlet.FrameworkServlet doGet()
28. HttpServlet.java line 707
 in javax.servlet.http.HttpServlet service()
29. HttpServlet.java line 820
 in javax.servlet.http.HttpServlet service()
30. ServletHolder.java line 511
 in org.mortbay.jetty.servlet.ServletHolder handle()
31. ServletHandler.java line 1221
 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
32. UserAgentFilter.java line 78
 in org.mortbay.servlet.UserAgentFilter doFilter()
33. GzipFilter.java line 131
 in org.mortbay.servlet.GzipFilter doFilter()
34. ServletHandler.java line 1212
 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
35. JAMonServletFilter.java line 48
 in com.jamonapi.http.JAMonServletFilter doFilter()
36. ServletHandler.java line 1212
 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
37. JavaMelodyFacade.java line 109
 in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
38. ServletHandler.java line 1212
 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
39. FilterChainProxy.java line 311
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
40. FilterSecurityInterceptor.java line 116
 in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
41. FilterSecurityInterceptor.java line 83
 in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
42. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
43. ExceptionTranslationFilter.java line 113
 in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
44. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
45. SessionManagementFilter.java line 101
 in org.springframework.security.web.session.SessionManagementFilter doFilter()
46. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
47. AnonymousAuthenticationFilter.java line 113
 in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
48. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
49. RememberMeAuthenticationFilter.java line 146
 in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
50. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
51. SecurityContextHolderAwareRequestFilter.java line 54
 in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
52. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
53. RequestCacheAwareFilter.java line 45
 in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
54. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
55. AbstractAuthenticationProcessingFilter.java line 182
 in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
56. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
57. LogoutFilter.java line 105
 in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
58. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
59. SecurityContextPersistenceFilter.java line 87
 in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
60. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
61. ConcurrentSessionFilter.java line 125
 in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
62. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
63. FilterChainProxy.java line 173
 in org.springframework.security.web.FilterChainProxy doFilter()
64. DelegatingFilterProxy.java line 237
 in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
65. DelegatingFilterProxy.java line 167
 in org.springframework.web.filter.DelegatingFilterProxy doFilter()
66. ServletHandler.java line 1212
 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
67. CharacterEncodingFilter.java line 88
 in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
68. OncePerRequestFilter.java line 76
 in org.springframework.web.filter.OncePerRequestFilter doFilter()
69. ServletHandler.java line 1212
 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
70. ServletHandler.java line 399
 in org.mortbay.jetty.servlet.ServletHandler handle()
71. SecurityHandler.java line 216
 in org.mortbay.jetty.security.SecurityHandler handle()
72. SessionHandler.java line 182
 in org.mortbay.jetty.servlet.SessionHandler handle()
73. SecurityHandler.java line 216
 in org.mortbay.jetty.security.SecurityHandler handle()
74. ContextHandler.java line 766
 in org.mortbay.jetty.handler.ContextHandler handle()
75. WebAppContext.java line 450
 in org.mortbay.jetty.webapp.WebAppContext handle()
76. HandlerWrapper.java line 152
 in org.mortbay.jetty.handler.HandlerWrapper handle()
77. StatisticsHandler.java line 53
 in org.mortbay.jetty.handler.StatisticsHandler handle()
78. HandlerWrapper.java line 152
 in org.mortbay.jetty.handler.HandlerWrapper handle()
79. Server.java line 326
 in org.mortbay.jetty.Server handle()
80. HttpConnection.java line 542
 in org.mortbay.jetty.HttpConnection handleRequest()
81. HttpConnection.java line 928
 in org.mortbay.jetty.HttpConnection$RequestHandler headerComplete()
82. HttpParser.java line 549
 in org.mortbay.jetty.HttpParser parseNext()
83. HttpParser.java line 212
 in org.mortbay.jetty.HttpParser parseAvailable()
84. HttpConnection.java line 404
 in org.mortbay.jetty.HttpConnection handle()
85. SelectChannelEndPoint.java line 410
 in org.mortbay.io.nio.SelectChannelEndPoint run()
86. QueuedThreadPool.java line 582
 in org.mortbay.thread.QueuedThreadPool$PoolThread run()

1 ACCEPTED SOLUTION

avatar
Rising Star

retrying.. after deleting everything in /dfs by:

 

cd /dfs

rm -r -f *

 

on each host including the CDh manager server...

 

This actually worked!!! thanks Darren, Thanks DIO.. here are the steps when you need to reinstall from a botched delete of hive or zookeeper and the service add gives you a null pointer exception, (see my prior posts).

 

1)  Stop the cluster

2) delete the cluster

3) cd /dfs on each host

4) rm -r -f *

5) using the manager add the cluster

6) select the hosts tab and select alll hosts

7) choose all the defaults and click continue to start firstrun()

 

results:

 

Command Progress

Completed 20 of 20 steps.

 

Waiting for ZooKeeper Service to initialize

Finished waiting

Details

 

Starting ZooKeeper Service

Completed 1/1 steps successfully

Details

 

Checking if the name directories of the NameNode are empty. Formatting HDFS only if empty.

Sucessfully formatted NameNode.

Details

 

Starting HDFS Service

Successfully started HDFS service

Details

 

Creating HDFS /tmp directory

HDFS directory /tmp already exists.

Details

 

Creating MR2 job history directory

Successfully created MR2 job history directory.

Details

 

Creating NodeManager remote application log directory

Successfully created NodeManager remote application log directory.

Details

 

Starting YARN (MR2 Included) Service

Successfully started service

Details

 

Creating Hive Metastore Database

Created Hive Metastore Database.

Details

 

Creating Hive Metastore Database Tables

Created Hive Metastore Database Tables successfully.

Details

 

Creating Hive user directory

Successfully created Hive user directory.

Details

 

Creating Hive warehouse directory

Successfully created Hive warehouse directory.

Details

 

Starting Hive Service

Service started successfully.

Details

 

Creating Oozie database

Oozie database created successfully.

Details

 

Installing Oozie ShareLib in HDFS

Successfully installed Oozie ShareLib

Details

 

Starting Oozie Service

Service started successfully.

Details

 

Creating Sqoop user directory

Successfully created Sqoop user directory.

Details

 

Starting Sqoop Service

Service started successfully.

Details

 

Starting Hue Service

Service started successfully.

Details

 

 

View solution in original post

34 REPLIES 34

avatar
Rising Star

Can someone tell me what error code 544 is, how to resolve it and retry?  Is there a manual for error codes?

avatar
Rising Star

What is command 544?  Is there a list of commands and pre-requisites for successful completion of same?

avatar
Since you re-created a cluster on the same hosts, HDFS was already formatted from your previous run. Your overall workflow should have succeeded and you can ignore this error. If all your services started, then you're good to go.

544 is the command ID in the CM database, not an error code. It's basically for debugging.

avatar
Rising Star

That's true except that Hive never got to recreate it's database, that's a later step that was not completed because of this error.  If we could skip the error and move through the resto of the procesess, (18 more beyond the HDFS format component), then that would be great...

 

With the current unprocessed initial setup, the hive user id and postresql was incomplete and now postresql is giving me an invalid user id/password which makes sense since the script never got to that point due to the above error...

avatar
Rising Star

Inspect hosts for correctness
 
 Run Again 
 


 

Validations

 

 

  Inspector ran on all 4 hosts. 
  Individual hosts resolved their own hostnames correctly. 
  No errors were found while looking for conflicting init scripts. 
  No errors were found while checking /etc/hosts. 
  All hosts resolved localhost to 127.0.0.1. 
  All hosts checked resolved each other's hostnames correctly and in a timely manner. 
  Host clocks are approximately in sync (within ten minutes). 
  Host time zones are consistent across the cluster. 
  No users or groups are missing. 
  No kernel versions that are known to be bad are running. 
  All hosts have /proc/sys/vm/swappiness set to 0. 
  No performance concerns with Transparent Huge Pages settings. 
   0 hosts are running CDH 4 and 4 hosts are running CDH5. 
  All checked hosts in each cluster are running the same version of components. 
  All managed hosts have consistent versions of Java. 
  All checked Cloudera Management Daemons versions are consistent with the server. 
  All checked Cloudera Management Agents versions are consistent with the server. 

Version Summary

 

 

Cluster 1 — CDH 5 

Hosts

hadoop0, hadoop1, hadoop2, hadoopmngr 

Component

Version

CDH Version

Bigtop-Tomcat (CDH 5 only) 0.7.0+cdh5.0.0+0 CDH5
Crunch (CDH 5 only) 0.9.0+cdh5.0.0+19 CDH5
Flume NG 1.4.0+cdh5.0.0+90 CDH5
MapReduce 1 2.2.0+cdh5.0.0+1610 CDH5
HDFS 2.2.0+cdh5.0.0+1610 CDH5
HttpFS 2.2.0+cdh5.0.0+1610 CDH5
MapReduce 2 2.2.0+cdh5.0.0+1610 CDH5
YARN 2.2.0+cdh5.0.0+1610 CDH5
Hadoop 2.2.0+cdh5.0.0+1610 CDH5
Lily HBase Indexer 1.3+cdh5.0.0+39 CDH5
HBase 0.96.1.1+cdh5.0.0+23 CDH5
HCatalog 0.12.0+cdh5.0.0+265 CDH5
Hive 0.12.0+cdh5.0.0+265 CDH5
Hue 3.5.0+cdh5.0.0+186 CDH5
Impala 1.2.3+cdh5.0.0+0 CDH5
Kite (CDH 5 only) 0.10.0+cdh5.0.0+69 CDH5
Llama (CDH 5 only) 1.0.0+cdh5.0.0+0 CDH5
Mahout 0.8+cdh5.0.0+28 CDH5
Oozie 4.0.0+cdh5.0.0+144 CDH5
Parquet 1.2.5+cdh5.0.0+29 CDH5
Pig 0.12.0+cdh5.0.0+20 CDH5
Solr 4.4.0+cdh5.0.0+163 CDH5
spark 0.9.0 CDH5
Sqoop 1.4.4+cdh5.0.0+40 CDH5
Sqoop2 1.99.3+cdh5.0.0+19 CDH5
Whirr 0.8.2+cdh5.0.0+20 CDH5
Zookeeper 3.4.5+cdh5.0.0+27 CDH5
Cloudera Manager Management Daemons 5.0.0-beta-2 Not applicable
Java 6 java version "1.6.0_31" Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)  Not applicable
Java 7 java version "1.7.0_25" Java(TM) SE Runtime Environment (build 1.7.0_25-b15) Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)  Not applicable
Cloudera Manager Agent 5.0.0-beta-2 Not applicable
 

avatar
Rising Star

 Skipped. Will create database in later step

avatar
Rising Star

 Basically DIO if you see what is happening, because the format HDFS does not complete, the hive database setup step is not executed either, then when we try to start hive, we get a postgresql access denied error.

 
 Progress
 

 


Command

Context

Status

Started at

Ended at


  First Run     Finished  Mar 31, 2014 9:35:56 PM EDT  Mar 31, 2014 9:36:55 PM EDT 

Failed to perform First Run of services.

Command Progress


Completed 3 of 20 steps.

 


 
Waiting for ZooKeeper Service to initialize
Finished waiting
Details
 
 
Starting ZooKeeper Service
Completed 1/1 steps successfully
Details
 
 
Checking if the name directories of the NameNode are empty. Formatting HDFS only if empty.
Command (563) has failed
Details
 
 
Starting HDFS Service
 
 
Creating HDFS /tmp directory
 
 
Creating MR2 job history directory
 
 
Creating NodeManager remote application log directory
 
 
Starting YARN (MR2 Included) Service
 
 
Creating Hive Metastore Database
 
 
Creating Hive Metastore Database Tables
 
 
Creating Hive user directory
 
 
Creating Hive warehouse directory
 
 
Starting Hive Service
 
 
Creating Oozie database
 
 
Installing Oozie ShareLib in HDFS
 
 
Starting Oozie Service
 
 
Creating Sqoop user directory
 
 
Starting Sqoop Service
 
 
Starting Hue Service
 
 
Deploying Client Configuration

avatar
Assuming there's nothing valuable in your NameNode, try deleting your NameNode data directories and retrying your first run. There should be a retry button on the page that says First Run. You can find the namenode data directories on the configuration page in the wizard, or by clicking on HDFS and viewing the configuration.

It may also help to see the stdout and stderr log of the NameNode format command, which you can find by clicking on HDFS, then commands.

avatar
Rising Star

I'll try that and reply.  Thanks!

avatar
Rising Star

okay.. progress... I've now gotten past the HDFS format by removing /dfs/nn on the manager but now I have 0 datanodes started because of the following error on each node:

 

10:03:06.781 PM FATAL org.apache.hadoop.hdfs.server.datanode.DataNode Initialization failed for block pool Block pool <registering> (Datanode Uuid unassigned) service to hadoopmngr/192.168.0.102:8022java.io.IOException: Incompatible clusterIDs in /dfs/dn: namenode clusterID = cluster111; datanode clusterID = cluster6