Error during service installation

[#1]

First in order to test the database login connectivity during installation path A, the server name was missing, (i.e. blank:7432 instead of servername:7432) and needed to be added.  Once it oast the connection /. login id pwd verrification test, I clicked continue and received the following error:

Error

A server error has occurred. Send the following information to Cloudera.

Path: http://192.168.0.102:7180/cmf/clusters/1/express-add-services/review

Version: Cloudera Enterprise Data Hub Trial 5.0.0-beta-2 (#119 built by jenkins on 20140209-0301 git: 8acd3c5559ccf82bf374d49bbf00bce58dee286e) 

 
java.lang.NullPointerException:Service must not be null to generate role name
 at Preconditions.java line 208
 in com.google.common.base.Preconditions checkNotNull()

 Stack Trace:
1. Preconditions.java line 208
 in com.google.common.base.Preconditions checkNotNull()
2. DbRoleNameGenerator.java line 85
 in com.cloudera.server.cmf.DbRoleNameGenerator generate()
3. RulesCluster.java line 406
 in com.cloudera.server.cmf.cluster.RulesCluster assignRole()
4. ExpressAddServicesWizardController.java line 587
 in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController buildCustomCluster()
5. ExpressAddServicesWizardController.java line 521
 in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController buildCluster()
6. ExpressAddServicesWizardController.java line 492
 in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController buildAccs()
7. ExpressAddServicesWizardController.java line 433
 in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController handleReview()
8. ExpressAddServicesWizardController.java line 405
 in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController renderReviewStep()
9. <generated> line -1
 in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController$$FastClassByCGLIB$$71adc282 invoke()
10. MethodProxy.java line 191
 in net.sf.cglib.proxy.MethodProxy invoke()
11. Cglib2AopProxy.java line 688
 in org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation invokeJoinpoint()
12. ReflectiveMethodInvocation.java line 150
 in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
13. MethodSecurityInterceptor.java line 61
 in org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor invoke()
14. ReflectiveMethodInvocation.java line 172
 in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
15. Cglib2AopProxy.java line 621
 in org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor intercept()
16. <generated> line -1
 in com.cloudera.server.web.cmf.wizard.express.ExpressAddServicesWizardController$$EnhancerByCGLIB$$5048f4b renderReviewStep()
17. NativeMethodAccessorImpl.java line -2
 in sun.reflect.NativeMethodAccessorImpl invoke0()
18. NativeMethodAccessorImpl.java line 57
 in sun.reflect.NativeMethodAccessorImpl invoke()
19. DelegatingMethodAccessorImpl.java line 43
 in sun.reflect.DelegatingMethodAccessorImpl invoke()
20. Method.java line 606
 in java.lang.reflect.Method invoke()
21. HandlerMethodInvoker.java line 176
 in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
22. AnnotationMethodHandlerAdapter.java line 436
 in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
23. AnnotationMethodHandlerAdapter.java line 424
 in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
24. DispatcherServlet.java line 790
 in org.springframework.web.servlet.DispatcherServlet doDispatch()
25. DispatcherServlet.java line 719
 in org.springframework.web.servlet.DispatcherServlet doService()
26. FrameworkServlet.java line 669
 in org.springframework.web.servlet.FrameworkServlet processRequest()
27. FrameworkServlet.java line 574
 in org.springframework.web.servlet.FrameworkServlet doGet()
28. HttpServlet.java line 707
 in javax.servlet.http.HttpServlet service()
29. HttpServlet.java line 820
 in javax.servlet.http.HttpServlet service()
30. ServletHolder.java line 511
 in org.mortbay.jetty.servlet.ServletHolder handle()
31. ServletHandler.java line 1221
 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
32. UserAgentFilter.java line 78
 in org.mortbay.servlet.UserAgentFilter doFilter()
33. GzipFilter.java line 131
 in org.mortbay.servlet.GzipFilter doFilter()
34. ServletHandler.java line 1212
 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
35. JAMonServletFilter.java line 48
 in com.jamonapi.http.JAMonServletFilter doFilter()
36. ServletHandler.java line 1212
 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
37. JavaMelodyFacade.java line 109
 in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
38. ServletHandler.java line 1212
 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
39. FilterChainProxy.java line 311
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
40. FilterSecurityInterceptor.java line 116
 in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
41. FilterSecurityInterceptor.java line 83
 in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
42. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
43. ExceptionTranslationFilter.java line 113
 in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
44. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
45. SessionManagementFilter.java line 101
 in org.springframework.security.web.session.SessionManagementFilter doFilter()
46. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
47. AnonymousAuthenticationFilter.java line 113
 in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
48. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
49. RememberMeAuthenticationFilter.java line 146
 in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
50. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
51. SecurityContextHolderAwareRequestFilter.java line 54
 in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
52. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
53. RequestCacheAwareFilter.java line 45
 in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
54. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
55. AbstractAuthenticationProcessingFilter.java line 182
 in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
56. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
57. LogoutFilter.java line 105
 in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
58. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
59. SecurityContextPersistenceFilter.java line 87
 in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
60. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
61. ConcurrentSessionFilter.java line 125
 in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
62. FilterChainProxy.java line 323
 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
63. FilterChainProxy.java line 173
 in org.springframework.security.web.FilterChainProxy doFilter()
64. DelegatingFilterProxy.java line 237
 in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
65. DelegatingFilterProxy.java line 167
 in org.springframework.web.filter.DelegatingFilterProxy doFilter()
66. ServletHandler.java line 1212
 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
67. CharacterEncodingFilter.java line 88
 in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
68. OncePerRequestFilter.java line 76
 in org.springframework.web.filter.OncePerRequestFilter doFilter()
69. ServletHandler.java line 1212
 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
70. ServletHandler.java line 399
 in org.mortbay.jetty.servlet.ServletHandler handle()
71. SecurityHandler.java line 216
 in org.mortbay.jetty.security.SecurityHandler handle()
72. SessionHandler.java line 182
 in org.mortbay.jetty.servlet.SessionHandler handle()
73. SecurityHandler.java line 216
 in org.mortbay.jetty.security.SecurityHandler handle()
74. ContextHandler.java line 766
 in org.mortbay.jetty.handler.ContextHandler handle()
75. WebAppContext.java line 450
 in org.mortbay.jetty.webapp.WebAppContext handle()
76. HandlerWrapper.java line 152
 in org.mortbay.jetty.handler.HandlerWrapper handle()
77. StatisticsHandler.java line 53
 in org.mortbay.jetty.handler.StatisticsHandler handle()
78. HandlerWrapper.java line 152
 in org.mortbay.jetty.handler.HandlerWrapper handle()
79. Server.java line 326
 in org.mortbay.jetty.Server handle()
80. HttpConnection.java line 542
 in org.mortbay.jetty.HttpConnection handleRequest()
81. HttpConnection.java line 928
 in org.mortbay.jetty.HttpConnection$RequestHandler headerComplete()
82. HttpParser.java line 549
 in org.mortbay.jetty.HttpParser parseNext()
83. HttpParser.java line 212
 in org.mortbay.jetty.HttpParser parseAvailable()
84. HttpConnection.java line 404
 in org.mortbay.jetty.HttpConnection handle()
85. SelectChannelEndPoint.java line 410
 in org.mortbay.io.nio.SelectChannelEndPoint run()
86. QueuedThreadPool.java line 582
 in org.mortbay.thread.QueuedThreadPool$PoolThread run()

[#2 · Accepted solution]

retrying.. after deleting everything in /dfs by:

cd /dfs

rm -r -f *

on each host including the CDh manager server...

This actually worked!!! thanks Darren, Thanks DIO.. here are the steps when you need to reinstall from a botched delete of hive or zookeeper and the service add gives you a null pointer exception, (see my prior posts).

1)  Stop the cluster

2) delete the cluster

3) cd /dfs on each host

4) rm -r -f *

5) using the manager add the cluster

6) select the hosts tab and select alll hosts

7) choose all the defaults and click continue to start firstrun()

results:

Command Progress

Completed 20 of 20 steps.

	Waiting for ZooKeeper Service to initialize Finished waiting Details
	Starting ZooKeeper Service Completed 1/1 steps successfully Details
	Checking if the name directories of the NameNode are empty. Formatting HDFS only if empty. Sucessfully formatted NameNode. Details
	Starting HDFS Service Successfully started HDFS service Details
	Creating HDFS /tmp directory HDFS directory /tmp already exists. Details
	Creating MR2 job history directory Successfully created MR2 job history directory. Details
	Creating NodeManager remote application log directory Successfully created NodeManager remote application log directory. Details
	Starting YARN (MR2 Included) Service Successfully started service Details
	Creating Hive Metastore Database Created Hive Metastore Database. Details
	Creating Hive Metastore Database Tables Created Hive Metastore Database Tables successfully. Details
	Creating Hive user directory Successfully created Hive user directory. Details
	Creating Hive warehouse directory Successfully created Hive warehouse directory. Details
	Starting Hive Service Service started successfully. Details
	Creating Oozie database Oozie database created successfully. Details
	Installing Oozie ShareLib in HDFS Successfully installed Oozie ShareLib Details
	Starting Oozie Service Service started successfully. Details
	Creating Sqoop user directory Successfully created Sqoop user directory. Details
	Starting Sqoop Service Service started successfully. Details
	Starting Hue Service Service started successfully. Details

[#3]

Happiness:

[#4]

Glad you got it working!

[#5]

Thanks Dio, the only minor issue on hiveserver2 was that it kept trying to restart because of the following error:

org.apache.hadoop.security.AccessControlException: Permission denied: user=hive, access=WRITE, inode="/tmp":hdfs:supergroup:drwxr-xr-x

Which clearly is a permission issue quickly solved with a chmod of the hadoop file system

[root@hadoopmngr tmp]# sudo -u hive hadoop fs -ls /tmp
Found 2 items
drwxr-xr-x   - hdfs   supergroup          0 2014-04-01 15:50 /tmp/.cloudera_health_monitoring_canary_files
drwxrwxrwt   - mapred hadoop              0 2014-03-31 22:20 /tmp/logs

[root@hadoopmngr tmp]# sudo -u hdfs hadoop fs -chmod -R 777 /
[root@hadoopmngr tmp]# sudo -u hive hadoop fs -ls /tmp
Found 2 items
drwxrwxrwx   - hdfs   supergroup          0 2014-04-01 15:55 /tmp/.cloudera_health_monitoring_canary_files
drwxrwxrwt   - mapred hadoop              0 2014-03-31 22:20 /tmp/logs

And t hen the hive server2 started just fine.

Thanks again for your help.

Carl

[#6]

Setting your entire HDFS to 777 perms is not recommended. You just needed to make sure there was a 777 /tmp directory. CM normally creates this for you as part of HDFS setup, and there's a command in HDFS to create the /tmp dir if you need to run it manually.

It's not really convenient to undo this, so if you're fine with having effectively no hdfs permissions, then you can keep things as is. Otherwise you can manually re-restrict permissions to each folder appropriately. Anybody else hitting this issue should just create /tmp instead of the recursive chmod.

[#7]

You are correct.  I only noticed this after the command.  Should have restricted to the /tmp. 

I'm not concerned as this is a research environment with no restricted data.

Thanks dlo,

Carl

[#8]

The solution worked for me, and my issue was due to not completing the Cloudera Manager install in the one window. I thought that I could restart from where I left off, but had nothing but issues.

I had tried to install Hive again and also Hue, but still had issues, so i suspect that there may have been issues prior to these, and there were dependencies.

The solution appears to be a good way to clean out and rebuild a cluster, for probably multiple type issues, and it does not really take that long to complete. 

[#9]

Express Cluster Installation - Add Hosts

Express Cluster Installation - Add Services

Choose the CDH 5 services that you want to install on your cluster.

Choose a combination of services to install.
 Core Hadoop

HDFS, YARN (Includes MapReduce 2), ZooKeeper, Oozie, Hive, Hue, and Sqoop
 Core with Real-Time Delivery

HDFS, YARN (Includes MapReduce 2), ZooKeeper, Oozie, Hive, Hue, Sqoop, and HBase

 Core with Real-Time Query

HDFS, YARN (Includes MapReduce 2), ZooKeeper, Oozie, Hive, Hue, Sqoop, and Impala

 Core with Real-Time Search

HDFS, YARN (Includes MapReduce 2), ZooKeeper, Oozie, Hive, Hue, Sqoop, and Solr

 Core with Spark

HDFS, YARN (Includes MapReduce 2), ZooKeeper, Oozie, Hive, Hue, Sqoop, and Spark

 All Services

HDFS, YARN (Includes MapReduce 2), ZooKeeper, Oozie, Hive, Hue, Sqoop, HBase, Impala, Solr, Spark, and Keystore Indexer

 Custom Services

Choose your own services. Services required by chosen services will automatically be included. Note that Flume can be added after your initial cluster has been set up.

 Back

 1 2 3

[#10]

Database Setup
 

On this page you configure and test database connections. If using custom databases, create the databases first according to the Installing and Configuring an External Database section of the Installation Guide.

Use Custom DatabasesUse Embedded Database

When using the Embedded Database, passwords are auto generated. Please copy them down.

Hive
 
 
 
 

Database Host Name:   Database Type: Database Name :   Username:   Password:
hadoopmngr:7432   MySQLPostgreSQLOracle   hive1   hive1   dAPyUZE0mt 

[#11]

 Skipped. Will create database in later step