Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Error while starting Cloudera Manager after reinstall

Error while starting Cloudera Manager after reinstall

New Contributor

Hello everyone, I hope you can help me with this. I tried to install CM but after a few turning points at the User Credentials for the cluster stage it just halted. So I killed all the processes, and because I didn't know how to start the server again I simply unistalled everything via the uninstall script in the /usr/share/cmf directory. The problem is that after reinstalling, the server won't start at all, so I can't access to the Cloudera Manager to finish my reinstall.

 

After searching my logs, the error happnes right after the following lines

 

2015-02-13 11:48:17,422  INFO [main:core.QuartzScheduler@238] Quartz Scheduler v.2.0.2 created.
2015-02-13 11:48:17,424  INFO [main:simpl.RAMJobStore@154] RAMJobStore initialized.
2015-02-13 11:48:17,427  INFO [main:core.QuartzScheduler@259] Scheduler meta-data: Quartz Scheduler (v2.0.2) 'com.cloudera.cmf.scheduler-1' with instanceId 'NON_CLUSTERED'
  Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
  NOT STARTED.
  Currently in standby mode.
  Number of jobs executed: 0
  Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 1 threads.
  Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered.

2015-02-13 11:48:17,428  INFO [main:impl.StdSchedulerFactory@1288] Quartz scheduler 'com.cloudera.cmf.scheduler-1' initialized from an externally provided properties instance.
2015-02-13 11:48:17,428  INFO [main:impl.StdSchedulerFactory@1292] Quartz scheduler version: 2.0.2
2015-02-13 11:48:17,481  INFO [main:scheduler.CmfScheduler@301] Added command schedule '1' to the scheduler
2015-02-13 11:48:17,521  INFO [main:cdhclient.CdhExecutorFactory@110] Adding options to bad options list: 
2015-02-13 11:48:19,798  INFO [main:components.PythonInterpreterFactory@38] Constructing Python interpreter
2015-02-13 11:48:22,398  INFO [main:components.PythonInterpreterFactory@45] Finished constructing Python interpreter
2015-02-13 11:48:22,420  INFO [main:components.JythonObjectFactoryImpl@53] Constructing Jython object factory
2015-02-13 11:48:28,670  INFO [MainThread:components.JythonObjectFactoryImpl@66] Finished constructing Jython object factory

 In my error log shows the following:

JAVA_HOME=/usr/lib/jvm/java-7-oracle
java.lang.OutOfMemoryError: PermGen space
Dumping heap to java_pid26146.hprof ...
Unable to create java_pid26146.hprof: Permission denied
#
# java.lang.OutOfMemoryError: PermGen space
# -XX:OnOutOfMemoryError="kill -9 %p"
#   Executing /bin/sh -c "kill -9 26146"...

 I've increased the PermGen memory by adding the following line at the begining of the file /usr/share/cmf/bin/cmf-server

CMF_OPTS = "$CMF_OPTS -XX:MaxPermSize=2048M"

And although the server wouldn't die with an OutOfMemoryError, it never budged from that point nor the http worked.

 

I've uploaded the heapdump here.

4 REPLIES 4

Re: Error while starting Cloudera Manager after reinstall

Are you starting the service using "service cloudera-scm-server
restart" or some other method? What do you see
/var/log/cloudera-scm-server/cloudera-scm-server.log ?

Regards,
Gautam Gopalakrishnan

Re: Error while starting Cloudera Manager after reinstall

New Contributor

GautamG,

 

Thanks for replying!

 

This is what happens after every restart.

fmartinez-Lenovo-G480 fmartinez # service cloudera-scm-server restart
Stopping cloudera-scm-server: /sbin/start-stop-daemon: warning: failed to kill 26704: No such process
 * cloudera-scm-server stopped
Starting cloudera-scm-server:  * cloudera-scm-server started

 I restart whenever I see I cannot access through the port. Also, my logs get very big. I have uploaded the log file here. But the last part of it is just what it says in my first post. It ends there.

 

Thanks, in advance.

Kind regards,

fmartinez

Re: Error while starting Cloudera Manager after reinstall

Strange that Cloudera Manager just dies without any error message. I see
your JAVA_HOME is set as

JAVA_HOME=/usr/lib/jvm/java-7-oracle

Are you able to use a different JVM. If you have installed Cloudera Manager
using the documented procedures, you'd have another JVM under
/usr/java/jdk1.7.0_67-cloudera which the CM start scripts will auto-detect.
You can simply "unser JAVA_HOME" and try starting Cloudera Manager

Regards,
Gautam Gopalakrishnan

Re: Error while starting Cloudera Manager after reinstall

New Contributor

I've tried to point Cloudera to that directory, but it just doesn't exists. The first time I installed CM, it installed it's own Java. But since I deleted that directory the first time to try to reinstall it, it just doesn't install it anymore.

 

I think the reason why CM dies without any message is because it's JVM that is dieing with an OutOfMemoryException.

 

Is there any hidden file that flags to make a partial installation or something like that? Any temporary file?  Or is there any command to force CM to try to make a fresh install and overwirte anything in its path?

 

Thanks!