Reply
New Contributor
Posts: 3
Registered: ‎07-13-2014

Cloudera Manager update problem

Hi,

 

We used Cloudera Manager 4 (with CDH4) for about 6 months on 8 nodes cluster. Couple days ago I try to update Cloudera Manager to version 5, but now it is not started.

Update finished well, I even login to console after update, but one day after I found that services cloudera-scm-server-db and cloudera-scm-server not started. After restart it go down again after some time. Logs are full of exceptions like:

 

Exception in thread "WebServerImpl" java.lang.NoClassDefFoundError: Could not initialize class org.opensaml.xml.Configuration
at org.opensaml.xml.XMLConfigurator.initializeObjectProviders(XMLConfigurator.java:257)
at org.opensaml.xml.XMLConfigurator.load(XMLConfigurator.java:182)
at org.opensaml.xml.XMLConfigurator.load(XMLConfigurator.java:166)
at org.opensaml.xml.XMLConfigurator.load(XMLConfigurator.java:143)
at org.opensaml.DefaultBootstrap.initializeXMLTooling(DefaultBootstrap.java:199)
at org.opensaml.DefaultBootstrap.bootstrap(DefaultBootstrap.java:91)
at org.opensaml.PaosBootstrap.bootstrap(PaosBootstrap.java:27)
at org.springframework.security.saml.SAMLBootstrap.postProcessBeanFactory(SAMLBootstrap.java:42)
at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:663)
at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:653)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:407)
at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:282)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:204)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:47)
at org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:549)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:136)
at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at com.cloudera.server.cmf.WebServerImpl.run(WebServerImpl.java:273)

 

I even try to delete embedded postgresql data to reinit database, but it doesnt help a lot. Also I try to downgrade back to Cloudera Manager 4, but it doesnt start also. I am afraid to lose our HBase data and Cloudera Search index.

Do you have any ideas how to fix it carefully?

Highlighted
Cloudera Employee
Posts: 79
Registered: ‎08-29-2013

Re: Cloudera Manager update problem

Hi Sergg,

 

As long as you secured a full, verified backup of your Cloudera Manager database before performing the upgrade, it would be possible to roll back the upgrade to CM5 back to CM4, just by:

 

- removing all CM5.x components 

- reinstalling CM4.x (exact same version as you were on pre-upgrade)

- sideline the Cloudera Manager database that had its schema upgraded to CM5.x

- put back in place the database which you backed up before the upgrade

- ensure that your /etc/cloudera-scm-server/db.properties file is correctly pointing at your Cloudera Manager database

- start Cloudera Manager

 

In your comment you say you deleted the embedded postgres database and reinit'd; I hope you took a backup before you did so? If you did, use the above rough steps to get it back in place.

 

If not, it's still not the end of the world. You can reinstall and reconfigure Cloudera Manager to sit atop an existing cluster withouth impacting any of the data in HDFS/Solr/HBase, you just have to be methodical in doing so, and ensure you re-enter the correct properties, paths and options as exist in the system which is currently running. There is another thread in this forum that talks about that process.

 

Now, the error you're getting: 

java.lang.NoClassDefFoundError: Could not initialize class org.opensaml.xml.Configuration

 

Since you have done upgrade/downgrade work, it would be good to confirm that you haven't lost some necessary files. Run

 

$ rpm -V cloudera-manager-server > server.rpm.check

$ rpm -V cloudera-manager-daemons > daemons.rpm.check

$ rpm -V cloudera-manager-agent > agent.rpm.check

 

to have 'rpm' check the contents on-disk against what files were provided in the RPM packages for Cloudera Manager. This is a simplistic check to make sure that no files got omitted/deleted that should be present.

 

--

Mark S.

New Contributor
Posts: 3
Registered: ‎07-13-2014

Re: Cloudera Manager update problem

Hi Mark,

 

Thanks for your answer.

 

I uninstall all CM5 packages, install CM 4.6.0 to management node. Web-UI was started successfully, but it cant find any other cluster nodes via wizard.

 

It even cant find management node itself. There are no errors in log, only endless node scan.

 

I already check connectivity between nodes. It could ping and access by SSH each other, FQDN dont changes and correct, SELinux disabled, firewall rules are correct (I also try to disable firewall for some time).

What can I do to fix this? Before update to CDH5 I upgrade Redhat 6.4 to version 6.5. Could it affect somehow?

New Contributor
Posts: 3
Registered: ‎07-13-2014

Re: Cloudera Manager update problem

I still trying to figure out the issue on cluster after update CM 4.6.0 to CM 5.

 

As Mark mentioned above, I try to uninstall CM5 and install CM 4.6.0 back, but no luck. Web-interface was started well, but in the wizard on "Host search" step it dont found own nodes.

 

The same situation if I install CM 4.7.0. Interesting, but our testing server (which is not a part of cluster and contains CM5 in pseudo-distributed mode) can find nodes of this cluster well.

What is it? Some connectivity problem? (but I already check hostnames, firewall, SELinux, etc.)

 

I also try to install CM 5.1 on cluster. In this case even web-interface wasnt started with exception

"Exception in thread "WebServerImpl" java.lang.NoClassDefFoundError: Could not initialize class org.opensaml.xml.Configuration"

 

I try command "rpm -V cloudera-manager-server" by Mark`s advice, the output of the command is just one line:

"S.5....T.  c /etc/cloudera-scm-server/db.properties". What does it mean? Anybody know?

 

Before all experiments I delete all information about previous CM install including PostgreSQL database.

 

Anybidy have any ideas?

Announcements