Could not start cloudera-scm-server after upgrading Cloudera Manager 4 to Cloudera Manager 5

[#1]

Hi,

After successful upgrading I tried to start cloudera-scm-server service but failed. There was a bunch of errors in cloudera-scm-server.log

ERROR [MainThread:cmf.Main@202] Server failed.
javax.persistence.PersistenceException: org.hibernate.PropertyAccessException: Exception occurred inside setter of com.cloudera.cmf.model.DbCommand.arguments
at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1387)
at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1310)
at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:273)
at com.cloudera.cmf.persist.CmfEntityManager.findCommands(CmfEntityManager.java:1499)
at com.cloudera.server.cmf.Main.rangeFindDeprecatedCommands(Main.java:786)
at com.cloudera.server.cmf.Main.deleteDeprecatedCommands(Main.java:717)
at com.cloudera.server.cmf.Main.<init>(Main.java:275)
at com.cloudera.server.cmf.Main.main(Main.java:197)
Caused by: org.hibernate.PropertyAccessException: Exception occurred inside setter of com.cloudera.cmf.model.DbCommand.arguments
at org.hibernate.property.BasicPropertyAccessor$BasicSetter.set(BasicPropertyAccessor.java:88)
at org.hibernate.tuple.entity.AbstractEntityTuplizer.setPropertyValues(AbstractEntityTuplizer.java:710)
at org.hibernate.tuple.entity.PojoEntityTuplizer.setPropertyValues(PojoEntityTuplizer.java:371)
at org.hibernate.persister.entity.AbstractEntityPersister.setPropertyValues(AbstractEntityPersister.java:4509)
at org.hibernate.engine.internal.TwoPhaseLoad.doInitializeEntity(TwoPhaseLoad.java:186)
at org.hibernate.engine.internal.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:137)
at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:1107)
at org.hibernate.loader.Loader.processResultSet(Loader.java:963)
at org.hibernate.loader.Loader.doQuery(Loader.java:910)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:341)
at org.hibernate.loader.Loader.doList(Loader.java:2522)
at org.hibernate.loader.Loader.doList(Loader.java:2508)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2338)
at org.hibernate.loader.Loader.list(Loader.java:2333)
at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:490)
at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:355)
at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:195)
at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1269)
at org.hibernate.internal.QueryImpl.list(QueryImpl.java:101)
at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:264)
... 5 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.hibernate.property.BasicPropertyAccessor$BasicSetter.set(BasicPropertyAccessor.java:65)
... 24 more
Caused by: java.lang.IllegalArgumentException: Serialized command arguments string empty for command 'AmonTestDatabaseConnection' with id '4'.
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:119)
at com.cloudera.cmf.model.DbCommand.setArguments(DbCommand.java:451)
... 29 more
2014-07-22 06:59:40,214 ERROR [ParcelUpdateService:components.ParcelDownloaderImpl@391] Unable to retrieve remote parcel repository manifest
java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused to http://myhost:18900/manifest.json
at com.ning.http.client.providers.netty.NettyResponseFuture.abort(NettyResponseFuture.java:297)
at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:104)
at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:399)
at org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:390)
at org.jboss.netty.channel.DefaultChannelFuture.setFailure(DefaultChannelFuture.java:352)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:409)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:366)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:282)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.net.ConnectException: Connection refused to http://myhost:18900/manifest.json
at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:100)
... 11 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:404)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:366)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:282)
... 3 more
2014-07-22 06:59:40,602 ERROR [ParcelUpdateService:components.ParcelDownloaderImpl@371] Failed to download manifest. Status code: 404 URI: http://archive.cloudera.com/sqoop-connectors/parcels/manifest.json
2014-07-22 07:00:35,206 INFO [CMMetricsForwarder-0:components.ClouderaManagerMetricsForwarder@172] Failed to send metrics.
java.lang.reflect.UndeclaredThrowableException
at com.sun.proxy.$Proxy79.writeMetrics(Unknown Source)
at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendWithAvro(ClouderaManagerMetricsForwarder.java:184)
at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendMetrics(ClouderaManagerMetricsForwarder.java:171)
at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.run(ClouderaManagerMetricsForwarder.java:99)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused
at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:88)
... 12 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:378)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:473)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:203)
at sun.net.www.http.HttpClient.New(HttpClient.java:290)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:995)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:931)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:849)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1090)
at org.apache.avro.ipc.HttpTransceiver.writeBuffers(HttpTransceiver.java:71)
at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:58)
at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:72)
at org.apache.avro.ipc.Requestor.request(Requestor.java:147)
at org.apache.avro.ipc.Requestor.request(Requestor.java:101)
at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:72)
... 12 more

it was trying to download the manifest from http://archive.cloudera.com/sqoop-connectors/parcels/manifest.json

However, it is actually located there http://archive.cloudera.com/sqoop-connectors/parcels/latest/manifest.json.

Could you, please, advise me if there is any way to fix it?

Thanks

[#2 · Accepted solution]

Hi,

This is a pretty strange situation. It seems like somehow you have an empty string in the COMMANDS table, column ARGUMENTS. From what we can tell in the code, this should be totally impossible, since we always reject attempts to set this to an empty string (Null is ok).

Was this table manually modified?

What kind of database are you using?

To fix the problem, the following should work:
1) stop CM (sudo service cloudera-scm-server stop)
2) back up your database (just in case!)
3) log in to the database (see db host / port / database name / username / password in /etc/cloudera-scm-server/db.properties)
4) Run the following query:
update PROCESSES set ARGUMENTS=NULL where ARGUMENTS='';
(note that '' is two single quote characters, not a double-quote character)
5) start CM (sudo service cloudera-scm-server start)

Thanks,
Darren

[#3]

Hi Darren,

Thanks a lot for your reply.
Your fix with updating COMMANDS table helped. The upgrading completed successfully.
We are using Oracle now. Have migrated from Postgres to Oracle recently. It might cause this kind of discrepancies.

I really appreciate your help.

Thanks,
Eugene

[#4]

Hi Eugene,

Glad you got it working!

That probably explains it. Oracle has special handling for empty string vs null, and it probably did something weird in your migration scripts.

Thanks,
Darren