cloudera-scm-agent and cloudera-scm-server failed after started.

[#1]

Hi friends,

    I got stuck while installing cloudera path B using parcels 5.6 with external database(oracle). Please help me.

cloudera-scm-agent and cloudera-scm-server failed after started.

> service cloudera-scm-agent status
cloudera-scm-agent dead but pid file exists

> service cloudera-scm-server status
cloudera-scm-server dead but pid file exists

Log files below
--------------------

/var/log/cloudera-scm-server/cloudera-scm-server.log

2016-05-31 03:30:15,156 INFO main:com.cloudera.enterprise.CommonMain: Statistics not enabled, JMX will not be registered
2016-05-31 03:30:15,357 WARN main:org.hibernate.engine.jdbc.spi.SqlExceptionHelper: SQL Error: 942, SQLState: 42000
2016-05-31 03:30:15,357 ERROR main:org.hibernate.engine.jdbc.spi.SqlExceptionHelper: ORA-00942: table or view does not exist

2016-05-31 03:30:15,358 WARN main:com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean: Failed to determine prior version, assuming no upgrade: org.hibernate.exception.SQLGrammarException: could not extract ResultSet
2016-05-31 03:30:15,409 INFO main:com.cloudera.enterprise.dbutil.DbUtil: Schema version table already exists.
2016-05-31 03:30:15,411 ERROR main:com.cloudera.enterprise.dbutil.DbUtil: Schema version table SCHEMA_VERSION exists but contains no rows.
2016-05-31 03:30:15,413 INFO main:org.springframework.beans.factory.support.DefaultListableBeanFactory: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@131ef10: defining beans [commandLineConfigurationBean,entityManagerFactoryBean,com.cloudera.server.cmf.TrialState,com.cloudera.server.cmf.TrialManager,com.cloudera.cmf.crypto.LicenseLoader]; root of factory hierarchy
2016-05-31 03:30:15,414 ERROR main:com.cloudera.server.cmf.Main: Server failed.
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.cloudera.server.cmf.TrialState': Cannot resolve reference to bean 'entityManagerFactoryBean' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is java.sql.SQLException: Schema version table SCHEMA_VERSION exists but contains no rows.
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:106)
at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:616)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:148)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1003)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:907)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
at com.cloudera.server.cmf.Main.bootstrapSpringContext(Main.java:370)
at com.cloudera.server.cmf.Main.<init>(Main.java:230)
at com.cloudera.server.cmf.Main.main(Main.java:204)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is java.sql.SQLException: Schema version table SCHEMA_VERSION exists but contains no rows.
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:149)

-----------------------------------------------------------------------------------------------------------------------------------------

/var/log/cloudera-scm-agent/cloudera-scm-agent.log

[31/May/2016 03:39:15 +0000] 16637 MainThread agent ERROR Heartbeating to lab1.cloudera.net:7182 failed.

Traceback (most recent call last):
File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 1072, in send_heartbeat
self.master_port)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 469, in __init__
self.conn.connect()
File "/usr/lib64/python2.6/httplib.py", line 720, in connect
self.timeout)
File "/usr/lib64/python2.6/socket.py", line 567, in create_connection
raise error, msg
error: [Errno 111] Connection refused

-------------------------------------------------------------------------------------------------

[#2 · Accepted solution]

Hello there,

Based on this line, it appears that there was some problem or interruption during your original upgrade attempt:

2016-05-31 03:30:15,411 ERROR main:com.cloudera.enterprise.dbutil.DbUtil: Schema version table SCHEMA_VERSION exists but contains no rows.

I would recommend cleaning out the database and then try starting Cloudera Manager again since it appears that the database tables are not in a "good" state.  Cloudera Manager's SCHEMA_VERSION table should have a row if it exists, so my guess is that the first attempt to start Cloudera Manager after install failed.

The agent error is happening because it cannot communicate, via heartbeat) to Cloudera Manager.

Ben

[#3]

To follow up on my response...

When using Oracle DB, the Cloudera Manager documentation does not list one grant that may be required for the first run of Cloudera Manager to work:

GRANT EXECUTE ON SYS.DBMS_LOB TO <user>

If you have another failrue after cleaning the db up, check the exception that occurred in the Cloudera Manager log and let us know...

[#4]

Great..! That worked... I just cleaned up the database (Removed the SCHEMA_VERSION table ) and now Cloudera-scm-server  is up and runnng.

 Cloudera-scm-agent  failed because the port number used by another service. I have killed the service and restarted the agent.


Thanks a lot bgooley

[#5]

below is the scm-agent log | anyone having any clue 

===============================================================================

[root@master cloudera-scm-agent]# tail -50 cloudera-scm-agent.log
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get file descriptor count for process 12198: [Errno 2] No such file or directory: '/proc/12198/fd/'
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get process metrics 12198: no process found with pid 12198
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter __init__ ERROR Could not get role update from: GenericMonitor YARN-NODEMANAGER for yarn-NODEMANAGER-a0309459eb6bd0198de5c8d1e0b9150f
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.11.1-py2.6.egg/cmf/monitor/__init__.py", line 220, in _safe_get_role_update
return monitor.get_role_update()
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.11.1-py2.6.egg/cmf/monitor/generic/__init__.py", line 181, in get_role_update
LOG)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.11.1-py2.6.egg/cmf/monitor/proc_metrics_utils.py", line 183, in add_with_descendants_proc_metrics
metrics = _get_process_and_descendant_metrics(pid)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.11.1-py2.6.egg/cmf/monitor/proc_metrics_utils.py", line 161, in _get_process_and_descendant_metrics
process = psutil.Process(pid)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/psutil-2.1.3-py2.6-linux-x86_64.egg/psutil/__init__.py", line 296, in __init__
self._init(pid)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/psutil-2.1.3-py2.6-linux-x86_64.egg/psutil/__init__.py", line 331, in _init
raise NoSuchProcess(pid, None, msg)
NoSuchProcess: no process found with pid 12198
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to read file descriptor max for process 12286: [Errno 2] No such file or directory: '/proc/12286/limits'
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get file descriptor count for process 12286: [Errno 2] No such file or directory: '/proc/12286/fd/'
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get process metrics 12286: no process found with pid 12286
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to read file descriptor max for process 13090: [Errno 2] No such file or directory: '/proc/13090/limits'
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get file descriptor count for process 13090: [Errno 2] No such file or directory: '/proc/13090/fd/'
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get process metrics 13090: no process found with pid 13090
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to read file descriptor max for process 12429: [Errno 2] No such file or directory: '/proc/12429/limits'
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get file descriptor count for process 12429: [Errno 2] No such file or directory: '/proc/12429/fd/'
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get process metrics 12429: no process found with pid 12429
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to read file descriptor max for process 13217: [Errno 2] No such file or directory: '/proc/13217/limits'
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get file descriptor count for process 13217: [Errno 2] No such file or directory: '/proc/13217/fd/'
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get process metrics 13217: no process found with pid 13217
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to read file descriptor max for process 12149: [Errno 2] No such file or directory: '/proc/12149/limits'
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get file descriptor count for process 12149: [Errno 2] No such file or directory: '/proc/12149/fd/'
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get process metrics 12149: no process found with pid 12149
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to read file descriptor max for process 12664: [Errno 2] No such file or directory: '/proc/12664/limits'
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get file descriptor count for process 12664: [Errno 2] No such file or directory: '/proc/12664/fd/'
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get process metrics 12664: no process found with pid 12664
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to read file descriptor max for process 12129: [Errno 2] No such file or directory: '/proc/12129/limits'
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get file descriptor count for process 12129: [Errno 2] No such file or directory: '/proc/12129/fd/'
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get process metrics 12129: no process found with pid 12129
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to read file descriptor max for process 12332: [Errno 2] No such file or directory: '/proc/12332/limits'
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get file descriptor count for process 12332: [Errno 2] No such file or directory: '/proc/12332/fd/'
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get process metrics 12332: no process found with pid 12332
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to read file descriptor max for process 12821: [Errno 2] No such file or directory: '/proc/12821/limits'
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get file descriptor count for process 12821: [Errno 2] No such file or directory: '/proc/12821/fd/'
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get process metrics 12821: no process found with pid 12821
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to read file descriptor max for process 12737: [Errno 2] No such file or directory: '/proc/12737/limits'
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get file descriptor count for process 12737: [Errno 2] No such file or directory: '/proc/12737/fd/'
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get process metrics 12737: no process found with pid 12737
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to read file descriptor max for process 13339: [Errno 2] No such file or directory: '/proc/13339/limits'
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get file descriptor count for process 13339: [Errno 2] No such file or directory: '/proc/13339/fd/'
[13/Sep/2017 16:54:23 +0000] 11497 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get process metrics 13339: no process found with pid 13339