I think in my case the issue was with multiple version of java. I dont remember exactly its been a while ... View more

Yes, the issue was with java, re-configuring it solved the issue ... View more

  Here are all the details ... View more

Server logs: 2019-01-12 00:34:01,743 INFO 11663147@scm-web-5:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 85.23.65.155 2019-01-12 00:34:01,895 ERROR 11663147@scm-web-5:com.cloudera.csd.components.CsdLocalRepository: Failed to load CSD from /opt/cloudera/csd/SPARK2_ON_YARN-2.3.0.cloudera4.jar, Error: CSD bundle data directory or Jar file doesn't exist 2019-01-12 00:34:11,801 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=53ms, min=18ms, max=407ms. 2019-01-12 00:35:11,851 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=44ms, min=16ms, max=407ms. 2019-01-12 00:35:24,396 INFO 11663147@scm-web-5:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 85.23.65.155 2019-01-12 00:36:11,906 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=40ms, min=16ms, max=407ms. 2019-01-12 00:37:11,943 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=37ms, min=16ms, max=407ms. 2019-01-12 00:38:11,987 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=35ms, min=16ms, max=407ms. 2019-01-12 00:39:12,064 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=33ms, min=15ms, max=407ms. 2019-01-12 00:40:12,104 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=31ms, min=12ms, max=407ms. 2019-01-12 00:41:12,146 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=30ms, min=12ms, max=407ms. 2019-01-12 00:42:10,669 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Reaped total of 0 deleted commands 2019-01-12 00:42:10,686 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Found no commands older than 2017-01-11T22:42:10.670Z to reap. 2019-01-12 00:42:10,688 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeScannerService: Reaped 0 requests. 2019-01-12 00:42:10,688 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeConfiguratorService: Reaped 0 requests. 2019-01-12 00:42:11,770 INFO 691546742@agentServer-0:com.cloudera.server.common.MonitoringThreadPool: agentServer: execution stats: average=30ms, min=1ms, max=421ms. 2019-01-12 00:42:11,770 INFO 691546742@agentServer-0:com.cloudera.server.common.MonitoringThreadPool: agentServer: waiting in queue stats: average=0ms, min=0ms, max=4ms. 2019-01-12 00:42:12,194 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=29ms, min=12ms, max=407ms. 2019-01-12 00:43:12,248 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=28ms, min=12ms, max=407ms. 2019-01-12 00:44:12,277 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=27ms, min=12ms, max=407ms. 2019-01-12 00:45:12,333 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=26ms, min=12ms, max=407ms. 2019-01-12 00:46:12,371 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=26ms, min=12ms, max=407ms. 2019-01-12 00:47:12,431 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=25ms, min=12ms, max=407ms. Agent Logs:   [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Executing command ['/usr/lib/cmf/service/common/alternatives.sh', 'activate', 'flume-ng', '/usr/bin/flume-ng', 'bin/flume-ng', '10', 'False'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Executing command ['/usr/lib/cmf/service/common/alternatives.sh', 'activate', 'sqoop-help', '/usr/bin/sqoop-help', 'bin/sqoop-help', '10', 'False'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Executing command ['/usr/lib/cmf/service/common/alternatives.sh', 'activate', 'impala-shell', '/usr/bin/impala-shell', 'bin/impala-shell', '10', 'False'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Executing command ['/usr/lib/cmf/service/common/alternatives.sh', 'activate', 'spark-shell', '/usr/bin/spark-shell', 'bin/spark-shell', '10', 'False'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Executing command ['/usr/lib/cmf/service/common/alternatives.sh', 'activate', 'hadoop-httpfs-conf', '/etc/hadoop-httpfs/conf', 'etc/hadoop-httpfs/conf.empty', '10', 'True'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Executing command ['/usr/lib/cmf/service/common/alternatives.sh', 'activate', 'sentry-conf', '/etc/sentry/conf', 'etc/sentry/conf.dist', '10', 'True'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread agent INFO Active parcel list updated; recalculating component info. [12/Jan/2019 00:51:39 +0000] 5084 MainThread throttling_logger INFO Identified java component java8 with full version java version "1.8.0_191" Java(TM) SE Runtime Environment (build 1.8.0_191-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode) for requested version . [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [216-hdfs-NAMENODE] Reinstantiating process 216 [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [216-hdfs-NAMENODE] Updating process: True {} [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [216-hdfs-NAMENODE] Refreshing process files: [u'cloudera-stack-monitor.properties', u'topology.map', u'navigator.client.properties', u'dfs_all_hosts.txt'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO prepare_environment begin: {u'SPARK2': u'2.3.0.cloudera4-1.cdh5.13.3.p0.611179', u'CDH': u'5.16.1-1.cdh5.16.1.p0.3'}, [u'cdh'], [u'cdh-plugin', u'hdfs-plugin'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO The following requested parcels are not available: {} [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Obtained tags ['spark2'] for parcel SPARK2 [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Obtained tags ['cdh', 'impala', 'kudu', 'sentry', 'solr', 'spark'] for parcel CDH [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO prepare_environment end: {'CDH': '5.16.1-1.cdh5.16.1.p0.3'} [12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__ INFO Extracted 4 files and 0 dirs to /run/cloudera-scm-agent/process/216-hdfs-NAMENODE. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [216-hdfs-NAMENODE] Evaluating resource: [{u'path': u'/var/log/hadoop-hdfs/stacks', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'hdfs', u'user': u'hdfs', u'mode': 493}] [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin audit plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Audit Plugin for hdfs-NAMENODE with count 2 pipelines names ['NavigatorLog4jAppender', 'HdfsAuditLogger']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin metadata plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Metadata Plugin for hdfs-NAMENODE with count 1 pipelines names ['']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin profile plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Profile Plugin for hdfs-NAMENODE with count 1 pipelines names ['TP']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin monitor refresh. [12/Jan/2019 00:51:39 +0000] 5084 MainThread abstract_monitor INFO Refreshing GenericMonitor HDFS-NAMENODE for None [12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__ INFO New monitor: (<cmf.monitor.generic.GenericMonitor object at 0x7f5e3c202a10>,) [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Daemon refresh complete for process 216-hdfs-NAMENODE. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [212-cloudera-mgmt-SERVICEMONITOR] Reinstantiating process 212 [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [212-cloudera-mgmt-SERVICEMONITOR] Updating process: True {} [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [212-cloudera-mgmt-SERVICEMONITOR] Refreshing process files: [u'cloudera-stack-monitor.properties'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO prepare_environment begin: {u'CDH': u'5.16.1-1.cdh5.16.1.p0.3'}, [], [] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Service does not request any parcels [12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__ INFO Extracted 1 files and 0 dirs to /run/cloudera-scm-agent/process/212-cloudera-mgmt-SERVICEMONITOR. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [212-cloudera-mgmt-SERVICEMONITOR] Evaluating resource: [{u'path': u'/var/log/cloudera-scm-firehose/stacks', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'cloudera-scm', u'user': u'cloudera-scm', u'mode': 493}] [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin audit plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Audit Plugin for cloudera-mgmt-SERVICEMONITOR with count 1 pipelines names ['']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin metadata plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Metadata Plugin for cloudera-mgmt-SERVICEMONITOR with count 1 pipelines names ['']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin profile plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Profile Plugin for cloudera-mgmt-SERVICEMONITOR with count 1 pipelines names ['TP']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin monitor refresh. [12/Jan/2019 00:51:39 +0000] 5084 MainThread abstract_monitor INFO Refreshing GenericMonitor MGMT-SERVICEMONITOR for None [12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__ INFO New monitor: (<cmf.monitor.generic.GenericMonitor object at 0x7f5e3c202250>,) [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Daemon refresh complete for process 212-cloudera-mgmt-SERVICEMONITOR. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [213-zookeeper-server] Reinstantiating process 213 [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [213-zookeeper-server] Updating process: True {} [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [213-zookeeper-server] Refreshing process files: [u'cloudera-stack-monitor.properties'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO prepare_environment begin: {u'SPARK2': u'2.3.0.cloudera4-1.cdh5.13.3.p0.611179', u'CDH': u'5.16.1-1.cdh5.16.1.p0.3'}, [u'cdh'], [u'cdh-plugin', u'zookeeper-plugin'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO The following requested parcels are not available: {} [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Obtained tags ['spark2'] for parcel SPARK2 [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Obtained tags ['cdh', 'impala', 'kudu', 'sentry', 'solr', 'spark'] for parcel CDH [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO prepare_environment end: {'CDH': '5.16.1-1.cdh5.16.1.p0.3'} [12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__ INFO Extracted 1 files and 0 dirs to /run/cloudera-scm-agent/process/213-zookeeper-server. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [213-zookeeper-server] Evaluating resource: [{u'path': u'/var/log/zookeeper/stacks', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'zookeeper', u'user': u'zookeeper', u'mode': 493}] [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin audit plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Audit Plugin for zookeeper-server with count 1 pipelines names ['']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin metadata plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Metadata Plugin for zookeeper-server with count 1 pipelines names ['']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin profile plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Profile Plugin for zookeeper-server with count 1 pipelines names ['TP']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin monitor refresh. [12/Jan/2019 00:51:39 +0000] 5084 MainThread abstract_monitor INFO Refreshing GenericMonitor ZOOKEEPER-SERVER for None [12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__ INFO New monitor: (<cmf.monitor.generic.GenericMonitor object at 0x7f5e3c110550>,) [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Daemon refresh complete for process 213-zookeeper-server. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [230-hive-HIVESERVER2] Reinstantiating process 230 [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [230-hive-HIVESERVER2] Updating process: True {} [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [230-hive-HIVESERVER2] Refreshing process files: [u'cloudera-stack-monitor.properties', u'navigator.client.properties', u'navigator.lineage.client.properties', u'fair-scheduler.xml'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO prepare_environment begin: {u'SPARK2': u'2.3.0.cloudera4-1.cdh5.13.3.p0.611179', u'CDH': u'5.16.1-1.cdh5.16.1.p0.3'}, [u'cdh'], [u'cdh-plugin', u'hive-plugin'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO The following requested parcels are not available: {} [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Obtained tags ['spark2'] for parcel SPARK2 [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Obtained tags ['cdh', 'impala', 'kudu', 'sentry', 'solr', 'spark'] for parcel CDH [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO prepare_environment end: {'CDH': '5.16.1-1.cdh5.16.1.p0.3'} [12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__ INFO Extracted 4 files and 0 dirs to /run/cloudera-scm-agent/process/230-hive-HIVESERVER2. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [230-hive-HIVESERVER2] Evaluating resource: [{u'path': u'/var/log/hive/stacks', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'hive', u'user': u'hive', u'mode': 493}] [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin audit plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Audit Plugin for hive-HIVESERVER2 with count 3 pipelines names ['HiveExecHookContext', 'HiveMetaStoreEventListener', 'HiveSentryOnFailureHook']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin metadata plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Metadata Plugin for hive-HIVESERVER2 with count 1 pipelines names ['']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin profile plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Profile Plugin for hive-HIVESERVER2 with count 1 pipelines names ['TP']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin monitor refresh. [12/Jan/2019 00:51:39 +0000] 5084 MainThread abstract_monitor INFO Refreshing GenericMonitor HIVE-HIVESERVER2 for None [12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__ INFO New monitor: (<cmf.monitor.generic.GenericMonitor object at 0x7f5e3c17ff90>,) [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Daemon refresh complete for process 230-hive-HIVESERVER2. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [231-oozie-OOZIE_SERVER] Reinstantiating process 231 [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [231-oozie-OOZIE_SERVER] Updating process: True {} [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [231-oozie-OOZIE_SERVER] Refreshing process files: [u'cloudera-stack-monitor.properties'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO prepare_environment begin: {u'SPARK2': u'2.3.0.cloudera4-1.cdh5.13.3.p0.611179', u'CDH': u'5.16.1-1.cdh5.16.1.p0.3'}, [u'cdh'], [u'cdh-plugin', u'oozie-plugin'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO The following requested parcels are not available: {} [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Obtained tags ['spark2'] for parcel SPARK2 [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Obtained tags ['cdh', 'impala', 'kudu', 'sentry', 'solr', 'spark'] for parcel CDH [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO prepare_environment end: {'CDH': '5.16.1-1.cdh5.16.1.p0.3'} [12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__ INFO Extracted 1 files and 0 dirs to /run/cloudera-scm-agent/process/231-oozie-OOZIE_SERVER. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [231-oozie-OOZIE_SERVER] Evaluating resource: [{u'path': u'/var/log/oozie/stacks', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'oozie', u'user': u'oozie', u'mode': 493}] [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin audit plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Audit Plugin for oozie-OOZIE_SERVER with count 1 pipelines names ['']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin metadata plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Metadata Plugin for oozie-OOZIE_SERVER with count 1 pipelines names ['']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin profile plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Profile Plugin for oozie-OOZIE_SERVER with count 1 pipelines names ['TP']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin monitor refresh. [12/Jan/2019 00:51:39 +0000] 5084 MainThread abstract_monitor INFO Refreshing GenericMonitor OOZIE-OOZIE_SERVER for None [12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__ INFO New monitor: (<cmf.monitor.generic.GenericMonitor object at 0x7f5e3e965510>,) [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Daemon refresh complete for process 231-oozie-OOZIE_SERVER. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process WARNING New process is not running: {u'refresh_files': [], u'config_generation': 0, u'auto_restart': False, u'running': False, u'required_tags': [], u'one_off': True, u'special_file_info': [], u'group': u'root', u'id': 234, u'status_links': {}, u'name': u'host-inspector', u'extra_groups': [], u'run_generation': 2, u'start_timeout_seconds': None, u'optional_tags': [], u'program': u'mgmt/mgmt.sh', u'arguments': [u'inspector', u'input.json', u'output.json', u'DEFAULT'], u'parcels': {}, u'resources': [], u'user': u'root'} [12/Jan/2019 00:51:39 +0000] 5084 MainThread agent INFO [234-host-inspector] Instantiating process [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [234-host-inspector] Updating process: False {} [12/Jan/2019 00:51:39 +0000] 5084 MainThread agent INFO Triggering supervisord update. [12/Jan/2019 00:51:39 +0000] 5084 MonitorDaemon-Scheduler __init__ INFO Monitor ready to report: ('GenericMonitor HDFS-NAMENODE for None',) [12/Jan/2019 00:51:39 +0000] 5084 MainThread process ERROR [234-host-inspector] Failed to write out saved config: [Errno 2] No such file or directory: '/run/cloudera-scm-agent/process/234-host-inspector/proc.json' Traceback (most recent call last): File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.1-py2.7.egg/cmf/process.py", line 367, in update_heartbeat f = self.agent.os_ops.open_write_file(os.path.join(self.dir, self.PROC_JSON)) File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.1-py2.7.egg/cmf/util/os_ops.py", line 134, in open_write_file fd = os.open(path, os.O_RDWR | os.O_CREAT | os.O_TRUNC, mode) OSError: [Errno 2] No such file or directory: '/run/cloudera-scm-agent/process/234-host-inspector/proc.json' [12/Jan/2019 00:51:39 +0000] 5084 MainThread process WARNING New process is not running: {u'refresh_files': [], u'config_generation': 0, u'auto_restart': False, u'running': False, u'required_tags': [u'cdh'], u'one_off': True, u'special_file_info': [], u'group': u'hdfs', u'id': 220, u'status_links': {}, u'name': u'hdfs-NAMENODE-nnRpcWait', u'extra_groups': [], u'run_generation': 2, u'start_timeout_seconds': None, u'optional_tags': [u'cdh-plugin', u'hdfs-plugin'], u'program': u'hdfs/hdfs.sh', u'arguments': [u'nnRpcWait', u'hdfs://pan0141.panoulu.net:8020'], u'parcels': {u'SPARK2': u'2.3.0.cloudera4-1.cdh5.13.3.p0.611179', u'CDH': u'5.16.1-1.cdh5.16.1.p0.3'}, u'resources': [], u'user': u'hdfs'} [12/Jan/2019 00:51:39 +0000] 5084 MainThread agent INFO [220-hdfs-NAMENODE-nnRpcWait] Instantiating process [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [220-hdfs-NAMENODE-nnRpcWait] Updating process: False {} [12/Jan/2019 00:51:39 +0000] 5084 MainThread agent INFO Triggering supervisord update. [12/Jan/2019 00:51:39 +0000] 5084 MonitorDaemon-Scheduler __init__ INFO Monitor ready to report: ('GenericMonitor MGMT-SERVICEMONITOR for None',) [12/Jan/2019 00:51:39 +0000] 5084 MonitorDaemon-Scheduler __init__ INFO Monitor ready to report: ('GenericMonitor HIVE-HIVESERVER2 for None',) [12/Jan/2019 00:51:39 +0000] 5084 MonitorDaemon-Scheduler __init__ INFO Monitor ready to report: ('GenericMonitor ZOOKEEPER-SERVER for None',) [12/Jan/2019 00:51:39 +0000] 5084 MonitorDaemon-Scheduler __init__ INFO Monitor ready to report: ('GenericMonitor OOZIE-OOZIE_SERVER for None',) [12/Jan/2019 00:51:39 +0000] 5084 MainThread process ERROR [220-hdfs-NAMENODE-nnRpcWait] Failed to write out saved config: [Errno 2] No such file or directory: '/run/cloudera-scm-agent/process/220-hdfs-NAMENODE-nnRpcWait/proc.json' Traceback (most recent call last): File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.1-py2.7.egg/cmf/process.py", line 367, in update_heartbeat f = self.agent.os_ops.open_write_file(os.path.join(self.dir, self.PROC_JSON)) File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.1-py2.7.egg/cmf/util/os_ops.py", line 134, in open_write_file fd = os.open(path, os.O_RDWR | os.O_CREAT | os.O_TRUNC, mode) OSError: [Errno 2] No such file or directory: '/run/cloudera-scm-agent/process/220-hdfs-NAMENODE-nnRpcWait/proc.json' [12/Jan/2019 00:51:39 +0000] 5084 MainThread agent WARNING Long HB processing time: 11.3919100761 [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Loading parcel manifest for: SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179 [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Loading parcel manifest for: CDH-5.16.1-1.cdh5.16.1.p0.3 [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Loading parcel manifest for: SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179 [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Loading parcel manifest for: CDH-5.16.1-1.cdh5.16.1.p0.3 [12/Jan/2019 00:51:40 +0000] 5084 MonitorDaemon-Reporter firehoses INFO Creating a connection to the ACTIVITYMONITOR. [12/Jan/2019 00:51:40 +0000] 5084 MonitorDaemon-Reporter firehoses INFO Creating a connection to the SERVICEMONITOR. [12/Jan/2019 00:51:40 +0000] 5084 MonitorDaemon-Reporter firehoses INFO Creating a connection to the HOSTMONITOR. [12/Jan/2019 00:51:41 +0000] 5084 MainThread agent INFO PID '5371' associated with process 'flood' with payload 'processname:flood groupname:flood from_state:RUNNING expected:0 pid:5371' exited unexpectedly [12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO Piplines updated for Audit Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO Refreshing Audit Plugin for hdfs-NAMENODE with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO Piplines updated for Profile Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO Piplines updated for Audit Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO Refreshing Audit Plugin for zookeeper-server with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO Refreshing Profile Plugin for cloudera-mgmt-SERVICEMONITOR with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO Piplines updated for Metadata Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO Refreshing Metadata Plugin for zookeeper-server with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO Piplines updated for Profile Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO Piplines updated for Metadata Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO Piplines updated for Audit Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO Refreshing Profile Plugin for zookeeper-server with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO Refreshing Metadata Plugin for cloudera-mgmt-SERVICEMONITOR with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO Refreshing Audit Plugin for hive-HIVESERVER2 with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO Piplines updated for Profile Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO Piplines updated for Metadata Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO Refreshing Metadata Plugin for oozie-OOZIE_SERVER with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO Refreshing Profile Plugin for hdfs-NAMENODE with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO Piplines updated for Metadata Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO Piplines updated for Audit Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO Piplines updated for Profile Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO Refreshing Metadata Plugin for hive-HIVESERVER2 with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO Piplines updated for Metadata Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO Refreshing Profile Plugin for oozie-OOZIE_SERVER with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO Refreshing Metadata Plugin for hdfs-NAMENODE with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO Refreshing Audit Plugin for oozie-OOZIE_SERVER with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO Piplines updated for Profile Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO Piplines updated for Audit Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO Refreshing Profile Plugin for hive-HIVESERVER2 with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO Refreshing Audit Plugin for cloudera-mgmt-SERVICEMONITOR with count 0 pipelines names []. ... View more

  you can see here as well, i am only using SPARK2_ON_YARN-2.3.0.cloudera4 ... View more

Deleted already, i experimented the other version that may be its build specific issues. here are the current logs from scm manager ERROR 1183968426@scm-web-53:com.cloudera.csd.components.CsdLocalRepository: Failed to load CSD from /opt/cloudera/csd/SPARK2_ON_YARN-2.3.0.cloudera4.jar, Error: CSD bundle data directory or Jar file doesn't exist   I am really looking forward to you , if we can solve this issue together ... View more

  Details on permissions:   ls -ltr /opt/cloudera/csd total 20 -rw-r--r-- 1 cloudera-scm cloudera-scm 19037 Jan 11 23:04 SPARK2_ON_YARN-2.3.0.cloudera4.jar   ls -ltr /opt/cloudera/ total 16 drw-r--r-- 2 cloudera-scm cloudera-scm 4096 Jan 11 23:04 csd drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Jan 11 23:09 parcel-repo drwxr-xr-x 2 root root 4096 Jan 11 23:09 parcel-cache drwxr-xr-x 5 root root 4096 Jan 11 23:09 parcels   Version: Cloudera Express 5.16.1 (#1 built by jenkins on 20181120-1759 git: 6a13b87a6fcdf4afad6d4474a68a9434b24d6c67) Java VM Name: Java HotSpot(TM) 64-Bit Server VM Java Version: 1.8.0_191   ... View more

I have mentioned in the question as well   Version: Cloudera Express 5.16.1 (#1 built by jenkins on 20181120-1759 git: 6a13b87a6fcdf4afad6d4474a68a9434b24d6c67) Java VM Name: Java HotSpot(TM) 64-Bit Server VM Java Version: 1.8.0_191 ... View more