I think in my case the issue was with multiple version of java. I dont remember exactly its been a while ... View more

Double check the SSH key, match it with what is in the host and what you are adding to CM ... View more

Yes, the issue was with java, re-configuring it solved the issue ... View more

  Here are all the details ... View more

Server logs: 2019-01-12 00:34:01,743 INFO 11663147@scm-web-5:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 85.23.65.155 2019-01-12 00:34:01,895 ERROR 11663147@scm-web-5:com.cloudera.csd.components.CsdLocalRepository: Failed to load CSD from /opt/cloudera/csd/SPARK2_ON_YARN-2.3.0.cloudera4.jar, Error: CSD bundle data directory or Jar file doesn't exist 2019-01-12 00:34:11,801 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=53ms, min=18ms, max=407ms. 2019-01-12 00:35:11,851 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=44ms, min=16ms, max=407ms. 2019-01-12 00:35:24,396 INFO 11663147@scm-web-5:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 85.23.65.155 2019-01-12 00:36:11,906 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=40ms, min=16ms, max=407ms. 2019-01-12 00:37:11,943 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=37ms, min=16ms, max=407ms. 2019-01-12 00:38:11,987 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=35ms, min=16ms, max=407ms. 2019-01-12 00:39:12,064 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=33ms, min=15ms, max=407ms. 2019-01-12 00:40:12,104 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=31ms, min=12ms, max=407ms. 2019-01-12 00:41:12,146 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=30ms, min=12ms, max=407ms. 2019-01-12 00:42:10,669 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Reaped total of 0 deleted commands 2019-01-12 00:42:10,686 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Found no commands older than 2017-01-11T22:42:10.670Z to reap. 2019-01-12 00:42:10,688 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeScannerService: Reaped 0 requests. 2019-01-12 00:42:10,688 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeConfiguratorService: Reaped 0 requests. 2019-01-12 00:42:11,770 INFO 691546742@agentServer-0:com.cloudera.server.common.MonitoringThreadPool: agentServer: execution stats: average=30ms, min=1ms, max=421ms. 2019-01-12 00:42:11,770 INFO 691546742@agentServer-0:com.cloudera.server.common.MonitoringThreadPool: agentServer: waiting in queue stats: average=0ms, min=0ms, max=4ms. 2019-01-12 00:42:12,194 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=29ms, min=12ms, max=407ms. 2019-01-12 00:43:12,248 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=28ms, min=12ms, max=407ms. 2019-01-12 00:44:12,277 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=27ms, min=12ms, max=407ms. 2019-01-12 00:45:12,333 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=26ms, min=12ms, max=407ms. 2019-01-12 00:46:12,371 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=26ms, min=12ms, max=407ms. 2019-01-12 00:47:12,431 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=25ms, min=12ms, max=407ms. Agent Logs:   [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Executing command ['/usr/lib/cmf/service/common/alternatives.sh', 'activate', 'flume-ng', '/usr/bin/flume-ng', 'bin/flume-ng', '10', 'False'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Executing command ['/usr/lib/cmf/service/common/alternatives.sh', 'activate', 'sqoop-help', '/usr/bin/sqoop-help', 'bin/sqoop-help', '10', 'False'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Executing command ['/usr/lib/cmf/service/common/alternatives.sh', 'activate', 'impala-shell', '/usr/bin/impala-shell', 'bin/impala-shell', '10', 'False'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Executing command ['/usr/lib/cmf/service/common/alternatives.sh', 'activate', 'spark-shell', '/usr/bin/spark-shell', 'bin/spark-shell', '10', 'False'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Executing command ['/usr/lib/cmf/service/common/alternatives.sh', 'activate', 'hadoop-httpfs-conf', '/etc/hadoop-httpfs/conf', 'etc/hadoop-httpfs/conf.empty', '10', 'True'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Executing command ['/usr/lib/cmf/service/common/alternatives.sh', 'activate', 'sentry-conf', '/etc/sentry/conf', 'etc/sentry/conf.dist', '10', 'True'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread agent INFO Active parcel list updated; recalculating component info. [12/Jan/2019 00:51:39 +0000] 5084 MainThread throttling_logger INFO Identified java component java8 with full version java version "1.8.0_191" Java(TM) SE Runtime Environment (build 1.8.0_191-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode) for requested version . [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [216-hdfs-NAMENODE] Reinstantiating process 216 [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [216-hdfs-NAMENODE] Updating process: True {} [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [216-hdfs-NAMENODE] Refreshing process files: [u'cloudera-stack-monitor.properties', u'topology.map', u'navigator.client.properties', u'dfs_all_hosts.txt'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO prepare_environment begin: {u'SPARK2': u'2.3.0.cloudera4-1.cdh5.13.3.p0.611179', u'CDH': u'5.16.1-1.cdh5.16.1.p0.3'}, [u'cdh'], [u'cdh-plugin', u'hdfs-plugin'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO The following requested parcels are not available: {} [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Obtained tags ['spark2'] for parcel SPARK2 [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Obtained tags ['cdh', 'impala', 'kudu', 'sentry', 'solr', 'spark'] for parcel CDH [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO prepare_environment end: {'CDH': '5.16.1-1.cdh5.16.1.p0.3'} [12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__ INFO Extracted 4 files and 0 dirs to /run/cloudera-scm-agent/process/216-hdfs-NAMENODE. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [216-hdfs-NAMENODE] Evaluating resource: [{u'path': u'/var/log/hadoop-hdfs/stacks', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'hdfs', u'user': u'hdfs', u'mode': 493}] [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin audit plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Audit Plugin for hdfs-NAMENODE with count 2 pipelines names ['NavigatorLog4jAppender', 'HdfsAuditLogger']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin metadata plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Metadata Plugin for hdfs-NAMENODE with count 1 pipelines names ['']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin profile plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Profile Plugin for hdfs-NAMENODE with count 1 pipelines names ['TP']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin monitor refresh. [12/Jan/2019 00:51:39 +0000] 5084 MainThread abstract_monitor INFO Refreshing GenericMonitor HDFS-NAMENODE for None [12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__ INFO New monitor: (<cmf.monitor.generic.GenericMonitor object at 0x7f5e3c202a10>,) [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Daemon refresh complete for process 216-hdfs-NAMENODE. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [212-cloudera-mgmt-SERVICEMONITOR] Reinstantiating process 212 [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [212-cloudera-mgmt-SERVICEMONITOR] Updating process: True {} [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [212-cloudera-mgmt-SERVICEMONITOR] Refreshing process files: [u'cloudera-stack-monitor.properties'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO prepare_environment begin: {u'CDH': u'5.16.1-1.cdh5.16.1.p0.3'}, [], [] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Service does not request any parcels [12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__ INFO Extracted 1 files and 0 dirs to /run/cloudera-scm-agent/process/212-cloudera-mgmt-SERVICEMONITOR. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [212-cloudera-mgmt-SERVICEMONITOR] Evaluating resource: [{u'path': u'/var/log/cloudera-scm-firehose/stacks', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'cloudera-scm', u'user': u'cloudera-scm', u'mode': 493}] [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin audit plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Audit Plugin for cloudera-mgmt-SERVICEMONITOR with count 1 pipelines names ['']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin metadata plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Metadata Plugin for cloudera-mgmt-SERVICEMONITOR with count 1 pipelines names ['']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin profile plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Profile Plugin for cloudera-mgmt-SERVICEMONITOR with count 1 pipelines names ['TP']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin monitor refresh. [12/Jan/2019 00:51:39 +0000] 5084 MainThread abstract_monitor INFO Refreshing GenericMonitor MGMT-SERVICEMONITOR for None [12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__ INFO New monitor: (<cmf.monitor.generic.GenericMonitor object at 0x7f5e3c202250>,) [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Daemon refresh complete for process 212-cloudera-mgmt-SERVICEMONITOR. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [213-zookeeper-server] Reinstantiating process 213 [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [213-zookeeper-server] Updating process: True {} [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [213-zookeeper-server] Refreshing process files: [u'cloudera-stack-monitor.properties'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO prepare_environment begin: {u'SPARK2': u'2.3.0.cloudera4-1.cdh5.13.3.p0.611179', u'CDH': u'5.16.1-1.cdh5.16.1.p0.3'}, [u'cdh'], [u'cdh-plugin', u'zookeeper-plugin'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO The following requested parcels are not available: {} [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Obtained tags ['spark2'] for parcel SPARK2 [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Obtained tags ['cdh', 'impala', 'kudu', 'sentry', 'solr', 'spark'] for parcel CDH [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO prepare_environment end: {'CDH': '5.16.1-1.cdh5.16.1.p0.3'} [12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__ INFO Extracted 1 files and 0 dirs to /run/cloudera-scm-agent/process/213-zookeeper-server. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [213-zookeeper-server] Evaluating resource: [{u'path': u'/var/log/zookeeper/stacks', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'zookeeper', u'user': u'zookeeper', u'mode': 493}] [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin audit plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Audit Plugin for zookeeper-server with count 1 pipelines names ['']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin metadata plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Metadata Plugin for zookeeper-server with count 1 pipelines names ['']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin profile plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Profile Plugin for zookeeper-server with count 1 pipelines names ['TP']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin monitor refresh. [12/Jan/2019 00:51:39 +0000] 5084 MainThread abstract_monitor INFO Refreshing GenericMonitor ZOOKEEPER-SERVER for None [12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__ INFO New monitor: (<cmf.monitor.generic.GenericMonitor object at 0x7f5e3c110550>,) [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Daemon refresh complete for process 213-zookeeper-server. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [230-hive-HIVESERVER2] Reinstantiating process 230 [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [230-hive-HIVESERVER2] Updating process: True {} [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [230-hive-HIVESERVER2] Refreshing process files: [u'cloudera-stack-monitor.properties', u'navigator.client.properties', u'navigator.lineage.client.properties', u'fair-scheduler.xml'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO prepare_environment begin: {u'SPARK2': u'2.3.0.cloudera4-1.cdh5.13.3.p0.611179', u'CDH': u'5.16.1-1.cdh5.16.1.p0.3'}, [u'cdh'], [u'cdh-plugin', u'hive-plugin'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO The following requested parcels are not available: {} [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Obtained tags ['spark2'] for parcel SPARK2 [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Obtained tags ['cdh', 'impala', 'kudu', 'sentry', 'solr', 'spark'] for parcel CDH [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO prepare_environment end: {'CDH': '5.16.1-1.cdh5.16.1.p0.3'} [12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__ INFO Extracted 4 files and 0 dirs to /run/cloudera-scm-agent/process/230-hive-HIVESERVER2. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [230-hive-HIVESERVER2] Evaluating resource: [{u'path': u'/var/log/hive/stacks', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'hive', u'user': u'hive', u'mode': 493}] [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin audit plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Audit Plugin for hive-HIVESERVER2 with count 3 pipelines names ['HiveExecHookContext', 'HiveMetaStoreEventListener', 'HiveSentryOnFailureHook']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin metadata plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Metadata Plugin for hive-HIVESERVER2 with count 1 pipelines names ['']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin profile plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Profile Plugin for hive-HIVESERVER2 with count 1 pipelines names ['TP']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin monitor refresh. [12/Jan/2019 00:51:39 +0000] 5084 MainThread abstract_monitor INFO Refreshing GenericMonitor HIVE-HIVESERVER2 for None [12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__ INFO New monitor: (<cmf.monitor.generic.GenericMonitor object at 0x7f5e3c17ff90>,) [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Daemon refresh complete for process 230-hive-HIVESERVER2. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [231-oozie-OOZIE_SERVER] Reinstantiating process 231 [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [231-oozie-OOZIE_SERVER] Updating process: True {} [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [231-oozie-OOZIE_SERVER] Refreshing process files: [u'cloudera-stack-monitor.properties'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO prepare_environment begin: {u'SPARK2': u'2.3.0.cloudera4-1.cdh5.13.3.p0.611179', u'CDH': u'5.16.1-1.cdh5.16.1.p0.3'}, [u'cdh'], [u'cdh-plugin', u'oozie-plugin'] [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO The following requested parcels are not available: {} [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Obtained tags ['spark2'] for parcel SPARK2 [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Obtained tags ['cdh', 'impala', 'kudu', 'sentry', 'solr', 'spark'] for parcel CDH [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO prepare_environment end: {'CDH': '5.16.1-1.cdh5.16.1.p0.3'} [12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__ INFO Extracted 1 files and 0 dirs to /run/cloudera-scm-agent/process/231-oozie-OOZIE_SERVER. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [231-oozie-OOZIE_SERVER] Evaluating resource: [{u'path': u'/var/log/oozie/stacks', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'oozie', u'user': u'oozie', u'mode': 493}] [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin audit plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Audit Plugin for oozie-OOZIE_SERVER with count 1 pipelines names ['']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin metadata plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Metadata Plugin for oozie-OOZIE_SERVER with count 1 pipelines names ['']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin profile plugin refresh [12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO Scheduling a refresh for Profile Plugin for oozie-OOZIE_SERVER with count 1 pipelines names ['TP']. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Begin monitor refresh. [12/Jan/2019 00:51:39 +0000] 5084 MainThread abstract_monitor INFO Refreshing GenericMonitor OOZIE-OOZIE_SERVER for None [12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__ INFO New monitor: (<cmf.monitor.generic.GenericMonitor object at 0x7f5e3e965510>,) [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO Daemon refresh complete for process 231-oozie-OOZIE_SERVER. [12/Jan/2019 00:51:39 +0000] 5084 MainThread process WARNING New process is not running: {u'refresh_files': [], u'config_generation': 0, u'auto_restart': False, u'running': False, u'required_tags': [], u'one_off': True, u'special_file_info': [], u'group': u'root', u'id': 234, u'status_links': {}, u'name': u'host-inspector', u'extra_groups': [], u'run_generation': 2, u'start_timeout_seconds': None, u'optional_tags': [], u'program': u'mgmt/mgmt.sh', u'arguments': [u'inspector', u'input.json', u'output.json', u'DEFAULT'], u'parcels': {}, u'resources': [], u'user': u'root'} [12/Jan/2019 00:51:39 +0000] 5084 MainThread agent INFO [234-host-inspector] Instantiating process [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [234-host-inspector] Updating process: False {} [12/Jan/2019 00:51:39 +0000] 5084 MainThread agent INFO Triggering supervisord update. [12/Jan/2019 00:51:39 +0000] 5084 MonitorDaemon-Scheduler __init__ INFO Monitor ready to report: ('GenericMonitor HDFS-NAMENODE for None',) [12/Jan/2019 00:51:39 +0000] 5084 MainThread process ERROR [234-host-inspector] Failed to write out saved config: [Errno 2] No such file or directory: '/run/cloudera-scm-agent/process/234-host-inspector/proc.json' Traceback (most recent call last): File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.1-py2.7.egg/cmf/process.py", line 367, in update_heartbeat f = self.agent.os_ops.open_write_file(os.path.join(self.dir, self.PROC_JSON)) File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.1-py2.7.egg/cmf/util/os_ops.py", line 134, in open_write_file fd = os.open(path, os.O_RDWR | os.O_CREAT | os.O_TRUNC, mode) OSError: [Errno 2] No such file or directory: '/run/cloudera-scm-agent/process/234-host-inspector/proc.json' [12/Jan/2019 00:51:39 +0000] 5084 MainThread process WARNING New process is not running: {u'refresh_files': [], u'config_generation': 0, u'auto_restart': False, u'running': False, u'required_tags': [u'cdh'], u'one_off': True, u'special_file_info': [], u'group': u'hdfs', u'id': 220, u'status_links': {}, u'name': u'hdfs-NAMENODE-nnRpcWait', u'extra_groups': [], u'run_generation': 2, u'start_timeout_seconds': None, u'optional_tags': [u'cdh-plugin', u'hdfs-plugin'], u'program': u'hdfs/hdfs.sh', u'arguments': [u'nnRpcWait', u'hdfs://pan0141.panoulu.net:8020'], u'parcels': {u'SPARK2': u'2.3.0.cloudera4-1.cdh5.13.3.p0.611179', u'CDH': u'5.16.1-1.cdh5.16.1.p0.3'}, u'resources': [], u'user': u'hdfs'} [12/Jan/2019 00:51:39 +0000] 5084 MainThread agent INFO [220-hdfs-NAMENODE-nnRpcWait] Instantiating process [12/Jan/2019 00:51:39 +0000] 5084 MainThread process INFO [220-hdfs-NAMENODE-nnRpcWait] Updating process: False {} [12/Jan/2019 00:51:39 +0000] 5084 MainThread agent INFO Triggering supervisord update. [12/Jan/2019 00:51:39 +0000] 5084 MonitorDaemon-Scheduler __init__ INFO Monitor ready to report: ('GenericMonitor MGMT-SERVICEMONITOR for None',) [12/Jan/2019 00:51:39 +0000] 5084 MonitorDaemon-Scheduler __init__ INFO Monitor ready to report: ('GenericMonitor HIVE-HIVESERVER2 for None',) [12/Jan/2019 00:51:39 +0000] 5084 MonitorDaemon-Scheduler __init__ INFO Monitor ready to report: ('GenericMonitor ZOOKEEPER-SERVER for None',) [12/Jan/2019 00:51:39 +0000] 5084 MonitorDaemon-Scheduler __init__ INFO Monitor ready to report: ('GenericMonitor OOZIE-OOZIE_SERVER for None',) [12/Jan/2019 00:51:39 +0000] 5084 MainThread process ERROR [220-hdfs-NAMENODE-nnRpcWait] Failed to write out saved config: [Errno 2] No such file or directory: '/run/cloudera-scm-agent/process/220-hdfs-NAMENODE-nnRpcWait/proc.json' Traceback (most recent call last): File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.1-py2.7.egg/cmf/process.py", line 367, in update_heartbeat f = self.agent.os_ops.open_write_file(os.path.join(self.dir, self.PROC_JSON)) File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.1-py2.7.egg/cmf/util/os_ops.py", line 134, in open_write_file fd = os.open(path, os.O_RDWR | os.O_CREAT | os.O_TRUNC, mode) OSError: [Errno 2] No such file or directory: '/run/cloudera-scm-agent/process/220-hdfs-NAMENODE-nnRpcWait/proc.json' [12/Jan/2019 00:51:39 +0000] 5084 MainThread agent WARNING Long HB processing time: 11.3919100761 [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Loading parcel manifest for: SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179 [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Loading parcel manifest for: CDH-5.16.1-1.cdh5.16.1.p0.3 [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Loading parcel manifest for: SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179 [12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel INFO Loading parcel manifest for: CDH-5.16.1-1.cdh5.16.1.p0.3 [12/Jan/2019 00:51:40 +0000] 5084 MonitorDaemon-Reporter firehoses INFO Creating a connection to the ACTIVITYMONITOR. [12/Jan/2019 00:51:40 +0000] 5084 MonitorDaemon-Reporter firehoses INFO Creating a connection to the SERVICEMONITOR. [12/Jan/2019 00:51:40 +0000] 5084 MonitorDaemon-Reporter firehoses INFO Creating a connection to the HOSTMONITOR. [12/Jan/2019 00:51:41 +0000] 5084 MainThread agent INFO PID '5371' associated with process 'flood' with payload 'processname:flood groupname:flood from_state:RUNNING expected:0 pid:5371' exited unexpectedly [12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO Piplines updated for Audit Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO Refreshing Audit Plugin for hdfs-NAMENODE with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO Piplines updated for Profile Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO Piplines updated for Audit Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO Refreshing Audit Plugin for zookeeper-server with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO Refreshing Profile Plugin for cloudera-mgmt-SERVICEMONITOR with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO Piplines updated for Metadata Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO Refreshing Metadata Plugin for zookeeper-server with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO Piplines updated for Profile Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO Piplines updated for Metadata Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO Piplines updated for Audit Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO Refreshing Profile Plugin for zookeeper-server with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO Refreshing Metadata Plugin for cloudera-mgmt-SERVICEMONITOR with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO Refreshing Audit Plugin for hive-HIVESERVER2 with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO Piplines updated for Profile Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO Piplines updated for Metadata Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO Refreshing Metadata Plugin for oozie-OOZIE_SERVER with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO Refreshing Profile Plugin for hdfs-NAMENODE with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO Piplines updated for Metadata Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO Piplines updated for Audit Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO Piplines updated for Profile Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO Refreshing Metadata Plugin for hive-HIVESERVER2 with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO Piplines updated for Metadata Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO Refreshing Profile Plugin for oozie-OOZIE_SERVER with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO Refreshing Metadata Plugin for hdfs-NAMENODE with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO Refreshing Audit Plugin for oozie-OOZIE_SERVER with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO Piplines updated for Profile Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO Piplines updated for Audit Plugin: [] [12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO Refreshing Profile Plugin for hive-HIVESERVER2 with count 0 pipelines names []. [12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO Refreshing Audit Plugin for cloudera-mgmt-SERVICEMONITOR with count 0 pipelines names []. ... View more

  you can see here as well, i am only using SPARK2_ON_YARN-2.3.0.cloudera4 ... View more

Deleted already, i experimented the other version that may be its build specific issues. here are the current logs from scm manager ERROR 1183968426@scm-web-53:com.cloudera.csd.components.CsdLocalRepository: Failed to load CSD from /opt/cloudera/csd/SPARK2_ON_YARN-2.3.0.cloudera4.jar, Error: CSD bundle data directory or Jar file doesn't exist   I am really looking forward to you , if we can solve this issue together ... View more

  Details on permissions:   ls -ltr /opt/cloudera/csd total 20 -rw-r--r-- 1 cloudera-scm cloudera-scm 19037 Jan 11 23:04 SPARK2_ON_YARN-2.3.0.cloudera4.jar   ls -ltr /opt/cloudera/ total 16 drw-r--r-- 2 cloudera-scm cloudera-scm 4096 Jan 11 23:04 csd drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Jan 11 23:09 parcel-repo drwxr-xr-x 2 root root 4096 Jan 11 23:09 parcel-cache drwxr-xr-x 5 root root 4096 Jan 11 23:09 parcels   Version: Cloudera Express 5.16.1 (#1 built by jenkins on 20181120-1759 git: 6a13b87a6fcdf4afad6d4474a68a9434b24d6c67) Java VM Name: Java HotSpot(TM) 64-Bit Server VM Java Version: 1.8.0_191   ... View more

I have mentioned in the question as well   Version: Cloudera Express 5.16.1 (#1 built by jenkins on 20181120-1759 git: 6a13b87a6fcdf4afad6d4474a68a9434b24d6c67) Java VM Name: Java HotSpot(TM) 64-Bit Server VM Java Version: 1.8.0_191 ... View more

 Error from cloudera scm server logs, beside i have tried different version as well please help     ERROR 1704117900@scm-web-7:com.cloudera.csd.components.C                                                                                        sdLocalRepository: Failed to load CSD from /opt/cloudera/csd/SPARK2_ON_YARN-2.3.                                                                                        0.cloudera2.jar, Error: CSD bundle data directory or Jar file doesn't exist ... View more

is it because of cloudera version ? ... View more

I have followed exact steps but it still fails ... View more

di dyou solve this issue? ... View more

check this thread, answer is here http://community.cloudera.com/t5/Cloudera-Manager-Installation/Yarn-Node-Manager-unexpected-exists-occurring-after/m-p/79048#M14736 ... View more

I have posted the answer, in previous reply, can you please be specific?  check this post, i replied there.  http://community.cloudera.com/t5/Cloudera-Manager-Installation/Yarn-Node-Manager-unexpected-exists-occurring-after/m-p/79048#M14736 ... View more

were you able to solve this issue? ... View more

sorry for late reply, are you still facing this issue ? ... View more

i tried your suggestion already, but did it again and now it gets stuck here. upon using ctrl plus c it skips to executor   Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLeve l(newLevel).       strange thing is it works fine on other nodes, should i use them then ? ... View more

how did you solve this? ... View more

Alright, but this wasn't the issue. I have fixed it some how. Thanks for your quick reply ! ... View more

restrict your cluster to only whitelisted ip, use some firewall, it will be solved ... View more

are you sure that resource manager is on this host ? ip-172-31-35-169 ... View more