Reply
Contributor
Posts: 55
Registered: ‎06-24-2018

Re: Spark 2.3 Csd failed to load

[ Edited ]

Deleted already, i experimented the other version that may be its build specific issues.

here are the current logs from scm manager

ERROR 1183968426@scm-web-53:com.cloudera.csd.components.CsdLocalRepository: Failed to load CSD from /opt/cloudera/csd/SPARK2_ON_YARN-2.3.0.cloudera4.jar, Error: CSD bundle data directory or Jar file doesn't exist

 

I am really looking forward to you , if we can solve this issue together

Contributor
Posts: 55
Registered: ‎06-24-2018

Re: Spark 2.3 Csd failed to load

tempsnip.pngCapture.PNG

 

you can see here as well, i am only using SPARK2_ON_YARN-2.3.0.cloudera4

Cloudera Employee
Posts: 14
Registered: ‎05-31-2017

Re: Spark 2.3 Csd failed to load

Please attach the clouderaserver and cloudera scm agent logs to further look into the issue along with screenshot of CM home page.

Contributor
Posts: 55
Registered: ‎06-24-2018

Re: Spark 2.3 Csd failed to load

Server logs:

2019-01-12 00:34:01,743 INFO 11663147@scm-web-5:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 85.23.65.155
2019-01-12 00:34:01,895 ERROR 11663147@scm-web-5:com.cloudera.csd.components.CsdLocalRepository: Failed to load CSD from /opt/cloudera/csd/SPARK2_ON_YARN-2.3.0.cloudera4.jar, Error: CSD bundle data directory or Jar file doesn't exist
2019-01-12 00:34:11,801 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=53ms, min=18ms, max=407ms.
2019-01-12 00:35:11,851 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=44ms, min=16ms, max=407ms.
2019-01-12 00:35:24,396 INFO 11663147@scm-web-5:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 85.23.65.155
2019-01-12 00:36:11,906 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=40ms, min=16ms, max=407ms.
2019-01-12 00:37:11,943 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=37ms, min=16ms, max=407ms.
2019-01-12 00:38:11,987 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=35ms, min=16ms, max=407ms.
2019-01-12 00:39:12,064 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=33ms, min=15ms, max=407ms.
2019-01-12 00:40:12,104 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=31ms, min=12ms, max=407ms.
2019-01-12 00:41:12,146 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=30ms, min=12ms, max=407ms.
2019-01-12 00:42:10,669 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Reaped total of 0 deleted commands
2019-01-12 00:42:10,686 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Found no commands older than 2017-01-11T22:42:10.670Z to reap.
2019-01-12 00:42:10,688 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeScannerService: Reaped 0 requests.
2019-01-12 00:42:10,688 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeConfiguratorService: Reaped 0 requests.
2019-01-12 00:42:11,770 INFO 691546742@agentServer-0:com.cloudera.server.common.MonitoringThreadPool: agentServer: execution stats: average=30ms, min=1ms, max=421ms.
2019-01-12 00:42:11,770 INFO 691546742@agentServer-0:com.cloudera.server.common.MonitoringThreadPool: agentServer: waiting in queue stats: average=0ms, min=0ms, max=4ms.
2019-01-12 00:42:12,194 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=29ms, min=12ms, max=407ms.
2019-01-12 00:43:12,248 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=28ms, min=12ms, max=407ms.
2019-01-12 00:44:12,277 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=27ms, min=12ms, max=407ms.
2019-01-12 00:45:12,333 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=26ms, min=12ms, max=407ms.
2019-01-12 00:46:12,371 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=26ms, min=12ms, max=407ms.
2019-01-12 00:47:12,431 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=25ms, min=12ms, max=407ms.

Agent Logs:

 

[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Executing command ['/usr/lib/cmf/service/common/alternatives.sh', 'activate', 'flume-ng', '/usr/bin/flume-ng', 'bin/flume-ng', '10', 'False']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Executing command ['/usr/lib/cmf/service/common/alternatives.sh', 'activate', 'sqoop-help', '/usr/bin/sqoop-help', 'bin/sqoop-help', '10', 'False']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Executing command ['/usr/lib/cmf/service/common/alternatives.sh', 'activate', 'impala-shell', '/usr/bin/impala-shell', 'bin/impala-shell', '10', 'False']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Executing command ['/usr/lib/cmf/service/common/alternatives.sh', 'activate', 'spark-shell', '/usr/bin/spark-shell', 'bin/spark-shell', '10', 'False']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Executing command ['/usr/lib/cmf/service/common/alternatives.sh', 'activate', 'hadoop-httpfs-conf', '/etc/hadoop-httpfs/conf', 'etc/hadoop-httpfs/conf.empty', '10', 'True']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Executing command ['/usr/lib/cmf/service/common/alternatives.sh', 'activate', 'sentry-conf', '/etc/sentry/conf', 'etc/sentry/conf.dist', '10', 'True']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread agent        INFO     Active parcel list updated; recalculating component info.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread throttling_logger INFO     Identified java component java8 with full version java version "1.8.0_191" Java(TM) SE Runtime Environment (build 1.8.0_191-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)  for requested version .
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [216-hdfs-NAMENODE] Reinstantiating process 216
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [216-hdfs-NAMENODE] Updating process: True {}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [216-hdfs-NAMENODE] Refreshing process files: [u'cloudera-stack-monitor.properties', u'topology.map', u'navigator.client.properties', u'dfs_all_hosts.txt']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     prepare_environment begin: {u'SPARK2': u'2.3.0.cloudera4-1.cdh5.13.3.p0.611179', u'CDH': u'5.16.1-1.cdh5.16.1.p0.3'}, [u'cdh'], [u'cdh-plugin', u'hdfs-plugin']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     The following requested parcels are not available: {}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Obtained tags ['spark2'] for parcel SPARK2
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Obtained tags ['cdh', 'impala', 'kudu', 'sentry', 'solr', 'spark'] for parcel CDH
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.16.1-1.cdh5.16.1.p0.3'}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__     INFO     Extracted 4 files and 0 dirs to /run/cloudera-scm-agent/process/216-hdfs-NAMENODE.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [216-hdfs-NAMENODE] Evaluating resource: [{u'path': u'/var/log/hadoop-hdfs/stacks', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'hdfs', u'user': u'hdfs', u'mode': 493}]
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin audit plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Audit Plugin for hdfs-NAMENODE with count 2 pipelines names ['NavigatorLog4jAppender', 'HdfsAuditLogger'].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin metadata plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Metadata Plugin for hdfs-NAMENODE with count 1 pipelines names [''].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin profile plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Profile Plugin for hdfs-NAMENODE with count 1 pipelines names ['TP'].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin monitor refresh.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread abstract_monitor INFO     Refreshing GenericMonitor HDFS-NAMENODE for None
[12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__     INFO     New monitor: (<cmf.monitor.generic.GenericMonitor object at 0x7f5e3c202a10>,)
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Daemon refresh complete for process 216-hdfs-NAMENODE.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [212-cloudera-mgmt-SERVICEMONITOR] Reinstantiating process 212
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [212-cloudera-mgmt-SERVICEMONITOR] Updating process: True {}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [212-cloudera-mgmt-SERVICEMONITOR] Refreshing process files: [u'cloudera-stack-monitor.properties']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     prepare_environment begin: {u'CDH': u'5.16.1-1.cdh5.16.1.p0.3'}, [], []
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Service does not request any parcels
[12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__     INFO     Extracted 1 files and 0 dirs to /run/cloudera-scm-agent/process/212-cloudera-mgmt-SERVICEMONITOR.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [212-cloudera-mgmt-SERVICEMONITOR] Evaluating resource: [{u'path': u'/var/log/cloudera-scm-firehose/stacks', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'cloudera-scm', u'user': u'cloudera-scm', u'mode': 493}]
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin audit plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Audit Plugin for cloudera-mgmt-SERVICEMONITOR with count 1 pipelines names [''].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin metadata plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Metadata Plugin for cloudera-mgmt-SERVICEMONITOR with count 1 pipelines names [''].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin profile plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Profile Plugin for cloudera-mgmt-SERVICEMONITOR with count 1 pipelines names ['TP'].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin monitor refresh.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread abstract_monitor INFO     Refreshing GenericMonitor MGMT-SERVICEMONITOR for None
[12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__     INFO     New monitor: (<cmf.monitor.generic.GenericMonitor object at 0x7f5e3c202250>,)
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Daemon refresh complete for process 212-cloudera-mgmt-SERVICEMONITOR.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [213-zookeeper-server] Reinstantiating process 213
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [213-zookeeper-server] Updating process: True {}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [213-zookeeper-server] Refreshing process files: [u'cloudera-stack-monitor.properties']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     prepare_environment begin: {u'SPARK2': u'2.3.0.cloudera4-1.cdh5.13.3.p0.611179', u'CDH': u'5.16.1-1.cdh5.16.1.p0.3'}, [u'cdh'], [u'cdh-plugin', u'zookeeper-plugin']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     The following requested parcels are not available: {}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Obtained tags ['spark2'] for parcel SPARK2
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Obtained tags ['cdh', 'impala', 'kudu', 'sentry', 'solr', 'spark'] for parcel CDH
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.16.1-1.cdh5.16.1.p0.3'}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__     INFO     Extracted 1 files and 0 dirs to /run/cloudera-scm-agent/process/213-zookeeper-server.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [213-zookeeper-server] Evaluating resource: [{u'path': u'/var/log/zookeeper/stacks', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'zookeeper', u'user': u'zookeeper', u'mode': 493}]
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin audit plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Audit Plugin for zookeeper-server with count 1 pipelines names [''].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin metadata plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Metadata Plugin for zookeeper-server with count 1 pipelines names [''].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin profile plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Profile Plugin for zookeeper-server with count 1 pipelines names ['TP'].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin monitor refresh.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread abstract_monitor INFO     Refreshing GenericMonitor ZOOKEEPER-SERVER for None
[12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__     INFO     New monitor: (<cmf.monitor.generic.GenericMonitor object at 0x7f5e3c110550>,)
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Daemon refresh complete for process 213-zookeeper-server.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [230-hive-HIVESERVER2] Reinstantiating process 230
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [230-hive-HIVESERVER2] Updating process: True {}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [230-hive-HIVESERVER2] Refreshing process files: [u'cloudera-stack-monitor.properties', u'navigator.client.properties', u'navigator.lineage.client.properties', u'fair-scheduler.xml']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     prepare_environment begin: {u'SPARK2': u'2.3.0.cloudera4-1.cdh5.13.3.p0.611179', u'CDH': u'5.16.1-1.cdh5.16.1.p0.3'}, [u'cdh'], [u'cdh-plugin', u'hive-plugin']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     The following requested parcels are not available: {}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Obtained tags ['spark2'] for parcel SPARK2
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Obtained tags ['cdh', 'impala', 'kudu', 'sentry', 'solr', 'spark'] for parcel CDH
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.16.1-1.cdh5.16.1.p0.3'}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__     INFO     Extracted 4 files and 0 dirs to /run/cloudera-scm-agent/process/230-hive-HIVESERVER2.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [230-hive-HIVESERVER2] Evaluating resource: [{u'path': u'/var/log/hive/stacks', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'hive', u'user': u'hive', u'mode': 493}]
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin audit plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Audit Plugin for hive-HIVESERVER2 with count 3 pipelines names ['HiveExecHookContext', 'HiveMetaStoreEventListener', 'HiveSentryOnFailureHook'].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin metadata plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Metadata Plugin for hive-HIVESERVER2 with count 1 pipelines names [''].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin profile plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Profile Plugin for hive-HIVESERVER2 with count 1 pipelines names ['TP'].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin monitor refresh.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread abstract_monitor INFO     Refreshing GenericMonitor HIVE-HIVESERVER2 for None
[12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__     INFO     New monitor: (<cmf.monitor.generic.GenericMonitor object at 0x7f5e3c17ff90>,)
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Daemon refresh complete for process 230-hive-HIVESERVER2.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [231-oozie-OOZIE_SERVER] Reinstantiating process 231
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [231-oozie-OOZIE_SERVER] Updating process: True {}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [231-oozie-OOZIE_SERVER] Refreshing process files: [u'cloudera-stack-monitor.properties']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     prepare_environment begin: {u'SPARK2': u'2.3.0.cloudera4-1.cdh5.13.3.p0.611179', u'CDH': u'5.16.1-1.cdh5.16.1.p0.3'}, [u'cdh'], [u'cdh-plugin', u'oozie-plugin']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     The following requested parcels are not available: {}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Obtained tags ['spark2'] for parcel SPARK2
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Obtained tags ['cdh', 'impala', 'kudu', 'sentry', 'solr', 'spark'] for parcel CDH
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.16.1-1.cdh5.16.1.p0.3'}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__     INFO     Extracted 1 files and 0 dirs to /run/cloudera-scm-agent/process/231-oozie-OOZIE_SERVER.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [231-oozie-OOZIE_SERVER] Evaluating resource: [{u'path': u'/var/log/oozie/stacks', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'oozie', u'user': u'oozie', u'mode': 493}]
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin audit plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Audit Plugin for oozie-OOZIE_SERVER with count 1 pipelines names [''].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin metadata plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Metadata Plugin for oozie-OOZIE_SERVER with count 1 pipelines names [''].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin profile plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Profile Plugin for oozie-OOZIE_SERVER with count 1 pipelines names ['TP'].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin monitor refresh.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread abstract_monitor INFO     Refreshing GenericMonitor OOZIE-OOZIE_SERVER for None
[12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__     INFO     New monitor: (<cmf.monitor.generic.GenericMonitor object at 0x7f5e3e965510>,)
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Daemon refresh complete for process 231-oozie-OOZIE_SERVER.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      WARNING  New process is not running: {u'refresh_files': [], u'config_generation': 0, u'auto_restart': False, u'running': False, u'required_tags': [], u'one_off': True, u'special_file_info': [], u'group': u'root', u'id': 234, u'status_links': {}, u'name': u'host-inspector', u'extra_groups': [], u'run_generation': 2, u'start_timeout_seconds': None, u'optional_tags': [], u'program': u'mgmt/mgmt.sh', u'arguments': [u'inspector', u'input.json', u'output.json', u'DEFAULT'], u'parcels': {}, u'resources': [], u'user': u'root'}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread agent        INFO     [234-host-inspector] Instantiating process
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [234-host-inspector] Updating process: False {}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread agent        INFO     Triggering supervisord update.
[12/Jan/2019 00:51:39 +0000] 5084 MonitorDaemon-Scheduler __init__     INFO     Monitor ready to report: ('GenericMonitor HDFS-NAMENODE for None',)
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      ERROR    [234-host-inspector] Failed to write out saved config: [Errno 2] No such file or directory: '/run/cloudera-scm-agent/process/234-host-inspector/proc.json'
Traceback (most recent call last):
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.1-py2.7.egg/cmf/process.py", line 367, in update_heartbeat
    f = self.agent.os_ops.open_write_file(os.path.join(self.dir, self.PROC_JSON))
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.1-py2.7.egg/cmf/util/os_ops.py", line 134, in open_write_file
    fd = os.open(path, os.O_RDWR | os.O_CREAT | os.O_TRUNC, mode)
OSError: [Errno 2] No such file or directory: '/run/cloudera-scm-agent/process/234-host-inspector/proc.json'
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      WARNING  New process is not running: {u'refresh_files': [], u'config_generation': 0, u'auto_restart': False, u'running': False, u'required_tags': [u'cdh'], u'one_off': True, u'special_file_info': [], u'group': u'hdfs', u'id': 220, u'status_links': {}, u'name': u'hdfs-NAMENODE-nnRpcWait', u'extra_groups': [], u'run_generation': 2, u'start_timeout_seconds': None, u'optional_tags': [u'cdh-plugin', u'hdfs-plugin'], u'program': u'hdfs/hdfs.sh', u'arguments': [u'nnRpcWait', u'hdfs://pan0141.panoulu.net:8020'], u'parcels': {u'SPARK2': u'2.3.0.cloudera4-1.cdh5.13.3.p0.611179', u'CDH': u'5.16.1-1.cdh5.16.1.p0.3'}, u'resources': [], u'user': u'hdfs'}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread agent        INFO     [220-hdfs-NAMENODE-nnRpcWait] Instantiating process
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [220-hdfs-NAMENODE-nnRpcWait] Updating process: False {}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread agent        INFO     Triggering supervisord update.
[12/Jan/2019 00:51:39 +0000] 5084 MonitorDaemon-Scheduler __init__     INFO     Monitor ready to report: ('GenericMonitor MGMT-SERVICEMONITOR for None',)
[12/Jan/2019 00:51:39 +0000] 5084 MonitorDaemon-Scheduler __init__     INFO     Monitor ready to report: ('GenericMonitor HIVE-HIVESERVER2 for None',)
[12/Jan/2019 00:51:39 +0000] 5084 MonitorDaemon-Scheduler __init__     INFO     Monitor ready to report: ('GenericMonitor ZOOKEEPER-SERVER for None',)
[12/Jan/2019 00:51:39 +0000] 5084 MonitorDaemon-Scheduler __init__     INFO     Monitor ready to report: ('GenericMonitor OOZIE-OOZIE_SERVER for None',)
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      ERROR    [220-hdfs-NAMENODE-nnRpcWait] Failed to write out saved config: [Errno 2] No such file or directory: '/run/cloudera-scm-agent/process/220-hdfs-NAMENODE-nnRpcWait/proc.json'
Traceback (most recent call last):
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.1-py2.7.egg/cmf/process.py", line 367, in update_heartbeat
    f = self.agent.os_ops.open_write_file(os.path.join(self.dir, self.PROC_JSON))
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.1-py2.7.egg/cmf/util/os_ops.py", line 134, in open_write_file
    fd = os.open(path, os.O_RDWR | os.O_CREAT | os.O_TRUNC, mode)
OSError: [Errno 2] No such file or directory: '/run/cloudera-scm-agent/process/220-hdfs-NAMENODE-nnRpcWait/proc.json'
[12/Jan/2019 00:51:39 +0000] 5084 MainThread agent        WARNING  Long HB processing time: 11.3919100761
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Loading parcel manifest for: SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Loading parcel manifest for: CDH-5.16.1-1.cdh5.16.1.p0.3
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Loading parcel manifest for: SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Loading parcel manifest for: CDH-5.16.1-1.cdh5.16.1.p0.3
[12/Jan/2019 00:51:40 +0000] 5084 MonitorDaemon-Reporter firehoses    INFO     Creating a connection to the ACTIVITYMONITOR.
[12/Jan/2019 00:51:40 +0000] 5084 MonitorDaemon-Reporter firehoses    INFO     Creating a connection to the SERVICEMONITOR.
[12/Jan/2019 00:51:40 +0000] 5084 MonitorDaemon-Reporter firehoses    INFO     Creating a connection to the HOSTMONITOR.
[12/Jan/2019 00:51:41 +0000] 5084 MainThread agent        INFO     PID '5371' associated with process 'flood' with payload 'processname:flood groupname:flood from_state:RUNNING expected:0 pid:5371' exited unexpectedly
[12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO     Piplines updated for Audit Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO     Refreshing Audit Plugin for hdfs-NAMENODE with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO     Piplines updated for Profile Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO     Piplines updated for Audit Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO     Refreshing Audit Plugin for zookeeper-server with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO     Refreshing Profile Plugin for cloudera-mgmt-SERVICEMONITOR with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO     Piplines updated for Metadata Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO     Refreshing Metadata Plugin for zookeeper-server with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO     Piplines updated for Profile Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO     Piplines updated for Metadata Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO     Piplines updated for Audit Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO     Refreshing Profile Plugin for zookeeper-server with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO     Refreshing Metadata Plugin for cloudera-mgmt-SERVICEMONITOR with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO     Refreshing Audit Plugin for hive-HIVESERVER2 with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO     Piplines updated for Profile Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO     Piplines updated for Metadata Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO     Refreshing Metadata Plugin for oozie-OOZIE_SERVER with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO     Refreshing Profile Plugin for hdfs-NAMENODE with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO     Piplines updated for Metadata Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO     Piplines updated for Audit Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO     Piplines updated for Profile Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO     Refreshing Metadata Plugin for hive-HIVESERVER2 with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO     Piplines updated for Metadata Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO     Refreshing Profile Plugin for oozie-OOZIE_SERVER with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO     Refreshing Metadata Plugin for hdfs-NAMENODE with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO     Refreshing Audit Plugin for oozie-OOZIE_SERVER with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO     Piplines updated for Profile Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO     Piplines updated for Audit Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO     Refreshing Profile Plugin for hive-HIVESERVER2 with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO     Refreshing Audit Plugin for cloudera-mgmt-SERVICEMONITOR with count 0 pipelines names [].
Contributor
Posts: 55
Registered: ‎06-24-2018

Re: Spark 2.3 Csd failed to load

x.PNG

 

Here are all the details

Contributor
Posts: 55
Registered: ‎06-24-2018

Re: Spark 2.3 Csd failed to load

I have solve the issue thanks for your help
Posts: 947
Kudos: 30
Solutions: 18
Registered: ‎05-27-2014

Re: Spark 2.3 Csd failed to load

Hi @hadoopNoob,

 

Very glad to see you solved the issue. Could you please share the solution here so others can benefit from your findings?

 

Thanks very much for your time!

Li

Li Wang, Technical Resolution Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

Contributor
Posts: 55
Registered: ‎06-24-2018

Re: Spark 2.3 Csd failed to load

Yes, the issue was with java, re-configuring it solved the issue
Announcements