question Re: Spark 2.3 Csd failed to load in Support Questions

Spark 2.3 Csd failed to load

hadoopNoob — Fri, 16 Sep 2022 14:02:19 GMT

Hello,

I am trying to install Spark 2.3.0 release 4, but after placing csd (jar file) to /opt/cloudera/csd i cannot add it from service.

I checked on this url http:xyz:7180/cmf/csd/refreshand got the following error:

placeholderHandlers":[],"invalidCsds":{"/opt/cloudera/csd/SPARK2_ON_YARN-2.3.0.cloudera4.jar":["Failed to load CSD from /opt/cloudera/csd/SPARK2_ON_YARN-2.3.0.cloudera4.jar, Error: CSD bundle data directory or Jar file doesn't exist"]},"repoEnabled":true}}

I followed these steps:

a. Download the spark 2.3.0 jar

b. Upload the CSD to /opt/cloudera/csd in the Cloudera Manager server.

c. Change the owner and group for the JAR

d. restart cloudera scm server and management service

Can anyone help ? I have CDH release 5.16.1-1.cdh5.16.1.p0.3

Re: Spark 2.3 Csd failed to load

Bharati — Tue, 08 Jan 2019 00:23:56 GMT

I think you are missing the parcel installation along with CDS download.

Whenever upgrading spark 2.x versions please follow all steps below and make sure there aren't any old version CSD in CSD folder

A) Make sure the Parcel is installed distributed and Activated. Refer Step 3 in the document below:
https://www.cloudera.com/documentation/spark2/latest/topics/spark2_installing.html

B) Make sure you download Spark 2 CDS on Cloudera Manager Server host and delete old CDS
https://www.cloudera.com/documentation/spark2/latest/topics/spark2_packaging.html#versions

C) and followed steps mentioned in the link below
https://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_addon_services.html#concept_kpt_spj_bn

- Installing the Custom Service Descriptor File

- Acquire the CSD file from Cloudera or an ISV.

- Log on to the Cloudera Manager Server host, and place the CSD file under the location configured for CSD files.

- Set the file ownership to cloudera-scm:cloudera-scm with permission 644.

- Restart the Cloudera Manager Server:

service cloudera-scm-server restart

- Log into the Cloudera Manager Admin Console and restart the Cloudera Management Service.
Do one of the following:
a)
Select Clusters > Cloudera Management Service.
Select Actions > Restart.

b)
On the Home > Status tab, click to the right of Cloudera Management Service and select Restart.

- Click Restart to confirm. The Command Details window shows the progress of stopping and then starting the roles.

- When Command completed with n/n successful subcommands appears, the task is complete. Click Close.

- Then try to Add Service for Spark 2.

Re: Spark 2.3 Csd failed to load

hadoopNoob — Fri, 11 Jan 2019 14:09:09 GMT

I have followed exact steps but it still fails

Re: Spark 2.3 Csd failed to load

hadoopNoob — Fri, 11 Jan 2019 14:17:06 GMT

is it because of cloudera version ?

Re: Spark 2.3 Csd failed to load

hadoopNoob — Fri, 11 Jan 2019 20:54:27 GMT

Error from cloudera scm server logs, beside i have tried different version as well please help

ERROR 1704117900@scm-web-7:com.cloudera.csd.components.C sdLocalRepository: Failed to load CSD from /opt/cloudera/csd/SPARK2_ON_YARN-2.3. 0.cloudera2.jar, Error: CSD bundle data directory or Jar file doesn't exist

Re: Spark 2.3 Csd failed to load

Bharati — Fri, 11 Jan 2019 21:34:04 GMT

I see that you are on CDH 5.16. Spark 2.3 is compatible with that. What is the CM version you are on?

- Can you confirm if you have installed Spark 2.3 parcel in CM > Hosts > Parcels?

- Please post the directory permission for following:

ls -ltr /opt/cloudera/

ls -ltr /opt/cloudera/csd

Re: Spark 2.3 Csd failed to load

hadoopNoob — Fri, 11 Jan 2019 21:31:04 GMT

I have mentioned in the question as well

Version: Cloudera Express 5.16.1 (#1 built by jenkins on 20181120-1759 git: 6a13b87a6fcdf4afad6d4474a68a9434b24d6c67)

Java VM Name: Java HotSpot(TM) 64-Bit Server VM

Java Version: 1.8.0_191

Re: Spark 2.3 Csd failed to load

Bharati — Fri, 11 Jan 2019 21:36:21 GMT

I see that you are on CDH 5.16. Spark 2.3 is compatible with that. What is the CM version you are on?

- Can you confirm if you have installed Spark 2.3 parcel in CM > Hosts > Parcels? Send us the screenshot of the entire page if possible.

- Also please post the directory permission for following:

ls -ltr /opt/cloudera/

ls -ltr /opt/cloudera/csd

Re: Spark 2.3 Csd failed to load

hadoopNoob — Fri, 11 Jan 2019 22:06:53 GMT

Details on permissions:

ls -ltr /opt/cloudera/csd

total 20

-rw-r--r-- 1 cloudera-scm cloudera-scm 19037 Jan 11 23:04 SPARK2_ON_YARN-2.3.0.cloudera4.jar

ls -ltr /opt/cloudera/

total 16
drw-r--r-- 2 cloudera-scm cloudera-scm 4096 Jan 11 23:04 csd
drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Jan 11 23:09 parcel-repo
drwxr-xr-x 2 root root 4096 Jan 11 23:09 parcel-cache
drwxr-xr-x 5 root root 4096 Jan 11 23:09 parcels

Version: Cloudera Express 5.16.1 (#1 built by jenkins on 20181120-1759 git: 6a13b87a6fcdf4afad6d4474a68a9434b24d6c67)

Java VM Name: Java HotSpot(TM) 64-Bit Server VM

Java Version: 1.8.0_191

Re: Spark 2.3 Csd failed to load

Bharati — Fri, 11 Jan 2019 22:13:19 GMT

1) The error that you showed talks about SPARK2_ON_YARN-2.3.0.cloudera2.jar not SPARK2_ON_YARN-2.3.0.cloudera4.jar. These are 2 different versions.

2) Do you multiple CSDs in /opt/clodera/csd or parcels activated? If that's the case please deactovate other versions and delete all old CSD. Make sure the parcel and the CSD jar version matches.

Re: Spark 2.3 Csd failed to load

hadoopNoob — Fri, 11 Jan 2019 22:16:05 GMT

Deleted already, i experimented the other version that may be its build specific issues.

here are the current logs from scm manager

ERROR 1183968426@scm-web-53:com.cloudera.csd.components.CsdLocalRepository: Failed to load CSD from /opt/cloudera/csd/SPARK2_ON_YARN-2.3.0.cloudera4.jar, Error: CSD bundle data directory or Jar file doesn't exist

I am really looking forward to you , if we can solve this issue together

Re: Spark 2.3 Csd failed to load

hadoopNoob — Fri, 11 Jan 2019 22:21:09 GMT

you can see here as well, i am only using SPARK2_ON_YARN-2.3.0.cloudera4

Re: Spark 2.3 Csd failed to load

Bharati — Fri, 11 Jan 2019 22:45:59 GMT

Please attach the clouderaserver and cloudera scm agent logs to further look into the issue along with screenshot of CM home page.

Re: Spark 2.3 Csd failed to load

hadoopNoob — Fri, 11 Jan 2019 22:53:19 GMT

Server logs:

2019-01-12 00:34:01,743 INFO 11663147@scm-web-5:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 85.23.65.155
2019-01-12 00:34:01,895 ERROR 11663147@scm-web-5:com.cloudera.csd.components.CsdLocalRepository: Failed to load CSD from /opt/cloudera/csd/SPARK2_ON_YARN-2.3.0.cloudera4.jar, Error: CSD bundle data directory or Jar file doesn't exist
2019-01-12 00:34:11,801 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=53ms, min=18ms, max=407ms.
2019-01-12 00:35:11,851 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=44ms, min=16ms, max=407ms.
2019-01-12 00:35:24,396 INFO 11663147@scm-web-5:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 85.23.65.155
2019-01-12 00:36:11,906 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=40ms, min=16ms, max=407ms.
2019-01-12 00:37:11,943 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=37ms, min=16ms, max=407ms.
2019-01-12 00:38:11,987 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=35ms, min=16ms, max=407ms.
2019-01-12 00:39:12,064 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=33ms, min=15ms, max=407ms.
2019-01-12 00:40:12,104 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=31ms, min=12ms, max=407ms.
2019-01-12 00:41:12,146 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=30ms, min=12ms, max=407ms.
2019-01-12 00:42:10,669 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Reaped total of 0 deleted commands
2019-01-12 00:42:10,686 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Found no commands older than 2017-01-11T22:42:10.670Z to reap.
2019-01-12 00:42:10,688 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeScannerService: Reaped 0 requests.
2019-01-12 00:42:10,688 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeConfiguratorService: Reaped 0 requests.
2019-01-12 00:42:11,770 INFO 691546742@agentServer-0:com.cloudera.server.common.MonitoringThreadPool: agentServer: execution stats: average=30ms, min=1ms, max=421ms.
2019-01-12 00:42:11,770 INFO 691546742@agentServer-0:com.cloudera.server.common.MonitoringThreadPool: agentServer: waiting in queue stats: average=0ms, min=0ms, max=4ms.
2019-01-12 00:42:12,194 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=29ms, min=12ms, max=407ms.
2019-01-12 00:43:12,248 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=28ms, min=12ms, max=407ms.
2019-01-12 00:44:12,277 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=27ms, min=12ms, max=407ms.
2019-01-12 00:45:12,333 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=26ms, min=12ms, max=407ms.
2019-01-12 00:46:12,371 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=26ms, min=12ms, max=407ms.
2019-01-12 00:47:12,431 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (15 skipped) AgentAvroServlet: heartbeat processing stats: average=25ms, min=12ms, max=407ms.

Agent Logs:

[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Executing command ['/usr/lib/cmf/service/common/alternatives.sh', 'activate', 'flume-ng', '/usr/bin/flume-ng', 'bin/flume-ng', '10', 'False']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Executing command ['/usr/lib/cmf/service/common/alternatives.sh', 'activate', 'sqoop-help', '/usr/bin/sqoop-help', 'bin/sqoop-help', '10', 'False']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Executing command ['/usr/lib/cmf/service/common/alternatives.sh', 'activate', 'impala-shell', '/usr/bin/impala-shell', 'bin/impala-shell', '10', 'False']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Executing command ['/usr/lib/cmf/service/common/alternatives.sh', 'activate', 'spark-shell', '/usr/bin/spark-shell', 'bin/spark-shell', '10', 'False']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Executing command ['/usr/lib/cmf/service/common/alternatives.sh', 'activate', 'hadoop-httpfs-conf', '/etc/hadoop-httpfs/conf', 'etc/hadoop-httpfs/conf.empty', '10', 'True']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Executing command ['/usr/lib/cmf/service/common/alternatives.sh', 'activate', 'sentry-conf', '/etc/sentry/conf', 'etc/sentry/conf.dist', '10', 'True']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread agent        INFO     Active parcel list updated; recalculating component info.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread throttling_logger INFO     Identified java component java8 with full version java version "1.8.0_191" Java(TM) SE Runtime Environment (build 1.8.0_191-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)  for requested version .
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [216-hdfs-NAMENODE] Reinstantiating process 216
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [216-hdfs-NAMENODE] Updating process: True {}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [216-hdfs-NAMENODE] Refreshing process files: [u'cloudera-stack-monitor.properties', u'topology.map', u'navigator.client.properties', u'dfs_all_hosts.txt']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     prepare_environment begin: {u'SPARK2': u'2.3.0.cloudera4-1.cdh5.13.3.p0.611179', u'CDH': u'5.16.1-1.cdh5.16.1.p0.3'}, [u'cdh'], [u'cdh-plugin', u'hdfs-plugin']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     The following requested parcels are not available: {}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Obtained tags ['spark2'] for parcel SPARK2
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Obtained tags ['cdh', 'impala', 'kudu', 'sentry', 'solr', 'spark'] for parcel CDH
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.16.1-1.cdh5.16.1.p0.3'}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__     INFO     Extracted 4 files and 0 dirs to /run/cloudera-scm-agent/process/216-hdfs-NAMENODE.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [216-hdfs-NAMENODE] Evaluating resource: [{u'path': u'/var/log/hadoop-hdfs/stacks', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'hdfs', u'user': u'hdfs', u'mode': 493}]
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin audit plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Audit Plugin for hdfs-NAMENODE with count 2 pipelines names ['NavigatorLog4jAppender', 'HdfsAuditLogger'].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin metadata plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Metadata Plugin for hdfs-NAMENODE with count 1 pipelines names [''].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin profile plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Profile Plugin for hdfs-NAMENODE with count 1 pipelines names ['TP'].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin monitor refresh.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread abstract_monitor INFO     Refreshing GenericMonitor HDFS-NAMENODE for None
[12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__     INFO     New monitor: (<cmf.monitor.generic.GenericMonitor object at 0x7f5e3c202a10>,)
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Daemon refresh complete for process 216-hdfs-NAMENODE.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [212-cloudera-mgmt-SERVICEMONITOR] Reinstantiating process 212
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [212-cloudera-mgmt-SERVICEMONITOR] Updating process: True {}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [212-cloudera-mgmt-SERVICEMONITOR] Refreshing process files: [u'cloudera-stack-monitor.properties']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     prepare_environment begin: {u'CDH': u'5.16.1-1.cdh5.16.1.p0.3'}, [], []
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Service does not request any parcels
[12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__     INFO     Extracted 1 files and 0 dirs to /run/cloudera-scm-agent/process/212-cloudera-mgmt-SERVICEMONITOR.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [212-cloudera-mgmt-SERVICEMONITOR] Evaluating resource: [{u'path': u'/var/log/cloudera-scm-firehose/stacks', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'cloudera-scm', u'user': u'cloudera-scm', u'mode': 493}]
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin audit plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Audit Plugin for cloudera-mgmt-SERVICEMONITOR with count 1 pipelines names [''].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin metadata plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Metadata Plugin for cloudera-mgmt-SERVICEMONITOR with count 1 pipelines names [''].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin profile plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Profile Plugin for cloudera-mgmt-SERVICEMONITOR with count 1 pipelines names ['TP'].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin monitor refresh.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread abstract_monitor INFO     Refreshing GenericMonitor MGMT-SERVICEMONITOR for None
[12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__     INFO     New monitor: (<cmf.monitor.generic.GenericMonitor object at 0x7f5e3c202250>,)
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Daemon refresh complete for process 212-cloudera-mgmt-SERVICEMONITOR.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [213-zookeeper-server] Reinstantiating process 213
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [213-zookeeper-server] Updating process: True {}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [213-zookeeper-server] Refreshing process files: [u'cloudera-stack-monitor.properties']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     prepare_environment begin: {u'SPARK2': u'2.3.0.cloudera4-1.cdh5.13.3.p0.611179', u'CDH': u'5.16.1-1.cdh5.16.1.p0.3'}, [u'cdh'], [u'cdh-plugin', u'zookeeper-plugin']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     The following requested parcels are not available: {}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Obtained tags ['spark2'] for parcel SPARK2
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Obtained tags ['cdh', 'impala', 'kudu', 'sentry', 'solr', 'spark'] for parcel CDH
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.16.1-1.cdh5.16.1.p0.3'}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__     INFO     Extracted 1 files and 0 dirs to /run/cloudera-scm-agent/process/213-zookeeper-server.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [213-zookeeper-server] Evaluating resource: [{u'path': u'/var/log/zookeeper/stacks', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'zookeeper', u'user': u'zookeeper', u'mode': 493}]
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin audit plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Audit Plugin for zookeeper-server with count 1 pipelines names [''].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin metadata plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Metadata Plugin for zookeeper-server with count 1 pipelines names [''].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin profile plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Profile Plugin for zookeeper-server with count 1 pipelines names ['TP'].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin monitor refresh.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread abstract_monitor INFO     Refreshing GenericMonitor ZOOKEEPER-SERVER for None
[12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__     INFO     New monitor: (<cmf.monitor.generic.GenericMonitor object at 0x7f5e3c110550>,)
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Daemon refresh complete for process 213-zookeeper-server.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [230-hive-HIVESERVER2] Reinstantiating process 230
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [230-hive-HIVESERVER2] Updating process: True {}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [230-hive-HIVESERVER2] Refreshing process files: [u'cloudera-stack-monitor.properties', u'navigator.client.properties', u'navigator.lineage.client.properties', u'fair-scheduler.xml']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     prepare_environment begin: {u'SPARK2': u'2.3.0.cloudera4-1.cdh5.13.3.p0.611179', u'CDH': u'5.16.1-1.cdh5.16.1.p0.3'}, [u'cdh'], [u'cdh-plugin', u'hive-plugin']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     The following requested parcels are not available: {}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Obtained tags ['spark2'] for parcel SPARK2
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Obtained tags ['cdh', 'impala', 'kudu', 'sentry', 'solr', 'spark'] for parcel CDH
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.16.1-1.cdh5.16.1.p0.3'}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__     INFO     Extracted 4 files and 0 dirs to /run/cloudera-scm-agent/process/230-hive-HIVESERVER2.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [230-hive-HIVESERVER2] Evaluating resource: [{u'path': u'/var/log/hive/stacks', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'hive', u'user': u'hive', u'mode': 493}]
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin audit plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Audit Plugin for hive-HIVESERVER2 with count 3 pipelines names ['HiveExecHookContext', 'HiveMetaStoreEventListener', 'HiveSentryOnFailureHook'].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin metadata plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Metadata Plugin for hive-HIVESERVER2 with count 1 pipelines names [''].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin profile plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Profile Plugin for hive-HIVESERVER2 with count 1 pipelines names ['TP'].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin monitor refresh.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread abstract_monitor INFO     Refreshing GenericMonitor HIVE-HIVESERVER2 for None
[12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__     INFO     New monitor: (<cmf.monitor.generic.GenericMonitor object at 0x7f5e3c17ff90>,)
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Daemon refresh complete for process 230-hive-HIVESERVER2.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [231-oozie-OOZIE_SERVER] Reinstantiating process 231
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [231-oozie-OOZIE_SERVER] Updating process: True {}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [231-oozie-OOZIE_SERVER] Refreshing process files: [u'cloudera-stack-monitor.properties']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     prepare_environment begin: {u'SPARK2': u'2.3.0.cloudera4-1.cdh5.13.3.p0.611179', u'CDH': u'5.16.1-1.cdh5.16.1.p0.3'}, [u'cdh'], [u'cdh-plugin', u'oozie-plugin']
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     The following requested parcels are not available: {}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Obtained tags ['spark2'] for parcel SPARK2
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Obtained tags ['cdh', 'impala', 'kudu', 'sentry', 'solr', 'spark'] for parcel CDH
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.16.1-1.cdh5.16.1.p0.3'}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__     INFO     Extracted 1 files and 0 dirs to /run/cloudera-scm-agent/process/231-oozie-OOZIE_SERVER.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [231-oozie-OOZIE_SERVER] Evaluating resource: [{u'path': u'/var/log/oozie/stacks', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'oozie', u'user': u'oozie', u'mode': 493}]
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin audit plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Audit Plugin for oozie-OOZIE_SERVER with count 1 pipelines names [''].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin metadata plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Metadata Plugin for oozie-OOZIE_SERVER with count 1 pipelines names [''].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin profile plugin refresh
[12/Jan/2019 00:51:39 +0000] 5084 MainThread navigator_plugin INFO     Scheduling a refresh for Profile Plugin for oozie-OOZIE_SERVER with count 1 pipelines names ['TP'].
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Begin monitor refresh.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread abstract_monitor INFO     Refreshing GenericMonitor OOZIE-OOZIE_SERVER for None
[12/Jan/2019 00:51:39 +0000] 5084 MainThread __init__     INFO     New monitor: (<cmf.monitor.generic.GenericMonitor object at 0x7f5e3e965510>,)
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     Daemon refresh complete for process 231-oozie-OOZIE_SERVER.
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      WARNING  New process is not running: {u'refresh_files': [], u'config_generation': 0, u'auto_restart': False, u'running': False, u'required_tags': [], u'one_off': True, u'special_file_info': [], u'group': u'root', u'id': 234, u'status_links': {}, u'name': u'host-inspector', u'extra_groups': [], u'run_generation': 2, u'start_timeout_seconds': None, u'optional_tags': [], u'program': u'mgmt/mgmt.sh', u'arguments': [u'inspector', u'input.json', u'output.json', u'DEFAULT'], u'parcels': {}, u'resources': [], u'user': u'root'}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread agent        INFO     [234-host-inspector] Instantiating process
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [234-host-inspector] Updating process: False {}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread agent        INFO     Triggering supervisord update.
[12/Jan/2019 00:51:39 +0000] 5084 MonitorDaemon-Scheduler __init__     INFO     Monitor ready to report: ('GenericMonitor HDFS-NAMENODE for None',)
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      ERROR    [234-host-inspector] Failed to write out saved config: [Errno 2] No such file or directory: '/run/cloudera-scm-agent/process/234-host-inspector/proc.json'
Traceback (most recent call last):
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.1-py2.7.egg/cmf/process.py", line 367, in update_heartbeat
    f = self.agent.os_ops.open_write_file(os.path.join(self.dir, self.PROC_JSON))
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.1-py2.7.egg/cmf/util/os_ops.py", line 134, in open_write_file
    fd = os.open(path, os.O_RDWR | os.O_CREAT | os.O_TRUNC, mode)
OSError: [Errno 2] No such file or directory: '/run/cloudera-scm-agent/process/234-host-inspector/proc.json'
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      WARNING  New process is not running: {u'refresh_files': [], u'config_generation': 0, u'auto_restart': False, u'running': False, u'required_tags': [u'cdh'], u'one_off': True, u'special_file_info': [], u'group': u'hdfs', u'id': 220, u'status_links': {}, u'name': u'hdfs-NAMENODE-nnRpcWait', u'extra_groups': [], u'run_generation': 2, u'start_timeout_seconds': None, u'optional_tags': [u'cdh-plugin', u'hdfs-plugin'], u'program': u'hdfs/hdfs.sh', u'arguments': [u'nnRpcWait', u'hdfs://pan0141.panoulu.net:8020'], u'parcels': {u'SPARK2': u'2.3.0.cloudera4-1.cdh5.13.3.p0.611179', u'CDH': u'5.16.1-1.cdh5.16.1.p0.3'}, u'resources': [], u'user': u'hdfs'}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread agent        INFO     [220-hdfs-NAMENODE-nnRpcWait] Instantiating process
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      INFO     [220-hdfs-NAMENODE-nnRpcWait] Updating process: False {}
[12/Jan/2019 00:51:39 +0000] 5084 MainThread agent        INFO     Triggering supervisord update.
[12/Jan/2019 00:51:39 +0000] 5084 MonitorDaemon-Scheduler __init__     INFO     Monitor ready to report: ('GenericMonitor MGMT-SERVICEMONITOR for None',)
[12/Jan/2019 00:51:39 +0000] 5084 MonitorDaemon-Scheduler __init__     INFO     Monitor ready to report: ('GenericMonitor HIVE-HIVESERVER2 for None',)
[12/Jan/2019 00:51:39 +0000] 5084 MonitorDaemon-Scheduler __init__     INFO     Monitor ready to report: ('GenericMonitor ZOOKEEPER-SERVER for None',)
[12/Jan/2019 00:51:39 +0000] 5084 MonitorDaemon-Scheduler __init__     INFO     Monitor ready to report: ('GenericMonitor OOZIE-OOZIE_SERVER for None',)
[12/Jan/2019 00:51:39 +0000] 5084 MainThread process      ERROR    [220-hdfs-NAMENODE-nnRpcWait] Failed to write out saved config: [Errno 2] No such file or directory: '/run/cloudera-scm-agent/process/220-hdfs-NAMENODE-nnRpcWait/proc.json'
Traceback (most recent call last):
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.1-py2.7.egg/cmf/process.py", line 367, in update_heartbeat
    f = self.agent.os_ops.open_write_file(os.path.join(self.dir, self.PROC_JSON))
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.1-py2.7.egg/cmf/util/os_ops.py", line 134, in open_write_file
    fd = os.open(path, os.O_RDWR | os.O_CREAT | os.O_TRUNC, mode)
OSError: [Errno 2] No such file or directory: '/run/cloudera-scm-agent/process/220-hdfs-NAMENODE-nnRpcWait/proc.json'
[12/Jan/2019 00:51:39 +0000] 5084 MainThread agent        WARNING  Long HB processing time: 11.3919100761
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Loading parcel manifest for: SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Loading parcel manifest for: CDH-5.16.1-1.cdh5.16.1.p0.3
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Loading parcel manifest for: SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179
[12/Jan/2019 00:51:39 +0000] 5084 MainThread parcel       INFO     Loading parcel manifest for: CDH-5.16.1-1.cdh5.16.1.p0.3
[12/Jan/2019 00:51:40 +0000] 5084 MonitorDaemon-Reporter firehoses    INFO     Creating a connection to the ACTIVITYMONITOR.
[12/Jan/2019 00:51:40 +0000] 5084 MonitorDaemon-Reporter firehoses    INFO     Creating a connection to the SERVICEMONITOR.
[12/Jan/2019 00:51:40 +0000] 5084 MonitorDaemon-Reporter firehoses    INFO     Creating a connection to the HOSTMONITOR.
[12/Jan/2019 00:51:41 +0000] 5084 MainThread agent        INFO     PID '5371' associated with process 'flood' with payload 'processname:flood groupname:flood from_state:RUNNING expected:0 pid:5371' exited unexpectedly
[12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO     Piplines updated for Audit Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO     Refreshing Audit Plugin for hdfs-NAMENODE with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO     Piplines updated for Profile Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO     Piplines updated for Audit Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO     Refreshing Audit Plugin for zookeeper-server with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO     Refreshing Profile Plugin for cloudera-mgmt-SERVICEMONITOR with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO     Piplines updated for Metadata Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO     Refreshing Metadata Plugin for zookeeper-server with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO     Piplines updated for Profile Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO     Piplines updated for Metadata Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO     Piplines updated for Audit Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO     Refreshing Profile Plugin for zookeeper-server with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO     Refreshing Metadata Plugin for cloudera-mgmt-SERVICEMONITOR with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO     Refreshing Audit Plugin for hive-HIVESERVER2 with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO     Piplines updated for Profile Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO     Piplines updated for Metadata Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO     Refreshing Metadata Plugin for oozie-OOZIE_SERVER with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO     Refreshing Profile Plugin for hdfs-NAMENODE with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO     Piplines updated for Metadata Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO     Piplines updated for Audit Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO     Piplines updated for Profile Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO     Refreshing Metadata Plugin for hive-HIVESERVER2 with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO     Piplines updated for Metadata Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO     Refreshing Profile Plugin for oozie-OOZIE_SERVER with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Metadata-Plugin navigator_plugin INFO     Refreshing Metadata Plugin for hdfs-NAMENODE with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO     Refreshing Audit Plugin for oozie-OOZIE_SERVER with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO     Piplines updated for Profile Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO     Piplines updated for Audit Plugin: []
[12/Jan/2019 00:51:43 +0000] 5084 Profile-Plugin navigator_plugin INFO     Refreshing Profile Plugin for hive-HIVESERVER2 with count 0 pipelines names [].
[12/Jan/2019 00:51:43 +0000] 5084 Audit-Plugin navigator_plugin INFO     Refreshing Audit Plugin for cloudera-mgmt-SERVICEMONITOR with count 0 pipelines names [].

Re: Spark 2.3 Csd failed to load

hadoopNoob — Fri, 11 Jan 2019 22:56:58 GMT

Here are all the details

Re: Spark 2.3 Csd failed to load

hadoopNoob — Sat, 12 Jan 2019 13:48:50 GMT

I have solve the issue thanks for your help

Re: Spark 2.3 Csd failed to load

lwang — Tue, 15 Jan 2019 21:09:14 GMT

Hi @hadoopNoob,

Very glad to see you solved the issue. Could you please share the solution here so others can benefit from your findings?

Thanks very much for your time!

Re: Spark 2.3 Csd failed to load

hadoopNoob — Tue, 30 Apr 2019 07:49:27 GMT

Yes, the issue was with java, re-configuring it solved the issue