Cloudera Manager CDH 5.7.1 Installation issues - the most un-smooth install till now :-(

[#1]

Hi guys

First of, my Iove for CDH and Cloudera Manager will never decrease !

I know I use the starving developers version of Cloudera and possibly have lesser or no rights to complain 🙂 ; however the support you guys have demonstrated over the past 4 years has kept my faith and confidence and I feel like a privileged Cloudera user and continue to report problems and issues with the hope that it helps other users and the feedback makes the product even more bulletproof !  

However yesterday while I was installing CM 5.7.1 on a new 3 node Ubntu 12.04 cluster , I faced the maximum issues ever

1. After selecting services to install (ZK, HDFS, Yarn, Hive, Impala, Spark on Yarn) , during the first services start phase of the CM install, spark failed to start

2. I aborted installation and started  manually adding service from CM

    - first ZK, then HDFS(had to format HDFS) , YARN, Spark

    - Spark failed...I copied spark-assemply jar from the

/opt/cloudera/parcels/CDH//lib/spark/assembly/lib/spark-assembly.jar

directory to /user/spark/share/lib/spark-assembly.jar on HDFS and put the appropriate chown and chmod settings

3. Spark on Yarn just fails to start...

Any ideas ? Recommendations ? Thoughts ? Would love to learn and implement 

warmly 

sanjay

[#2]

Add a Spark Service to Cluster 1
First Run Command
Status: Failed Start Time: Jul 20, 12:15:05 PM Duration: 15.37s Retry
Failed to perform First Run of services.
All Failed Only Running Only Details
Completed 1 of 5 step(s).
Step Context Start Time Duration Actions
Deploy Client Configuration
Failed to deploy client configuration to the cluster.
Cluster 1
Jul 20, 12:15:05 PM 15.36s
Execute DeployClusterClientConfig for {yarn,hdfs,hive,spark_on_yarn} in parallel.
Completed only 3/4 steps. First failure: Failed to execute command Deploy Client Configuration on service Spark
Jul 20, 12:15:05 PM 15.36s
Deploy Client Configuration
Successfully deployed client configuration.
YARN (MR2 Included)
Jul 20, 12:15:05 PM 242ms
Deploy Client Configuration
Successfully deployed client configuration.
HDFS
Jul 20, 12:15:05 PM 212ms
Deploy Client Configuration
Successfully deployed client configuration.
Hive
Jul 20, 12:15:05 PM 170ms
Deploy Client Configuration
Deploy Client Configuration failed.
Spark
Jul 20, 12:15:05 PM 15.21s
Generate and deploy client configuration.
Completed only 2/3 steps. First failure: Client configuration (id=25) on host n1-3hadoop-dev01 (id=1) exited with 1 and expected 0.
Jul 20, 12:15:05 PM 15.21s
Execute command Create Spark User Dir on service Spark
Execute command Create Spark History Log Dir on service Spark
Execute command Install Spark JAR on service Spark
Start Spark

[#3]

These are errors from 

/var/run/cloudera-scm-agent/process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_-8951873637091761528/logs/stderr.log

+ perl -pi -e 's#{{HIVE_HBASE_JAR}}#/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hive/lib/hive-hbase-handler-1.1.0-cdh5.7.1.jar,/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hbase/hbase-hadoop-compat.jar,/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hbase/hbase-common.jar,/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hbase/hbase-protocol.jar,/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hbase/lib/htrace-core.jar,/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hbase/lib/htrace-core-3.2.0-incubating.jar,/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hbase/lib/htrace-core4-4.0.1-incubating.jar,/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hbase/hbase-server.jar,/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hbase/hbase-hadoop2-compat.jar,/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hbase/hbase-client.jar#g' /run/cloudera-scm-agent/process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_-8951873637091761528/spark-conf/hive-env.sh

Can't open /run/cloudera-scm-agent/process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_-8951873637091761528/spark-conf/hive-env.sh: No such file or directory.

+ /run/cloudera-scm-agent/process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_-8951873637091761528/scripts/control.sh client

/usr/lib/cmf/service/client/deploy-cc.sh: line 190: /run/cloudera-scm-agent/process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_-8951873637091761528/scripts/control.sh: Permission denied

[#4 · Accepted solution]

So here is how I solved this issue.

- Installed 5.8.0 Cloudera Manager and corresponding 5.8.0 parcels for the cluster

- Added Spark (on Yarn) as a service

- no issues faced

Maybe something was not right in CM 5.7.1 ?

[#5]

Considering the amount of effort required to get Cloudera to work would it not just be easier to install and configure Hadoop on your own?