Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

deployClientConfig on service Spark fails in CDH5

deployClientConfig on service Spark fails in CDH5

Explorer

When I add in the Spark service using the CM in CDH5, the service adds successfully, but when I complete the process, I'm advised that the client configs for Spark need to be redeployed. When I run the deployment of the client configs for Spark, this process fails with the message "Deploy Client Configuration failed." Command (5809) has failed.

11 REPLIES 11

Re: deployClientConfig on service Spark fails in CDH5

What does the stderr log say?

If you can't find it in the UI, you can ssh into the host and look in /var/run/cloudera-scm-agent/process/

Sort by most recent (ls -ltr) and look for the directory corresponding with your deploy command for Spark. Look in the logs dir for stderr.

Re: deployClientConfig on service Spark fails in CDH5

Explorer

Thanks.

 

So we're using a CM manager node, two name nodes and 8 datanodes. Would I start looking at the logs on the management node, since this is where the error in deployment seemed to initiate? Or the Spark master node?

Re: deployClientConfig on service Spark fails in CDH5

Good question. Look on the node where the deploy command failed. This is going to be a node with a Spark role assigned to it.

If you don't see it there, then maybe it failed before the command was sent to the host, so check the CM server's logs for a relevant error message (/var/log/cloudera-scm-server/).

Re: deployClientConfig on service Spark fails in CDH5

Explorer

This is the very last entry in the stderr.log on the spark master in the last deploy-client-config for Spark.

 

Tue Aug 26 11:52:22 PDT 2014
Tue Aug 26 11:52:22 PDT 2014: Detected CDH_VERSION of [5]
Tue Aug 26 11:52:22 PDT 2014: Found a master on [hostname-removed] listening on port 7077
Tue Aug 26 11:52:22 PDT 2014: Deploying client configuration
+ echo 'deploy script exited with 0'
+ popd
+ rm -rf conf.cloudera.spark
+ cp -a /var/run/cloudera-scm-agent/process/4503-deploy-client-config/spark-conf conf.cloudera.spark
+ chmod -R ugo+r conf.cloudera.spark
+ [[ SPARK == HIVE ]]
+ /usr/sbin/update-alternatives --install /etc/spark/conf spark-conf /etc/spark/conf.cloudera.spark 50
/var/lib/alternatives/spark-conf empty!

Re: deployClientConfig on service Spark fails in CDH5

Expert Contributor

please i am facing same problem , have you fixed it ? 

Re: deployClientConfig on service Spark fails in CDH5

Explorer
I had to upgrade Cloudera Manager to be the same version as the CDH I was currently running. As soon as Cloudera Manager and CDH were the same, the problem was resolved.

Re: deployClientConfig on service Spark fails in CDH5

Expert Contributor

i ran hosts => hosts inspector   and showed no conflicts , although the CDH and CM versions are:

 

cloudera manager agent 5.0.2

 

CDH 5.3.1

 

so is that the problem ? 

Re: deployClientConfig on service Spark fails in CDH5

Expert Contributor

today i upgraded cloudera manager , but i went to far complicated issues with postgresql and other conflicts , luckly i had taken a snapshot of the VM , so i reverted back , is there any other solution than upgrading cloudera manager ?  

Re: deployClientConfig on service Spark fails in CDH5

Expert Contributor

when i run deploy client configuration in both spark and hbase  services

the log goes like this : 

 

for spark service for example : 

 

Generate client configuration.
Successfully generated client configuration.
Run 3 steps in parallel
Completed 2/3 steps successfully
Cleanup client configs that failed to deploy.
Successfully cleaned up client configs that failed to deploy.

 

Program: client/deploy-cc.sh ["spark-clientconfig.zip","conf.cloudera.spark"]

Recent Log Entries Links to full logs: Stderr Stdout
Stderr
Stdout
Can't open /var/run/cloudera-scm-agent/process/800-deploy-client-config/spark-conf/hive-env.sh: No such file or directory.
+ '[' '!' -e /etc/spark ']'
+ cd /etc/spark
+ for SPECIAL_FILE in '$DEST_PATH/{taskcontroller.cfg,container-executor.cfg,topology.py,topology.map}'
+ '[' -e /etc/spark/conf.cloudera.spark/taskcontroller.cfg ']'
+ for SPECIAL_FILE in '$DEST_PATH/{taskcontroller.cfg,container-executor.cfg,topology.py,topology.map}'
+ '[' -e /etc/spark/conf.cloudera.spark/container-executor.cfg ']'
+ for SPECIAL_FILE in '$DEST_PATH/{taskcontroller.cfg,container-executor.cfg,topology.py,topology.map}'
+ '[' -e /etc/spark/conf.cloudera.spark/topology.py ']'
+ for SPECIAL_FILE in '$DEST_PATH/{taskcontroller.cfg,container-executor.cfg,topology.py,topology.map}'
+ '[' -e /etc/spark/conf.cloudera.spark/topology.map ']'
+ '[' -d conf -a '!' -L conf ']'
+ '[' -n scripts/control.sh ']'
+ chmod u+x /var/run/cloudera-scm-agent/process/800-deploy-client-config/scripts/control.sh
+ echo 'invoking optional deploy script scripts/control.sh'
+ SRC_PATH=/var/run/cloudera-scm-agent/process/800-deploy-client-config/spark-conf
+ '[' '!' -d /var/run/cloudera-scm-agent/process/800-deploy-client-config/spark-conf ']'
+ pushd /var/run/cloudera-scm-agent/process/800-deploy-client-config/spark-conf
+ /var/run/cloudera-scm-agent/process/800-deploy-client-config/scripts/control.sh client master.properties
Tue Mar 10 18:28:29 AST 2015
Tue Mar 10 18:28:29 AST 2015: Detected CDH_VERSION of [5]
Tue Mar 10 18:28:29 AST 2015: Found a master on bdvm01.ejada.com listening on port 7077
Tue Mar 10 18:28:29 AST 2015: Deploying client configuration
+ echo 'deploy script exited with 0'
+ popd
+ rm -rf conf.cloudera.spark
+ cp -a /var/run/cloudera-scm-agent/process/800-deploy-client-config/spark-conf conf.cloudera.spark
+ chmod -R ugo+r conf.cloudera.spark
+ [[ SPARK == HIVE ]]
+ /usr/sbin/update-alternatives --install /etc/spark/conf spark-conf /etc/spark/conf.cloudera.spark 50
+ /usr/sbin/update-alternatives --auto spark-conf
Program: client/deploy-cc.sh ["spark-clientconfig.zip","conf.cloudera.spark"]

Recent Log Entries Links to full logs: Stderr Stdout
Stderr
Stdout
Can't open /var/run/cloudera-scm-agent/process/799-deploy-client-config/spark-conf/hive-env.sh: No such file or directory.
+ '[' '!' -e /etc/spark ']'
+ cd /etc/spark
+ for SPECIAL_FILE in '$DEST_PATH/{taskcontroller.cfg,container-executor.cfg,topology.py,topology.map}'
+ '[' -e /etc/spark/conf.cloudera.spark/taskcontroller.cfg ']'
+ for SPECIAL_FILE in '$DEST_PATH/{taskcontroller.cfg,container-executor.cfg,topology.py,topology.map}'
+ '[' -e /etc/spark/conf.cloudera.spark/container-executor.cfg ']'
+ for SPECIAL_FILE in '$DEST_PATH/{taskcontroller.cfg,container-executor.cfg,topology.py,topology.map}'
+ '[' -e /etc/spark/conf.cloudera.spark/topology.py ']'
+ for SPECIAL_FILE in '$DEST_PATH/{taskcontroller.cfg,container-executor.cfg,topology.py,topology.map}'
+ '[' -e /etc/spark/conf.cloudera.spark/topology.map ']'
+ '[' -d conf -a '!' -L conf ']'
+ '[' -n scripts/control.sh ']'
+ chmod u+x /var/run/cloudera-scm-agent/process/799-deploy-client-config/scripts/control.sh
+ echo 'invoking optional deploy script scripts/control.sh'
+ SRC_PATH=/var/run/cloudera-scm-agent/process/799-deploy-client-config/spark-conf
+ '[' '!' -d /var/run/cloudera-scm-agent/process/799-deploy-client-config/spark-conf ']'
+ pushd /var/run/cloudera-scm-agent/process/799-deploy-client-config/spark-conf
+ /var/run/cloudera-scm-agent/process/799-deploy-client-config/scripts/control.sh client master.properties
Tue Mar 10 18:28:08 AST 2015
Tue Mar 10 18:28:08 AST 2015: Detected CDH_VERSION of [5]
Tue Mar 10 18:28:08 AST 2015: Found a master on bdvm01.ejada.com listening on port 7077
Tue Mar 10 18:28:08 AST 2015: Deploying client configuration
+ echo 'deploy script exited with 0'
+ popd
+ rm -rf conf.cloudera.spark
+ cp -a /var/run/cloudera-scm-agent/process/799-deploy-client-config/spark-conf conf.cloudera.spark
+ chmod -R ugo+r conf.cloudera.spark
+ [[ SPARK == HIVE ]]
+ /usr/sbin/update-alternatives --install /etc/spark/conf spark-conf /etc/spark/conf.cloudera.spark 50
/var/lib/alternatives/spark-conf empty!
Program: client/deploy-cc.sh ["spark-clientconfig.zip","conf.cloudera.spark"]

Recent Log Entries Links to full logs: Stderr Stdout
Stderr
Stdout
Can't open /var/run/cloudera-scm-agent/process/798-deploy-client-config/spark-conf/hive-env.sh: No such file or directory.
+ '[' '!' -e /etc/spark ']'
+ cd /etc/spark
+ for SPECIAL_FILE in '$DEST_PATH/{taskcontroller.cfg,container-executor.cfg,topology.py,topology.map}'
+ '[' -e /etc/spark/conf.cloudera.spark/taskcontroller.cfg ']'
+ for SPECIAL_FILE in '$DEST_PATH/{taskcontroller.cfg,container-executor.cfg,topology.py,topology.map}'
+ '[' -e /etc/spark/conf.cloudera.spark/container-executor.cfg ']'
+ for SPECIAL_FILE in '$DEST_PATH/{taskcontroller.cfg,container-executor.cfg,topology.py,topology.map}'
+ '[' -e /etc/spark/conf.cloudera.spark/topology.py ']'
+ for SPECIAL_FILE in '$DEST_PATH/{taskcontroller.cfg,container-executor.cfg,topology.py,topology.map}'
+ '[' -e /etc/spark/conf.cloudera.spark/topology.map ']'
+ '[' -d conf -a '!' -L conf ']'
+ '[' -n scripts/control.sh ']'
+ chmod u+x /var/run/cloudera-scm-agent/process/798-deploy-client-config/scripts/control.sh
+ echo 'invoking optional deploy script scripts/control.sh'
+ SRC_PATH=/var/run/cloudera-scm-agent/process/798-deploy-client-config/spark-conf
+ '[' '!' -d /var/run/cloudera-scm-agent/process/798-deploy-client-config/spark-conf ']'
+ pushd /var/run/cloudera-scm-agent/process/798-deploy-client-config/spark-conf
+ /var/run/cloudera-scm-agent/process/798-deploy-client-config/scripts/control.sh client master.properties
Tue Mar 10 18:28:28 AST 2015
Tue Mar 10 18:28:28 AST 2015: Detected CDH_VERSION of [5]
Tue Mar 10 18:28:28 AST 2015: Found a master on bdvm01.ejada.com listening on port 7077
Tue Mar 10 18:28:28 AST 2015: Deploying client configuration
+ echo 'deploy script exited with 0'
+ popd
+ rm -rf conf.cloudera.spark
+ cp -a /var/run/cloudera-scm-agent/process/798-deploy-client-config/spark-conf conf.cloudera.spark
+ chmod -R ugo+r conf.cloudera.spark
+ [[ SPARK == HIVE ]]
+ /usr/sbin/update-alternatives --install /etc/spark/conf spark-conf /etc/spark/conf.cloudera.spark 50
+ /usr/sbin/update-alternatives --auto spark-conf

 

 

 

also similar error log is generated when trying deploy configuration for , everything was working fine , but this issue is aroused when i upgradted from CDH 4 to CDH 5

currently using 

cloudera manager agent 5.0.2

CDH 5.3.1

 

please help me , thanks in advance 

Don't have an account?
Coming from Hortonworks? Activate your account here