Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Deploy Client Configuration failed

Deploy Client Configuration failed

New Contributor

I try to setup a new cluster with Cloudera Manager 5.14.2.

On the first run of deployment I got the following error:

 

Completed only 0/3 steps. First failure: Client configuration (id=7) on host osr-node-2 (id=1) exited with 127 and expected 0.

The deployment of Spark failed on the second node (I installed 3 nodes only).

 

Agent error log from this node:

 

[07/Jun/2018 16:30:52 +0000] 2720 MainThread __init__ INFO Extracted 16 files and 0 dirs to /run/cloudera-scm-agent/process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_-2478320698396427578.
[07/Jun/2018 16:31:07 +0000] 2720 MainThread client_configs INFO Client configs to delete: set([])
[07/Jun/2018 16:31:07 +0000] 2720 MainThread client_configs INFO Client configs to deploy: set([<spark-conf,/etc/spark/conf.cloudera.spark_on_yarn>])
[07/Jun/2018 16:31:07 +0000] 2720 MainThread client_configs INFO Client configs to update: set([])
[07/Jun/2018 16:31:07 +0000] 2720 MainThread parcel INFO prepare_environment begin: {}, [u'cdh', u'spark'], [u'spark-plugin']
[07/Jun/2018 16:31:07 +0000] 2720 MainThread parcel INFO No parcels activated for use
[07/Jun/2018 16:31:07 +0000] 2720 MainThread __init__ INFO Extracted 16 files and 0 dirs to /run/cloudera-scm-agent/process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_-2478320698396427578.
[07/Jun/2018 16:31:22 +0000] 2720 MainThread client_configs INFO Client configs to delete: set([])
[07/Jun/2018 16:31:22 +0000] 2720 MainThread client_configs INFO Client configs to deploy: set([<spark-conf,/etc/spark/conf.cloudera.spark_on_yarn>])
[07/Jun/2018 16:31:22 +0000] 2720 MainThread client_configs INFO Client configs to update: set([])
[07/Jun/2018 16:31:22 +0000] 2720 MainThread parcel INFO prepare_environment begin: {}, [u'cdh', u'spark'], [u'spark-plugin']
[07/Jun/2018 16:31:22 +0000] 2720 MainThread parcel INFO No parcels activated for use
[07/Jun/2018 16:31:22 +0000] 2720 MainThread __init__ INFO Extracted 16 files and 0 dirs to /run/cloudera-scm-agent/process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_-2478320698396427578.
[07/Jun/2018 16:31:36 +0000] 2720 MainThread client_configs INFO Client configs to delete: set([])
[07/Jun/2018 16:31:36 +0000] 2720 MainThread client_configs INFO Client configs to deploy: set([<spark-conf,/etc/spark/conf.cloudera.spark_on_yarn>])
[07/Jun/2018 16:31:36 +0000] 2720 MainThread client_configs INFO Client configs to update: set([])
[07/Jun/2018 16:31:36 +0000] 2720 MainThread parcel INFO prepare_environment begin: {}, [u'cdh', u'spark'], [u'spark-plugin']
[07/Jun/2018 16:31:36 +0000] 2720 MainThread parcel INFO No parcels activated for use
[07/Jun/2018 16:31:37 +0000] 2720 MainThread __init__ INFO Extracted 16 files and 0 dirs to /run/cloudera-scm-agent/process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_-2478320698396427578.
[07/Jun/2018 16:31:51 +0000] 2720 MainThread client_configs INFO Client configs to delete: set([])
[07/Jun/2018 16:31:51 +0000] 2720 MainThread client_configs INFO Client configs to deploy: set([<spark-conf,/etc/spark/conf.cloudera.spark_on_yarn>])
[07/Jun/2018 16:31:51 +0000] 2720 MainThread client_configs INFO Client configs to update: set([])
[07/Jun/2018 16:31:51 +0000] 2720 MainThread parcel INFO prepare_environment begin: {}, [u'cdh', u'spark'], [u'spark-plugin']
[07/Jun/2018 16:31:51 +0000] 2720 MainThread parcel INFO No parcels activated for use
[07/Jun/2018 16:31:52 +0000] 2720 MainThread __init__ INFO Extracted 16 files and 0 dirs to /run/cloudera-scm-agent/process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_-2478320698396427578.
[07/Jun/2018 16:32:07 +0000] 2720 MainThread client_configs INFO Client configs to delete: set([])
[07/Jun/2018 16:32:07 +0000] 2720 MainThread client_configs INFO Client configs to deploy: set([<spark-conf,/etc/spark/conf.cloudera.spark_on_yarn>])
[07/Jun/2018 16:32:07 +0000] 2720 MainThread client_configs INFO Client configs to update: set([])
[07/Jun/2018 16:32:07 +0000] 2720 MainThread parcel INFO prepare_environment begin: {}, [u'cdh', u'spark'], [u'spark-plugin']
[07/Jun/2018 16:32:07 +0000] 2720 MainThread parcel INFO No parcels activated for use
[07/Jun/2018 16:32:07 +0000] 2720 MainThread __init__ INFO Extracted 16 files and 0 dirs to /run/cloudera-scm-agent/process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_-2478320698396427578.

 

Cloudera Manager log:


2018-06-07 16:31:07,500 WARN EventStorePublisherWithRetry-0:com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: Failed to publish event: SimpleEvent{attributes={ROLE_TYPE=[EVENTSERVER], CATEGORY=[AUDIT_EVENT], ROLE=[mgmt-EVENTSERVER-5cf8c4e29aa6830d24fee33c4b53d12a], SEVERITY=[INFORMATIONAL], SERVICE=[mgmt], HOST_IDS=[f4d7e0c5-6ece-4cb8-8fd7-d558019f704a], SERVICE_TYPE=[MGMT], USER=[admin], EVENTCODE=[EV_ROLE_CREATED], MESSAGE_CODES=[ROLE_CREATED], HOSTS=[osr-node-1]}, content=User admin created role mgmt-EVENTSERVER-5cf8c4e29aa6830d24fee33c4b53d12a for service mgmt., timestamp=1528369263096} - 1 of 343 failure(s) in last 1800s
2018-06-07 16:31:19,697 INFO CMMetricsForwarder-0:com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder: (29 skipped) Failed to send metrics.
java.lang.reflect.UndeclaredThrowableException
at com.sun.proxy.$Proxy130.writeMetrics(Unknown Source)
at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendWithAvro(ClouderaManagerMetricsForwarder.java:325)
at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendMetrics(ClouderaManagerMetricsForwarder.java:312)
at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.run(ClouderaManagerMetricsForwarder.java:146)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused
at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:88)
... 11 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
at sun.net.www.http.HttpClient.New(HttpClient.java:308)
at sun.net.www.http.HttpClient.New(HttpClient.java:326)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091)
at org.apache.avro.ipc.HttpTransceiver.writeBuffers(HttpTransceiver.java:71)
at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:58)
at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:72)
at org.apache.avro.ipc.Requestor.request(Requestor.java:147)
at org.apache.avro.ipc.Requestor.request(Requestor.java:101)
at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:72)
... 11 more
2018-06-07 16:31:21,710 WARN 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:31:21,711 INFO 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.135 over PT0.001S. Rate: 76.853 MB/s
2018-06-07 16:31:21,820 WARN 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:31:21,821 INFO 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.133 over PT0.001S. Rate: 76.853 MB/s
2018-06-07 16:31:36,729 WARN 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:31:36,732 INFO 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.135 over PT0.003S. Rate: 25.618 MB/s
2018-06-07 16:31:36,848 WARN 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:31:36,849 INFO 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.133 over PT0.001S. Rate: 76.853 MB/s
2018-06-07 16:31:51,699 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (11 skipped) AgentAvroServlet: heartbeat processing stats: average=30ms, min=9ms, max=270ms.
2018-06-07 16:31:51,848 WARN 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:31:51,868 INFO 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.135 over PT0.019S. Rate: 4.045 MB/s
2018-06-07 16:31:51,877 WARN 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:31:51,939 INFO 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.133 over PT0.062S. Rate: 1.240 MB/s
2018-06-07 16:32:03,654 INFO JvmPauseMonitor:com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1501ms: no GCs detected.
2018-06-07 16:32:06,764 WARN 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:32:06,779 INFO 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.135 over PT0.015S. Rate: 5.124 MB/s
2018-06-07 16:32:06,865 WARN 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:32:06,866 INFO 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.133 over PT0.001S. Rate: 76.853 MB/s
2018-06-07 16:32:21,758 WARN 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:32:21,760 INFO 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.135 over PT0.002S. Rate: 38.426 MB/s
2018-06-07 16:32:21,871 WARN 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:32:21,872 INFO 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.133 over PT0.001S. Rate: 76.853 MB/s
2018-06-07 16:32:36,776 WARN 1018229141@scm-web-146:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:32:36,779 INFO 1018229141@scm-web-146:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.135 over PT0.003S. Rate: 25.618 MB/s
2018-06-07 16:32:36,933 WARN 1018229141@scm-web-146:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:32:36,934 INFO 1018229141@scm-web-146:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.133 over PT0.001S. Rate: 76.853 MB/s
2018-06-07 16:32:51,729 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (11 skipped) AgentAvroServlet: heartbeat processing stats: average=30ms, min=9ms, max=270ms.
2018-06-07 16:32:51,797 WARN 1018229141@scm-web-146:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:32:51,801 INFO 1018229141@scm-web-146:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.135 over PT0.003S. Rate: 25.618 MB/s
2018-06-07 16:32:51,907 WARN 1018229141@scm-web-146:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:32:51,909 INFO 1018229141@scm-web-146:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.133 over PT0.001S. Rate: 76.853 MB/s
2018-06-07 16:33:06,767 WARN 1018229141@scm-web-146:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:33:06,768 INFO 1018229141@scm-web-146:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.135 over PT0.001S. Rate: 76.853 MB/s
2018-06-07 16:33:06,923 WARN 1018229141@scm-web-146:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:33:06,933 INFO 1018229141@scm-web-146:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.133 over PT0.010S. Rate: 7.685 MB/s

 

What is the issue?