Reply
Highlighted
New Contributor
Posts: 5
Registered: ‎06-07-2018

Deploy Client Configuration failed

[ Edited ]

I try to setup a new cluster with Cloudera Manager 5.14.2.

On the first run of deployment I got the following error:

 

Completed only 0/3 steps. First failure: Client configuration (id=7) on host osr-node-2 (id=1) exited with 127 and expected 0.

The deployment of Spark failed on the second node (I installed 3 nodes only).

 

Agent error log from this node:

 

[07/Jun/2018 16:30:52 +0000] 2720 MainThread __init__ INFO Extracted 16 files and 0 dirs to /run/cloudera-scm-agent/process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_-2478320698396427578.
[07/Jun/2018 16:31:07 +0000] 2720 MainThread client_configs INFO Client configs to delete: set([])
[07/Jun/2018 16:31:07 +0000] 2720 MainThread client_configs INFO Client configs to deploy: set([<spark-conf,/etc/spark/conf.cloudera.spark_on_yarn>])
[07/Jun/2018 16:31:07 +0000] 2720 MainThread client_configs INFO Client configs to update: set([])
[07/Jun/2018 16:31:07 +0000] 2720 MainThread parcel INFO prepare_environment begin: {}, [u'cdh', u'spark'], [u'spark-plugin']
[07/Jun/2018 16:31:07 +0000] 2720 MainThread parcel INFO No parcels activated for use
[07/Jun/2018 16:31:07 +0000] 2720 MainThread __init__ INFO Extracted 16 files and 0 dirs to /run/cloudera-scm-agent/process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_-2478320698396427578.
[07/Jun/2018 16:31:22 +0000] 2720 MainThread client_configs INFO Client configs to delete: set([])
[07/Jun/2018 16:31:22 +0000] 2720 MainThread client_configs INFO Client configs to deploy: set([<spark-conf,/etc/spark/conf.cloudera.spark_on_yarn>])
[07/Jun/2018 16:31:22 +0000] 2720 MainThread client_configs INFO Client configs to update: set([])
[07/Jun/2018 16:31:22 +0000] 2720 MainThread parcel INFO prepare_environment begin: {}, [u'cdh', u'spark'], [u'spark-plugin']
[07/Jun/2018 16:31:22 +0000] 2720 MainThread parcel INFO No parcels activated for use
[07/Jun/2018 16:31:22 +0000] 2720 MainThread __init__ INFO Extracted 16 files and 0 dirs to /run/cloudera-scm-agent/process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_-2478320698396427578.
[07/Jun/2018 16:31:36 +0000] 2720 MainThread client_configs INFO Client configs to delete: set([])
[07/Jun/2018 16:31:36 +0000] 2720 MainThread client_configs INFO Client configs to deploy: set([<spark-conf,/etc/spark/conf.cloudera.spark_on_yarn>])
[07/Jun/2018 16:31:36 +0000] 2720 MainThread client_configs INFO Client configs to update: set([])
[07/Jun/2018 16:31:36 +0000] 2720 MainThread parcel INFO prepare_environment begin: {}, [u'cdh', u'spark'], [u'spark-plugin']
[07/Jun/2018 16:31:36 +0000] 2720 MainThread parcel INFO No parcels activated for use
[07/Jun/2018 16:31:37 +0000] 2720 MainThread __init__ INFO Extracted 16 files and 0 dirs to /run/cloudera-scm-agent/process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_-2478320698396427578.
[07/Jun/2018 16:31:51 +0000] 2720 MainThread client_configs INFO Client configs to delete: set([])
[07/Jun/2018 16:31:51 +0000] 2720 MainThread client_configs INFO Client configs to deploy: set([<spark-conf,/etc/spark/conf.cloudera.spark_on_yarn>])
[07/Jun/2018 16:31:51 +0000] 2720 MainThread client_configs INFO Client configs to update: set([])
[07/Jun/2018 16:31:51 +0000] 2720 MainThread parcel INFO prepare_environment begin: {}, [u'cdh', u'spark'], [u'spark-plugin']
[07/Jun/2018 16:31:51 +0000] 2720 MainThread parcel INFO No parcels activated for use
[07/Jun/2018 16:31:52 +0000] 2720 MainThread __init__ INFO Extracted 16 files and 0 dirs to /run/cloudera-scm-agent/process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_-2478320698396427578.
[07/Jun/2018 16:32:07 +0000] 2720 MainThread client_configs INFO Client configs to delete: set([])
[07/Jun/2018 16:32:07 +0000] 2720 MainThread client_configs INFO Client configs to deploy: set([<spark-conf,/etc/spark/conf.cloudera.spark_on_yarn>])
[07/Jun/2018 16:32:07 +0000] 2720 MainThread client_configs INFO Client configs to update: set([])
[07/Jun/2018 16:32:07 +0000] 2720 MainThread parcel INFO prepare_environment begin: {}, [u'cdh', u'spark'], [u'spark-plugin']
[07/Jun/2018 16:32:07 +0000] 2720 MainThread parcel INFO No parcels activated for use
[07/Jun/2018 16:32:07 +0000] 2720 MainThread __init__ INFO Extracted 16 files and 0 dirs to /run/cloudera-scm-agent/process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_-2478320698396427578.

 

Cloudera Manager log:


2018-06-07 16:31:07,500 WARN EventStorePublisherWithRetry-0:com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: Failed to publish event: SimpleEvent{attributes={ROLE_TYPE=[EVENTSERVER], CATEGORY=[AUDIT_EVENT], ROLE=[mgmt-EVENTSERVER-5cf8c4e29aa6830d24fee33c4b53d12a], SEVERITY=[INFORMATIONAL], SERVICE=[mgmt], HOST_IDS=[f4d7e0c5-6ece-4cb8-8fd7-d558019f704a], SERVICE_TYPE=[MGMT], USER=[admin], EVENTCODE=[EV_ROLE_CREATED], MESSAGE_CODES=[ROLE_CREATED], HOSTS=[osr-node-1]}, content=User admin created role mgmt-EVENTSERVER-5cf8c4e29aa6830d24fee33c4b53d12a for service mgmt., timestamp=1528369263096} - 1 of 343 failure(s) in last 1800s
2018-06-07 16:31:19,697 INFO CMMetricsForwarder-0:com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder: (29 skipped) Failed to send metrics.
java.lang.reflect.UndeclaredThrowableException
at com.sun.proxy.$Proxy130.writeMetrics(Unknown Source)
at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendWithAvro(ClouderaManagerMetricsForwarder.java:325)
at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendMetrics(ClouderaManagerMetricsForwarder.java:312)
at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.run(ClouderaManagerMetricsForwarder.java:146)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused
at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:88)
... 11 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
at sun.net.www.http.HttpClient.New(HttpClient.java:308)
at sun.net.www.http.HttpClient.New(HttpClient.java:326)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091)
at org.apache.avro.ipc.HttpTransceiver.writeBuffers(HttpTransceiver.java:71)
at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:58)
at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:72)
at org.apache.avro.ipc.Requestor.request(Requestor.java:147)
at org.apache.avro.ipc.Requestor.request(Requestor.java:101)
at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:72)
... 11 more
2018-06-07 16:31:21,710 WARN 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:31:21,711 INFO 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.135 over PT0.001S. Rate: 76.853 MB/s
2018-06-07 16:31:21,820 WARN 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:31:21,821 INFO 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.133 over PT0.001S. Rate: 76.853 MB/s
2018-06-07 16:31:36,729 WARN 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:31:36,732 INFO 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.135 over PT0.003S. Rate: 25.618 MB/s
2018-06-07 16:31:36,848 WARN 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:31:36,849 INFO 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.133 over PT0.001S. Rate: 76.853 MB/s
2018-06-07 16:31:51,699 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (11 skipped) AgentAvroServlet: heartbeat processing stats: average=30ms, min=9ms, max=270ms.
2018-06-07 16:31:51,848 WARN 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:31:51,868 INFO 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.135 over PT0.019S. Rate: 4.045 MB/s
2018-06-07 16:31:51,877 WARN 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:31:51,939 INFO 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.133 over PT0.062S. Rate: 1.240 MB/s
2018-06-07 16:32:03,654 INFO JvmPauseMonitor:com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1501ms: no GCs detected.
2018-06-07 16:32:06,764 WARN 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:32:06,779 INFO 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.135 over PT0.015S. Rate: 5.124 MB/s
2018-06-07 16:32:06,865 WARN 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:32:06,866 INFO 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.133 over PT0.001S. Rate: 76.853 MB/s
2018-06-07 16:32:21,758 WARN 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:32:21,760 INFO 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.135 over PT0.002S. Rate: 38.426 MB/s
2018-06-07 16:32:21,871 WARN 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:32:21,872 INFO 829108619@scm-web-144:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.133 over PT0.001S. Rate: 76.853 MB/s
2018-06-07 16:32:36,776 WARN 1018229141@scm-web-146:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:32:36,779 INFO 1018229141@scm-web-146:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.135 over PT0.003S. Rate: 25.618 MB/s
2018-06-07 16:32:36,933 WARN 1018229141@scm-web-146:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:32:36,934 INFO 1018229141@scm-web-146:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.133 over PT0.001S. Rate: 76.853 MB/s
2018-06-07 16:32:51,729 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (11 skipped) AgentAvroServlet: heartbeat processing stats: average=30ms, min=9ms, max=270ms.
2018-06-07 16:32:51,797 WARN 1018229141@scm-web-146:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:32:51,801 INFO 1018229141@scm-web-146:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.135 over PT0.003S. Rate: 25.618 MB/s
2018-06-07 16:32:51,907 WARN 1018229141@scm-web-146:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:32:51,909 INFO 1018229141@scm-web-146:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.133 over PT0.001S. Rate: 76.853 MB/s
2018-06-07 16:33:06,767 WARN 1018229141@scm-web-146:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:33:06,768 INFO 1018229141@scm-web-146:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.135 over PT0.001S. Rate: 76.853 MB/s
2018-06-07 16:33:06,923 WARN 1018229141@scm-web-146:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent
2018-06-07 16:33:06,933 INFO 1018229141@scm-web-146:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.torrent (80586 bytes) to 192.168.74.133 over PT0.010S. Rate: 7.685 MB/s

 

What is the issue?

 

 

Announcements