Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cloudera DC 711 stuck while distributing core binary

Cloudera DC 711 stuck while distributing core binary

New Contributor

I have tried quite a few times (cancel and restart, etc.) to get the Cloudera RunTime binary distributed. This activity started right after successful downloading, it stuck for > 1 day.

 

I am running with on AWS M5.xlarge with 100 GB disk space (still 80GB available). This is a dedicated instance, nothing else is running and there are no networking issues either. 

 

I am using Open JDK provided by Cloudera, not a custom selection. CDC7_distributing.jpg

8 REPLIES 8
Highlighted

Re: Cloudera DC 711 stuck while distributing core binary

Super Collaborator

@EnableMe Seems you are hitting a known issue. There is currently a bug.  Please try the following:

 

Click Parcel Repositories & Network Settings

Remove that URL, and replace it with:

https://archive.cloudera.com/cdh7/7.1.1.0/parcels/

 

This solution could be resolved or could change over time as newer versions of CM and Cloudera Runtime are released. 

 

There is already a discussion available on this thread .


Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Highlighted

Re: Cloudera DC 711 stuck while distributing core binary

New Contributor

Thanks for the suggestion.

 

I am already with the recent version (possibly the latest). I was previously at 703, that was quite a story

 

In other words, this solution is not applicable - attaching the screenshot as it appears.

(Disregard CDH 5 & 6 references - I attempted with 6.x as well, services do not come up).

 

June28_2020.jpg

Highlighted

Re: Cloudera DC 711 stuck while distributing core binary

Super Collaborator

@EnableMe Can you remove the CDH6 parcel link form this and then save changes. This should resolve the issue as the CDH6 repo you have entered needs a valid credentials comes under a support subscription in contrast with out new Licensing or you can say the Repo is behind the Paywall.

 

 


Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Highlighted

Re: Cloudera DC 711 stuck while distributing core binary

New Contributor

This did not help either. I took out all references to pre-7 (namely, 5 and 6). 

 

There are two major problems:

1) Cloudera 7111 not getting distributed - downloaded, 0% distributed.

2) Services not coming up - likely related to (1). 

 

Thank you.

Highlighted

Re: Cloudera DC 711 stuck while distributing core binary

Super Collaborator

Can you send me the CM server logs and the repo file you are using. The message clearly saying it's not able to authenticate so most probably there is issue with credentials. I am not facing this issue in my cluster. 


Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Highlighted

Re: Cloudera DC 711 stuck while distributing core binary

New Contributor

Sure, that's interesting.

 

This is a single instance so there is not as much credentials issue (Cloudera Manager points to the same AWS instance) - nothing else runs here except Cloudera stack.

The same AWS instance has no problem distributing 6.x (done so by default, I did not setup that way; that is a different topic) - in other words, I did not encounter the same issue with 6.x  - Pasting parcel file settings after this block (next section).

 

SCM logs here (there is no attachment feature available here? not sure).

cat -n /var/log/cloudera-scm-server/cloudera-scm-server.log
1 2020-07-01 23:20:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
2 2020-07-01 23:20:15,968 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (7 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
3 2020-07-01 23:20:36,310 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: Found files CDH-7.1.1-1.cdh7.1.1.p0.3266817-el7.parcel under /opt/cloudera/parcel-repo
4 2020-07-01 23:20:36,313 INFO ParcelUpdateService:com.cloudera.cmf.paywall.PaywallHelper: (6 skipped) License incomplete; unable to use it for authentication UUID=72ef3466-2863-41bb-b89c-11126931ef69, name=Trial License
5 2020-07-01 23:20:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
6 2020-07-01 23:20:37,086 INFO agentServer-95:com.cloudera.server.common.MonitoringThreadPool: agentServer: execution stats: average=9ms, min=7ms, max=45ms.
7 2020-07-01 23:20:37,086 INFO agentServer-95:com.cloudera.server.common.MonitoringThreadPool: agentServer: waiting in queue stats: average=0ms, min=0ms, max=3ms.
8 2020-07-01 23:20:46,369 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Reaped total of 0 deleted commands
9 2020-07-01 23:20:46,370 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Found no commands older than 2018-07-02T23:20:46.369Z to reap.
10 2020-07-01 23:20:46,370 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeScannerService: Reaped 0 requests.
11 2020-07-01 23:20:46,370 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeConfiguratorService: Reaped 0 requests.
12 2020-07-01 23:21:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
13 2020-07-01 23:21:19,476 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (6 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
14 2020-07-01 23:21:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
15 2020-07-01 23:21:50,727 INFO ScmActive-0:com.cloudera.server.cmf.components.ScmActive: (119 skipped) ScmActive completed successfully.
16 2020-07-01 23:22:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
17 2020-07-01 23:22:23,218 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (4 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
18 2020-07-01 23:22:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
19 2020-07-01 23:23:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
20 2020-07-01 23:23:25,253 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (8 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
21 2020-07-01 23:23:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
22 2020-07-01 23:23:50,532 WARN EventStorePublisherWithRetry-0:com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: Failed to publish event: SimpleEvent{attributes={CATEGORY=[AUDIT_EVENT], EVENTCODE=[EV_LOGIN_SUCCESS], SERVICE=[ClouderaManager], USER=[admin], SERVICE_TYPE=[ManagerServer], MESSAGE_CODES=[LOGIN_SUCCESS], SEVERITY=[INFORMATIONAL]}, content=User admin logged in successfully., timestamp=1593190812507} - 1 of 28 failure(s) in last 1827s
23 java.io.IOException: Error connecting to c711.afs.com/18.214.236.92:7184
24 at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:292)
25 at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:206)
26 at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:141)
27 at com.cloudera.cmf.event.publish.AvroEventStorePublishProxy.checkSpecificRequestor(AvroEventStorePublishProxy.java:135)
28 at com.cloudera.cmf.event.publish.AvroEventStorePublishProxy.publishEvent(AvroEventStorePublishProxy.java:210)
29 at com.cloudera.cmf.event.publish.EventStorePublisherWithRetry$PublishEventTask.run(EventStorePublisherWithRetry.java:242)
30 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
31 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
32 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
33 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
34 at java.lang.Thread.run(Thread.java:748)
35 2020-07-01 23:24:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
36 2020-07-01 23:24:25,344 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (4 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
37 2020-07-01 23:24:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
38 2020-07-01 23:25:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
39 2020-07-01 23:25:25,674 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (5 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
40 2020-07-01 23:25:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
41 2020-07-01 23:26:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
42 2020-07-01 23:26:26,612 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (9 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
43 2020-07-01 23:26:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
44 2020-07-01 23:27:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
45 2020-07-01 23:27:32,659 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (4 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
46 2020-07-01 23:27:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
47 2020-07-01 23:28:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
48 2020-07-01 23:28:32,944 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (7 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
49 2020-07-01 23:28:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
50 2020-07-01 23:29:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
51 2020-07-01 23:29:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
52 2020-07-01 23:29:43,861 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (7 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
53 2020-07-01 23:30:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
54 2020-07-01 23:30:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
55 2020-07-01 23:30:46,227 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (4 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
56 2020-07-01 23:30:46,227 INFO agentServer-95:com.cloudera.server.common.MonitoringThreadPool: agentServer: execution stats: average=9ms, min=7ms, max=45ms.
57 2020-07-01 23:30:46,227 INFO agentServer-95:com.cloudera.server.common.MonitoringThreadPool: agentServer: waiting in queue stats: average=0ms, min=0ms, max=3ms.
58 2020-07-01 23:30:46,373 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Reaped total of 0 deleted commands
59 2020-07-01 23:30:46,374 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Found no commands older than 2018-07-02T23:30:46.373Z to reap.
60 2020-07-01 23:30:46,374 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeScannerService: Reaped 0 requests.
61 2020-07-01 23:30:46,374 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeConfiguratorService: Reaped 0 requests.
62 2020-07-01 23:31:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
63 2020-07-01 23:31:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
64 2020-07-01 23:31:48,895 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (8 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
65 2020-07-01 23:32:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
66 2020-07-01 23:32:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
67 2020-07-01 23:32:48,994 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (4 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
68 2020-07-01 23:33:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
69 2020-07-01 23:33:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
70 2020-07-01 23:33:55,581 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (5 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
71 2020-07-01 23:34:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
72 2020-07-01 23:34:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
73 2020-07-01 23:35:06,057 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (8 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
74 2020-07-01 23:35:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
75 2020-07-01 23:35:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
76 2020-07-01 23:36:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
77 2020-07-01 23:36:12,160 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (5 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
78 2020-07-01 23:36:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
79 2020-07-01 23:37:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
80 2020-07-01 23:37:21,721 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (9 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
81 2020-07-01 23:37:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
82 2020-07-01 23:37:36,730 INFO CMMetricsForwarder-0:com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder: (19 skipped) Failed to send metrics.
83 java.lang.reflect.UndeclaredThrowableException
84 at com.sun.proxy.$Proxy240.writeMetrics(Unknown Source)
85 at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendWithAvro(ClouderaManagerMetricsForwarder.java:434)
86 at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendMetrics(ClouderaManagerMetricsForwarder.java:400)
87 at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.run(ClouderaManagerMetricsForwarder.java:181)
88 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
89 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
90 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
91 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
92 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
93 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
94 at java.lang.Thread.run(Thread.java:748)
95 Caused by: org.apache.avro.AvroRemoteException: java.net.SocketTimeoutException: connect timed out
96 at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:104)
97 ... 11 more
98 Caused by: java.net.SocketTimeoutException: connect timed out
99 at java.net.PlainSocketImpl.socketConnect(Native Method)
100 at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
101 at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
102 at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
103 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
104 at java.net.Socket.connect(Socket.java:607)
105 at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
106 at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
107 at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
108 at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
109 at sun.net.www.http.HttpClient.New(HttpClient.java:339)
110 at sun.net.www.http.HttpClient.New(HttpClient.java:357)
111 at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1226)
112 at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
113 at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
114 at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990)
115 at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1340)
116 at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1315)
117 at org.apache.avro.ipc.HttpTransceiver.writeBuffers(HttpTransceiver.java:77)
118 at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:58)
119 at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:72)
120 at org.apache.avro.ipc.Requestor.request(Requestor.java:147)
121 at org.apache.avro.ipc.Requestor.request(Requestor.java:101)
122 at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:88)
123 ... 11 more
124 2020-07-01 23:38:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
125 2020-07-01 23:38:27,816 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (6 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
126 2020-07-01 23:38:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
127 2020-07-01 23:39:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
128 2020-07-01 23:39:30,864 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (6 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
129 2020-07-01 23:39:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
130 2020-07-01 23:40:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
131 2020-07-01 23:40:33,289 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (8 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
132 2020-07-01 23:40:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
133 2020-07-01 23:40:46,376 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Reaped total of 0 deleted commands
134 2020-07-01 23:40:46,377 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Found no commands older than 2018-07-02T23:40:46.376Z to reap.
135 2020-07-01 23:40:46,378 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeScannerService: Reaped 0 requests.
136 2020-07-01 23:40:46,378 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeConfiguratorService: Reaped 0 requests.
137 2020-07-01 23:40:54,302 INFO agentServer-97:com.cloudera.server.common.MonitoringThreadPool: agentServer: execution stats: average=9ms, min=7ms, max=45ms.
138 2020-07-01 23:40:54,302 INFO agentServer-97:com.cloudera.server.common.MonitoringThreadPool: agentServer: waiting in queue stats: average=0ms, min=0ms, max=8ms.
139 2020-07-01 23:41:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
140 2020-07-01 23:41:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
141 2020-07-01 23:41:39,330 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (4 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
142 2020-07-01 23:42:06,417 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
143 2020-07-01 23:42:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
144 2020-07-01 23:42:52,894 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (9 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
145 2020-07-01 23:43:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
146 2020-07-01 23:43:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
147 2020-07-01 23:43:54,621 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (5 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
148 2020-07-01 23:44:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
149 2020-07-01 23:44:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
150 2020-07-01 23:44:58,891 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (5 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
151 2020-07-01 23:45:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer:

Highlighted

Re: Cloudera DC 711 stuck while distributing core binary

New Contributor
Highlighted

Re: Cloudera DC 711 stuck while distributing core binary

New Contributor

Earlier I was commenting

1) This is an AWS instance, nothing else runs. Cloudera Manager is also on the same host - not sure why credentials is a potential problem in this respect.

2) I had no problem getting 6.x (Cloudera automatically applied to this cluster though downloaded 7.11 binary for installation) distributed.

 

cat -n /var/log/cloudera-scm-server/cloudera-scm-server.log
1 2020-07-01 23:20:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
2 2020-07-01 23:20:15,968 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (7 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
3 2020-07-01 23:20:36,310 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: Found files CDH-7.1.1-1.cdh7.1.1.p0.3266817-el7.parcel under /opt/cloudera/parcel-repo
4 2020-07-01 23:20:36,313 INFO ParcelUpdateService:com.cloudera.cmf.paywall.PaywallHelper: (6 skipped) License incomplete; unable to use it for authentication UUID=72ef3466-2863-41bb-b89c-11126931ef69, name=Trial License
5 2020-07-01 23:20:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
6 2020-07-01 23:20:37,086 INFO agentServer-95:com.cloudera.server.common.MonitoringThreadPool: agentServer: execution stats: average=9ms, min=7ms, max=45ms.
7 2020-07-01 23:20:37,086 INFO agentServer-95:com.cloudera.server.common.MonitoringThreadPool: agentServer: waiting in queue stats: average=0ms, min=0ms, max=3ms.
8 2020-07-01 23:20:46,369 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Reaped total of 0 deleted commands
9 2020-07-01 23:20:46,370 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Found no commands older than 2018-07-02T23:20:46.369Z to reap.
10 2020-07-01 23:20:46,370 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeScannerService: Reaped 0 requests.
11 2020-07-01 23:20:46,370 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeConfiguratorService: Reaped 0 requests.
12 2020-07-01 23:21:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
13 2020-07-01 23:21:19,476 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (6 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
14 2020-07-01 23:21:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
15 2020-07-01 23:21:50,727 INFO ScmActive-0:com.cloudera.server.cmf.components.ScmActive: (119 skipped) ScmActive completed successfully.
16 2020-07-01 23:22:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
17 2020-07-01 23:22:23,218 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (4 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
18 2020-07-01 23:22:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
19 2020-07-01 23:23:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
20 2020-07-01 23:23:25,253 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (8 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
21 2020-07-01 23:23:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
22 2020-07-01 23:23:50,532 WARN EventStorePublisherWithRetry-0:com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: Failed to publish event: SimpleEvent{attributes={CATEGORY=[AUDIT_EVENT], EVENTCODE=[EV_LOGIN_SUCCESS], SERVICE=[ClouderaManager], USER=[admin], SERVICE_TYPE=[ManagerServer], MESSAGE_CODES=[LOGIN_SUCCESS], SEVERITY=[INFORMATIONAL]}, content=User admin logged in successfully., timestamp=1593190812507} - 1 of 28 failure(s) in last 1827s
23 java.io.IOException: Error connecting to c711.afs.com/18.214.236.92:7184
24 at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:292)
25 at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:206)
26 at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:141)
27 at com.cloudera.cmf.event.publish.AvroEventStorePublishProxy.checkSpecificRequestor(AvroEventStorePublishProxy.java:135)
28 at com.cloudera.cmf.event.publish.AvroEventStorePublishProxy.publishEvent(AvroEventStorePublishProxy.java:210)
29 at com.cloudera.cmf.event.publish.EventStorePublisherWithRetry$PublishEventTask.run(EventStorePublisherWithRetry.java:242)
30 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
31 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
32 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
33 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
34 at java.lang.Thread.run(Thread.java:748)
35 2020-07-01 23:24:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
36 2020-07-01 23:24:25,344 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (4 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
37 2020-07-01 23:24:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
38 2020-07-01 23:25:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
39 2020-07-01 23:25:25,674 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (5 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
40 2020-07-01 23:25:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
41 2020-07-01 23:26:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
42 2020-07-01 23:26:26,612 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (9 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
43 2020-07-01 23:26:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
44 2020-07-01 23:27:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
45 2020-07-01 23:27:32,659 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (4 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
46 2020-07-01 23:27:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
47 2020-07-01 23:28:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
48 2020-07-01 23:28:32,944 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (7 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
49 2020-07-01 23:28:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
50 2020-07-01 23:29:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
51 2020-07-01 23:29:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
52 2020-07-01 23:29:43,861 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (7 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
53 2020-07-01 23:30:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
54 2020-07-01 23:30:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
55 2020-07-01 23:30:46,227 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (4 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
56 2020-07-01 23:30:46,227 INFO agentServer-95:com.cloudera.server.common.MonitoringThreadPool: agentServer: execution stats: average=9ms, min=7ms, max=45ms.
57 2020-07-01 23:30:46,227 INFO agentServer-95:com.cloudera.server.common.MonitoringThreadPool: agentServer: waiting in queue stats: average=0ms, min=0ms, max=3ms.
58 2020-07-01 23:30:46,373 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Reaped total of 0 deleted commands
59 2020-07-01 23:30:46,374 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Found no commands older than 2018-07-02T23:30:46.373Z to reap.
60 2020-07-01 23:30:46,374 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeScannerService: Reaped 0 requests.
61 2020-07-01 23:30:46,374 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeConfiguratorService: Reaped 0 requests.
62 2020-07-01 23:31:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
63 2020-07-01 23:31:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
64 2020-07-01 23:31:48,895 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (8 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
65 2020-07-01 23:32:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
66 2020-07-01 23:32:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
67 2020-07-01 23:32:48,994 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (4 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
68 2020-07-01 23:33:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
69 2020-07-01 23:33:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
70 2020-07-01 23:33:55,581 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (5 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
71 2020-07-01 23:34:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
72 2020-07-01 23:34:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
73 2020-07-01 23:35:06,057 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (8 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
74 2020-07-01 23:35:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
75 2020-07-01 23:35:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
76 2020-07-01 23:36:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
77 2020-07-01 23:36:12,160 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (5 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
78 2020-07-01 23:36:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
79 2020-07-01 23:37:06,416 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Cleanup is started.
80 2020-07-01 23:37:21,721 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (9 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=7ms, max=44ms.
81 2020-07-01 23:37:36,415 INFO pool-201-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: Syncup is started.
82 2020-07-01 23:37:36,730 INFO CMMetricsForwarder-0:com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder: (19 skipped) Failed to send metrics.
83 java.lang.reflect.UndeclaredThrowableException
84 at com.sun.proxy.$Proxy240.writeMetrics(Unknown Source)
85 at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendWithAvro(ClouderaManagerMetricsForwarder.java:434)
86 at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendMetrics(ClouderaManagerMetricsForwarder.java:400)
87 at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.run(ClouderaManagerMetricsForwarder.java:181)
88 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
89 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
90 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
91 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
92 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
93 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
94 at java.lang.Thread.run(Thread.java:748)
95 Caused by: org.apache.avro.AvroRemoteException: java.net.SocketTimeoutException: connect timed out
96 at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:104)
97 ... 11 more
98 Caused by: java.net.SocketTimeoutException: connect timed out
99 at java.net.PlainSocketImpl.socketConnect(Native Method)
100 at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
101 at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
102 at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
103 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
104 at java.net.Socket.connect(Socket.java:607)
105 at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
106 at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
107 at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
108 at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
109 at sun.net.www.http.HttpClient.New(HttpClient.java:339)
110 at sun.net.www.http.HttpClient.New(HttpClient.java:357)
111 at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1226)
112 at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
113 at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
114 at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990)
115 at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1340)
116 at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1315)
117 at org.apache.avro.ipc.HttpTransceiver.writeBuffers(HttpTransceiver.java:77)
118 at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:58)
119 at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:72)
120 at org.apache.avro.ipc.Requestor.request(Requestor.java:147)
121 at org.apache.avro.ipc.Requestor.request(Requestor.java:101)
122 at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:88)
123 ... 11 more

Don't have an account?
Coming from Hortonworks? Activate your account here