Unable to retrieve remote parcel repository manifest java.util.concurrent.ExecutionException Error

[#1]

Hello All,

Agent on ont of the nodes on cluster is not running. status of agent is as below.

sudo service cloudera-scm-agent status
cloudera-scm-agent dead but pid file exists

When I checked the logs for that host , I came across below error.

(5 skipped) Unable to retrieve remote parcel repository manifest
java.util.concurrent.ExecutionException: java.net.ConnectException: http://archive.cloudera.com/spark/parcels/latest/manifest.json
	at com.ning.http.client.providers.netty.NettyResponseFuture.abort(NettyResponseFuture.java:297)
	at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:104)
	at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:399)
	at org.jboss.netty.channel.DefaultChannelFuture.addListener(DefaultChannelFuture.java:145)
	at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.doConnect(NettyAsyncHttpProvider.java:1041)
	at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.execute(NettyAsyncHttpProvider.java:858)
	at com.ning.http.client.AsyncHttpClient.executeRequest(AsyncHttpClient.java:512)
	at com.ning.http.client.AsyncHttpClient$BoundRequestBuilder.execute(AsyncHttpClient.java:234)
	at com.cloudera.parcel.components.ParcelDownloaderImpl.getRepositoryInfoFuture(ParcelDownloaderImpl.java:530)
	at com.cloudera.parcel.components.ParcelDownloaderImpl.getRepositoryInfo(ParcelDownloaderImpl.java:488)
	at com.cloudera.parcel.components.ParcelDownloaderImpl.syncRemoteRepos(ParcelDownloaderImpl.java:342)
	at com.cloudera.parcel.components.ParcelDownloaderImpl$1.run(ParcelDownloaderImpl.java:412)
	at com.cloudera.parcel.components.ParcelDownloaderImpl$1.run(ParcelDownloaderImpl.java:407)
	at com.cloudera.cmf.persist.ReadWriteDatabaseTaskCallable.call(ReadWriteDatabaseTaskCallable.java:36)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: http://archive.cloudera.com/spark/parcels/latest/manifest.json
	at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:100)
	... 16 more
Caused by: java.nio.channels.UnresolvedAddressException
	at sun.nio.ch.Net.checkAddress(Net.java:127)
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:644)
	at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:139)
	at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:102)
	at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:55)
	at org.jboss.netty.handler.codec.http.HttpClientCodec.handleDownstream(HttpClientCodec.java:97)
	at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:108)
	at org.jboss.netty.channel.Channels.connect(Channels.java:642)
	at org.jboss.netty.channel.AbstractChannel.connect(AbstractChannel.java:204)
	at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:230)
	at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:183)
	at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.doConnect(NettyAsyncHttpProvider.java:999)
	... 13 more

Also, under hosts parcels section I see error as

• Error when distributing to LinuxUL.ad.infosys.com : Host is in bad health. for two parcels CDH 5 and KAFKA.

We have used 

http://www.cloudera.com/documentation/enterprise/5-4-x/topics/cm_ig_install_path_c.html for installation of cloudera cluster. We are using 3-node CDH 5.4 cluster.

Can you please help me with this issue?

Thanks,

Priya

[#2]

Hi,

There are also other errors that I came across in the log as below.

{"name":"HeapDumpOnOutOfMemoryError","origin":"VM_CREATION","value":"true","writeable":true},


Error accessing http://LinuxUL.ad.infosys.com:9000/process/2718-hdfs-DATANODE/files/logs/stdout.log
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200

a:745)
2017-08-29 05:23:53,995 ERROR DataArchiver-22:com.cloudera.cmf.command.datacollection.AgentLogArchiver: Failed to collect agent log from host LinuxUL.ad.infosys.com
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at


RROR DataArchiver-19:com.cloudera.cmf.command.datacollection.ProcessStdoutStderrArchiver: Error accessing http://LinuxUL.ad.infosys.com:9000/process/2720-hdfs-JOURNALNODE/files/logs/stdout.log
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSo


5)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2017-08-29 05:23:53,995 ERROR DataArchiver-12:com.cloudera.cmf.command.datacollection.ProcessStdoutStderrArchiver: Error accessing http://LinuxUL.ad.infosys.com:9000/process/2828-zookeeper-server/files/logs/stdout.log
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at


ERROR DataArchiver-17:com.cloudera.cmf.command.datacollection.FullTextLogArchiver: Error collecting log /home/log/hadoop-0.20-mapreduce/hadoop-cmf-mapreduce-TASKTRACKER-LinuxUL.ad.infosys.com.log.out
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
a

ERROR DataArchiver-24:com.cloudera.cmf.command.datacollection.ProcessStdoutStderrArchiver: Error accessing http://LinuxUL.ad.infosys.com:9000/process/2718-hdfs-DATANODE/files/logs/stderr.log
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native

ERROR DataArchiver-21:com.cloudera.cmf.command.datacollection.ProcessStdoutStderrArchiver: Error accessing http://LinuxUL.ad.infosys.com:9000/process/2756-mapreduce-TASKTRACKER/files/logs/stderr.log
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at

ERROR DataArchiver-2:com.cloudera.cmf.command.datacollection.ProcessStdoutStderrArchiver: Error accessing http://LinuxUL.ad.infosys.com:9000/process/2867-hbase-REGIONSERVER/files/logs/stdout.log
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.Abstr

ERROR DataArchiver-16:com.cloudera.cmf.command.datacollection.ProcessStdoutStderrArchiver: Error accessing http://LinuxUL.ad.infosys.com:9000/process/2905-yarn-NODEMANAGER/files/logs/stderr.log
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)


ERROR DataArchiver-15:com.cloudera.cmf.command.datacollection.ProcessStdoutStderrArchiver: Error accessing http://LinuxUL.ad.infosys.com:9000/process/2778-kafka-KAFKA_BROKER/files/logs/stdout.log
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at




Please help.

Thanks,
Priya

[#3]

/etc/hosts file look fine. Below is the output.
[root@LinuxUL etc]# cat hosts
127.0.0.1 localhost

10.68.200.34 LinuxUL.ad.infosys.com LinuxUL
10.68.200.152 linux152.ad.infosys.com linux152
10.68.200.170 linux170.ad.infosys.com linux170
172.21.5.224 nfrsat01.ad.infosys.com nfrsat01
10.67.200.77 blrsat06.ad.infosys.com blrsat06
[root@LinuxUL etc]# netstat -ltnp | grep 7182
[root@LinuxUL etc]# netstat -ltnp | grep 9000
[root@LinuxUL etc]# netstat -ltnp | grep 9001
tcp 0 0 127.0.0.1:19001 0.0.0.0:* LISTEN 8086/python