Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

connection refused on spark client code connections

connection refused on spark client code connections

Explorer

I have a 5 node cluster installed using Cloudera manager 5.0.1.

 

I can run spark-shell on each of the nodes with no problem. Then I tried connecting using spark client code. I am not able to connect, every node whether master or worker dumps out logs like this.

 

akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused:

 

each node shows a diffeent port number. When I test telnet to those ports, I always get a connection refused. any one knows how these ports are set, whether fail to telnet means trouble? in general how to approach this problem?

 

3 REPLIES 3

Re: connection refused on spark client code connections

New Contributor

it seems firewall problem; try this command on each machine to sure it:

 

service iptables stop

Re: connection refused on spark client code connections

Explorer

Actuallty, it is not an issue with firewall. apparently no firewall was set.

 

I am able to run the same scala code using spark-shell. What I noticed is that when spark-shell connects to workers, it uses all kinds of port numbers but not 7000. inital akka tcp seems to be connecting to spark master on port 55375.

 

However, when I run the sbt code, it always tries to connect to spark master using 7000. is there a way to set this port number in the scala code.

 

Re: connection refused on spark client code connections

New Contributor

set the port number in spark-en.sh file, SPARK-WORKER-PORT.

Don't have an account?
Coming from Hortonworks? Activate your account here