connection refused on spark client code connections

[#1]

I have a 5 node cluster installed using Cloudera manager 5.0.1.

I can run spark-shell on each of the nodes with no problem. Then I tried connecting using spark client code. I am not able to connect, every node whether master or worker dumps out logs like this.

akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused:

each node shows a diffeent port number. When I test telnet to those ports, I always get a connection refused. any one knows how these ports are set, whether fail to telnet means trouble? in general how to approach this problem?

[#2]

it seems firewall problem; try this command on each machine to sure it:

service iptables stop

[#3]

Actuallty, it is not an issue with firewall. apparently no firewall was set.

I am able to run the same scala code using spark-shell. What I noticed is that when spark-shell connects to workers, it uses all kinds of port numbers but not 7000. inital akka tcp seems to be connecting to spark master on port 55375.

However, when I run the sbt code, it always tries to connect to spark master using 7000. is there a way to set this port number in the scala code.

[#4]

set the port number in spark-en.sh file, SPARK-WORKER-PORT.