I'm a newbie of spark, and curently stryggling to run first spark-submit job. I initially configured spark on yarn on 4 node cloudera cluster, but if i do so, i could not see master and worker roles. Furthermore i cannot see any instances of spark except History server in cloudera manager UI.  And i'm not sure what to pass for --master argument while doing spark-submit. If i do yarn-cluster nothing happend and job dies automatically, but if i do spark://master-node:7077 there is some error about cannot finding workers. I have to do start-all.sh manually, and even if i do so, it canot find workers. What actually am i doing wrong here? Is is spark-configuratioon issue with spark on yarn or something else? ... View more

Initially, i set up 2 machines (Ubuntu 12.04, x64) on vSphere server. The name and ip of these two machines were host ip vm-cluster-node1 10.211.55.100 vm-cluster-node2 10.211.55.101 I have installed cloudera manager in vm-cluster-node1. Then i cloned second one (vm-cluster-node2) to create 2 more hosts, and changed the ip and names as: host ip vm-cluster-node3 10.211.55.102 vm-cluster-node4 10.211.55.103 But the problem is, when i add these all 4 machines from cloudera, no matter how many times i try, i can only see two machines in hosts tab. later i realized that, if i refresh the web-page, i can see 2 machines only, but the second machine is switched between vm-cluster-node2, vm-cluster-node3 and vm-cluster-node4. to illustrate, i have included images to make things clear.     So, as far i've understood, the cloudera manager is not able to resolve the hosts cloned from same source as different machines. Even though the host-names and IPs have been changed. So is there anything that is common in these machines and this problem is occurring? ... View more