Thanks for the help,
I follow the instruction and get this error:
Error: Cannot load main class from JAR: file:/var/lib/hadoop-hdfs/class
Can you give any advise ?
That sounds like a bad command line. I don't see that path in the instructions either. Check that you are following the instructions for 5.2 in the previous link.
Thanks for your reply sowen,
I'm just trying with another link: https://spark.apache.org/docs/1.1.0/running-on-yarn.html and it work.
I got the result:
14:52:41 INFO Client: Application report from ResourceManager:
application identifier: application_1416365742014_0003
Problem is i can't find where the result of Pi is like when we run Pi example on Hadoop (it'll print the resutl 3.14333...) , where can i find it ?
Yes, in that example you are clearly running on YARN. So you see it in the history, right?
It looks like the example uses yarn-cluster mode, which means the driver was launched on YARN, not locally. The output will be on the YARN container that had the driver.
Try yarn-client instead to make your local process the driver and it should print the result on your console.
Thanks again owen,
The example go well, i can see the Pi result now, still got some error :
WARN YarnClientClusterScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(03slave.mabu.com,42930) not found
WARN ConnectionManager: All connections not cleaned up.
Don't know if it's because of the poor connection or the amount of RAM on my cluster, but this is still a good start for me anyway.
By the way, do you know where i can find more information about Spark system ( how it work, it's operation, when to user yarn-clsuter/yarn-client ...).
Thanks alot !