I wont be surprised if these have been asked before, but I am trying with CDH manager 5.4 on AWS to install Spark.
first up, the install sometimes hangs at "deploying client config" although it sucessfuly installs Oozie and several other services.
I downloaded the Spark parcel.
Set up the master and worker ( for worker I specfied the master as one of the workers, not sure about that).
See page - Installing Spark with Cloudera Manager
Now how do I know from the UI that my master and workers are up ?
When I run spark shell it gets into an infinite loop INFO Client: Application report for application_1430414416377_0004 (state: ACCEPTED)
I tried spark submit, but the file spark-examples_version.jar is not there on disk, on either master or worker.
If someone can make notes about missing steps/errors, that would be great.
Not sure why the deploy config step is hanging there. I'm not sure what you mean about an infinite loop. If your executors are accepted then you're running. The examples JAR is shipped, in the parcels directory. I don't know where you're looking for it.
thanks for replying.
#deploy config step is hanging there ->
saw the log as directed by another blog post, from my memory the log said something like "persistence test passed" and then did nothing. I cannot remember if its was the same error every time. ( I tried the install several times).
#I'm not sure what you mean about an infinite loop ->
I mean spark shell keeps on printing that message, and does not give me a prompt. I had read some other blog post which suggested to raise the log level to higher than INFO.
#The examples JAR is shipped, in the parcels directory. I don't know where you're looking for it. -->
find / -name spark-examples_version.jar
I copy pasted from the website where the name of the jar is wrong.
#If your executors are accepted then you're running. -->
can I see in the UI who is the master and who are the workers?
When specifying the worker IPs, I have included the master as a worker as well. Is that valid?