Did a clean installaiton of cdh 5.2 ( on AWS , 7 nodes doing a complete install , accepting all components ) , Spark would only install a history server ( and allow me to add gateway servers ) there seems to be no way to add executor nodes. In CDH 5.1 spark executors were installed on the data nodes by default , I would expect the same behaviour in 5.2. Is there a work around
In 5.2, Spark on YARN is now the default. This uses YARN NodeManagers for execution and does not require assigning Spark executor nodes.
Spark on YARN is strongly preferred over the old Spark, which is why we made this change. If you'd really like to use the old Spark, then you can add it to your cluster after the initial setup, just like how you add any other service to a cluster. You can't add the old standalone Spark in the cluster setup wizard.
If Spark on YARN, but I can't find the job status from YARN Manager. No history in YARN manager is generated. Actually the job ran well.
Any ideas? how can we get the status of all Spark jobs ?
this is my history server page. No records there..
in master machine ran the following spark submit command
it works in CDH5.1, but when I re-install to CDH5.2, the spark history gone..
Not upgrade, I installed CDH5.2 in a clean environment.
Cluster MetricsApps Submitted Apps Pending Apps Running Apps Completed Containers Running Memory Used Memory Total Memory Reserved VCores Used VCores Total VCores Reserved Active Nodes Decommissioned Nodes Lost Nodes Unhealthy Nodes Rebooted Nodes
IDUserNameApplication TypeQueueStartTimeFinishTimeStateFinalStatusProgressTracking UI
Showing 0 to 0 of 0 entries
this is YARN UI, no executed or executing jobs here..
I will check logs ..