A vcore is a virtual core. You can define it however you want. You could, as an example, define that a vcore is the processing power that is delivered by a 1GHz thread core. A 3GHz core would than be comparable to 3 vcores in the node manager. Your container request then needs to use multiple vcores which handles the difference in speed.   Not a lot of clusters do this due to the administrative overhead and the fact that if the end users do not use the vcore correctly it can overload the faster machines.   Wilfred ... View more

CDH 5.3 does not come with Spark 1.5. You are running an unsupported cluster. Please be aware of that.   Weight has nothing to do with the pre-emption. It is a common misunderstanding. The weight is just to decide which queue gets a higher priority during the scheduling cycle. So if I have queues with the weights 3:1:1 then from every 10 schedule attempts 6 will go to the queue with weight 3 and 2 attempts will be for each queue with weight 1, totalling 10 attempts.   Minimum share preemption works only if you have the minimum and maximum shares for a queue set. Make sure you have that. The fair share of a queue is calculated based on the demand in the queue (i.e. the applications that are running in the queue). You thus might not be hitting the fair share preemption threshold....   Wilfred   ... View more

To rule out that we have a custom jar issue can you run the pi example to make sure that the cluster is (not) setup correctly? We have documented how to run a spark application, with the example in our docs.   The error that you show points to a classpath error and you can not find the Spark classes on your class path.   WIlfred ... View more