https://community.cloudera.com/t5/Support-Questions/Difference-between-local-vs-yarn-cluster-vs-yarn-client-for/m-p/161246#M123625 <P>My Scenario</P><P>I would like to expose a java micro service &lt;Springboot appln&gt; which should eventually run a spark submit to yield the required results,typically as a on demand service</P><P>I have been allotted with 2 data nodes and 1 edge node for development, where this edge node has the micro services deployed. When I tried yarn-cluster, got an exception 'Detected yarn-cluster mode, but isn't running on a cluster. Deployment to YARN is not supported directly by SparkContext. Please use spark-submit.'</P><P>Help me to get an ideal way to deal with it. What should be the approach to be looked at? Since the service is on demand, I cannot deal with YARN Client to have more Main Class than one which is already used up for springboot starter.</P><P>Codes here</P><P>MicroServiceController.java:</P><P>@RequestMapping(value = "/transform", method = RequestMethod.POST, consumes = MediaType.APPLICATION_JSON_VALUE, produces = MediaType.APPLICATION_JSON_VALUE) </P><P>public String initiateTransformation(@RequestBody TransformationRequestVO requestVO){ </P><P>PublicationProcessor.run(); </P><P>return "SUCCESS"; </P><P>}</P><P>PublicationProcessor.java</P><P>public static void run() { </P><P>try{ </P><P>SparkConf sC = new SparkConf().setAppName("NPUB_TRANSFORMATION_US") .setMaster("yarn-clsuter") .set("spark.executor.instances", PropertyBundle.getConfigurationValue("spark.executor.instances")) .set("spark.executor.cores", PropertyBundle.getConfigurationValue("spark.executor.cores")) .set("spark.driver.memory",PropertyBundle.getConfigurationValue("spark.driver.memory")) .set("spark.executor.memory",PropertyBundle.getConfigurationValue("spark.executor.memory")) .set("spark.driver.maxResultSize", PropertyBundle.getConfigurationValue("spark.driver.maxResultSize")) .set("spark.network.timeout",PropertyBundle.getConfigurationValue("spark.network.timeout")); </P><P> JavaSparkContext jSC = new JavaSparkContext(sC); </P><P>sqlContext = new SQLContext(jSC);</P><P>processTransformation(); </P><P>}catch(Exception e){</P><P> System.out.println("REQUEST ABORTED..."+e.getMessage()); </P><P>} </P><P>}</P><P>,</P> Fri, 17 Mar 2017 02:43:16 GMT Faisalr_ahamed 2017-03-17T02:43:16Z