My Scenario
I would like to expose a java micro service <Springboot appln> which should eventually run a spark submit to yield the required results,typically as a on demand service
I have been allotted with 2 data nodes and 1 edge node for development, where this edge node has the micro services deployed. When I tried yarn-cluster, got an exception 'Detected yarn-cluster mode, but isn't running on a cluster. Deployment to YARN is not supported directly by SparkContext. Please use spark-submit.'
Help me to get an ideal way to deal with it. What should be the approach to be looked at? Since the service is on demand, I cannot deal with YARN Client to have more Main Class than one which is already used up for springboot starter.
Codes here
MicroServiceController.java:
@RequestMapping(value = "/transform", method = RequestMethod.POST, consumes = MediaType.APPLICATION_JSON_VALUE, produces = MediaType.APPLICATION_JSON_VALUE)
public String initiateTransformation(@RequestBody TransformationRequestVO requestVO){
PublicationProcessor.run();
return "SUCCESS";
}
PublicationProcessor.java
public static void run() {
try{
SparkConf sC = new SparkConf().setAppName("NPUB_TRANSFORMATION_US")
.setMaster("yarn-clsuter")
.set("spark.executor.instances", PropertyBundle.getConfigurationValue("spark.executor.instances"))
.set("spark.executor.cores", PropertyBundle.getConfigurationValue("spark.executor.cores"))
.set("spark.driver.memory",PropertyBundle.getConfigurationValue("spark.driver.memory"))
.set("spark.executor.memory",PropertyBundle.getConfigurationValue("spark.executor.memory"))
.set("spark.driver.maxResultSize", PropertyBundle.getConfigurationValue("spark.driver.maxResultSize"))
.set("spark.network.timeout",PropertyBundle.getConfigurationValue("spark.network.timeout"));
JavaSparkContext jSC = new JavaSparkContext(sC);
sqlContext = new SQLContext(jSC);
processTransformation();
}catch(Exception e){
System.out.println("REQUEST ABORTED..."+e.getMessage());
}
}
,
