Livy rest api with web app

[#1]

Hi,

I have a requirement to train a predictive machine learning model from web application. User can change tuning parameters/ algorithm from User Interface which goes to Spark job and returns regression metric.

Right now, I have developed a web application REST API with Jersey and Jetty.

I have a Spark machine learning application which implements Livy job. I am referring this jar from web application.

The flow is as follows. User changes tuning parameter/ algorithm from UI -> Jersey API calls Livy client with parameters -> Livy job runs in YARN and returns metric results -> Jersey returns metric results to UI.

Am I doing it correct? Is there any other way that I can interact with SPARK from web app? I am creating LivyClient singleton object and I am using this object for submitting job throughout the application. The code was working fine. I am getting the below exception now. Do I have to create LivyClient object and upload jar (Spark machine learning application with Livy) each and every time?

javax.servlet.ServletException: java.util.concurrent.ExecutionException: java.io.IOException: Not Found: "Session '339' not found."

[#2 · Accepted solution]

The flow seems right. Thats a good use case for livy. Assuming it goes YourApp->Livy->Spark and back.

You will need to look at Livy client logs or livy logs for session id 339. Seems like the client is asking for a session (livy spark job) that does not exist anymore. Could have been not started and abandoned or lost.

[#3]

@Bikas, Is this correct to create LivyClient object during application start up and use that object for all the API requests to submit jar?