Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Livy rest api with web app

Solved Go to solution

Livy rest api with web app

Hi,

I have a requirement to train a predictive machine learning model from web application. User can change tuning parameters/ algorithm from User Interface which goes to Spark job and returns regression metric.

Right now, I have developed a web application REST API with Jersey and Jetty.

I have a Spark machine learning application which implements Livy job. I am referring this jar from web application.

The flow is as follows. User changes tuning parameter/ algorithm from UI -> Jersey API calls Livy client with parameters -> Livy job runs in YARN and returns metric results -> Jersey returns metric results to UI.

Am I doing it correct? Is there any other way that I can interact with SPARK from web app? I am creating LivyClient singleton object and I am using this object for submitting job throughout the application. The code was working fine. I am getting the below exception now. Do I have to create LivyClient object and upload jar (Spark machine learning application with Livy) each and every time?

javax.servlet.ServletException: java.util.concurrent.ExecutionException: java.io.IOException: Not Found: "Session '339' not found."

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Livy rest api with web app

Expert Contributor

The flow seems right. Thats a good use case for livy. Assuming it goes YourApp->Livy->Spark and back.

You will need to look at Livy client logs or livy logs for session id 339. Seems like the client is asking for a session (livy spark job) that does not exist anymore. Could have been not started and abandoned or lost.

2 REPLIES 2
Highlighted

Re: Livy rest api with web app

Expert Contributor

The flow seems right. Thats a good use case for livy. Assuming it goes YourApp->Livy->Spark and back.

You will need to look at Livy client logs or livy logs for session id 339. Seems like the client is asking for a session (livy spark job) that does not exist anymore. Could have been not started and abandoned or lost.

Re: Livy rest api with web app

@Bikas, Is this correct to create LivyClient object during application start up and use that object for all the API requests to submit jar?

Don't have an account?
Coming from Hortonworks? Activate your account here