Batch job not completing in Livy under Yarn mode

linchan — Fri, 16 Sep 2022 09:42:28 GMT

I noticed the following submiting batch job (jar) in Yarn mode. I am able to submit a jar and seem it is being issued to spark-submit correctly (the logged spark-submit command runs fine without livy context).

However, after it is done, the status of the batch is as below:

{"id":0,"state":"running","log":["\t diagnostics: N/A","\t ApplicationMaster host: 10.0.0.167","\t ApplicationMaster RPC port: 0","\t queue: default","\t start time: 1443732089755","\t final status: SUCCEEDED","\t tracking URL: http://servername:8088/proxy/application_1443668789912_0010/","\t user: root","15/10/01 20:42:31 INFO Utils: Shutdown hook called","15/10/01 20:42:31 INFO Utils: Deleting directory /tmp/spark-0343eaf9-c8e6-4f85-9200-6d3366851b1e"]}

Livy still thinks the state is running but the log seems to indicate that the job is finished and SUCCEEDED. Any idea what might be happening here?

Also once batch is in that state and I issue a delete, it will hang for a bit and then give the following timeout response.

< HTTP/1.1 504 Gateway Timeout
< Date: Thu, 01 Oct 2015 20:54:58 GMT
< Content-Type: application/json; charset=UTF-8
< Content-Length: 15
< Server: Jetty(9.2.z-SNAPSHOT)
<
* Connection #0 to host headnode0 left intact
* Closing connection #0
Gateway timeout

Thanks

Lin

Re: Batch job not completing in Livy under Yarn mode

linchan — Fri, 02 Oct 2015 17:18:33 GMT

Never mind, found the problem. When I setup the HADOOP_CONF_DIR, I didn't put a trailing /. As a result, Livy is not finding the right Yarn port to talk to and get the application status.

It would be nice if the code can handle both the case where trailing / is present or not present.

Thanks

Lin

Re: Batch job not completing in Livy under Yarn mode

cjervis — Fri, 02 Oct 2015 17:23:19 GMT

I am happy to see you solved your issue. Feel free to mark it as a solution in case others have a similar problem in the future.

question Re: Batch job not completing in Livy under Yarn mode in Archives of Support Questions (Read Only)

Batch job not completing in Livy under Yarn mode

Re: Batch job not completing in Livy under Yarn mode

Re: Batch job not completing in Livy under Yarn mode