Livy HA Setup : Using 2 Standalone Livy Servers & Proxy Load Balancer

[#1]

Hi,

We are trying to Setup HA for our 2 standalone Livy Servers.

Current Setup: We have setup 2 standalone Livy servers and we are able to submit spark jobs on the cluster using the individual livy instance URL's.

End State: We are trying to setup an Nginx Proxy Load Balancer with a common livy URL, which will enable automatic fail-over when one of the Livy server crashes.

Does anyone have idea on how to setup and configure Livy servers using load balancer.

[#2]

Hi George,
Actually, putting multiple servers behind a load balancer NOT works right now due to this bug: https://issues.apache.org/jira/browse/LIVY-541
In contrast, for deployments that require high availability, Livy supports session recovery, which ensures that a Spark cluster remains available if the Livy server fails. After a restart, the Livy server can connect to existing sessions and roll back to the state before failing.