We are trying to Setup HA for our 2 standalone Livy Servers.
Current Setup: We have setup 2 standalone Livy servers and we are able to submit spark jobs on the cluster using the individual livy instance URL's.
End State: We are trying to setup an Nginx Proxy Load Balancer with a common livy URL, which will enable automatic fail-over when one of the Livy server crashes.
Does anyone have idea on how to setup and configure Livy servers using load balancer.
Actually, putting multiple servers behind a load balancer NOT works right now due to this bug: https://issues.apache.org/jira/browse/LIVY-541
In contrast, for deployments that require high availability, Livy supports session recovery, which ensures that a Spark cluster remains available if the Livy server fails. After a restart, the Livy server can connect to existing sessions and roll back to the state before failing.