Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Livy HA Setup : Using 2 Standalone Livy Servers & Proxy Load Balancer

avatar
New Contributor

Hi,

We are trying to Setup HA for our 2 standalone Livy Servers.

Current Setup: We have setup 2 standalone Livy servers and we are able to submit spark jobs on the cluster using the individual livy instance URL's.

End State: We are trying to setup an Nginx Proxy Load Balancer with a common livy URL, which will enable automatic fail-over when one of the Livy server crashes.

Does anyone have idea on how to setup and configure Livy servers using load balancer.

1 REPLY 1

avatar
Cloudera Employee

Hi George,
Actually, putting multiple servers behind a load balancer NOT works right now due to this bug: https://issues.apache.org/jira/browse/LIVY-541
In contrast, for deployments that require high availability, Livy supports session recovery, which ensures that a Spark cluster remains available if the Livy server fails. After a restart, the Livy server can connect to existing sessions and roll back to the state before failing.