Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

​How to prevent Livy to use random ports?

​How to prevent Livy to use random ports?

I am building an HDP 2.6 cluster with separate VLANS for our presentation, functions and application. There is a firewall between these VLANS. Livy runs on the same machine as Zeppelin in the Presentation VLAN. The problem is that Livy uses random port numbers for the RFC launcher port. I cannot combine this behavior with the firewall because I would need to open every port between our VLANS which will create a security threat.

A logical solution would be to put Livy in the Application VLAN but this would create a security threat because encrypted traffic between Livy and Zeppelin is not supported.

Another solution would be to place Livy & Zeppelin in the application VLAN but this would create a different security threat, allowing end-users to enter our application layer.

Any solution for this is appreciated.

6 REPLIES 6

Re: ​How to prevent Livy to use random ports?

@Daniel Kozlowski

By default Livy runs on port 8998 (which can be changed with the livy.server.port config option).

You have the option to put a proxy in front of Livy. That proxy will always use the same port. Think this as a load balancer. Ideally, you would use Knox configured for Livy Server. However, some work is still to be done: https://issues.apache.org/jira/browse/KNOX-843. Until then, HAProxy could help. HAProxy would be in the same VLAN with Livy and would expose a single IP:port for entry.

Re: ​How to prevent Livy to use random ports?

My response was addressing the concern of a single entry point then redirected to a randomly generated port for the RPC Server, port range that can be added in firewall exclusion list with one liner. I understood that part, just did not address it :)

Regarding those randomly generated ports, there are a few JIRA tickets for them:

https://issues.cloudera.org/browse/LIVY-267

https://issues.cloudera.org/browse/LIVY-337

Look at the first link for some possible solutions, including a patch.

It seems that build 1335 handles this situation: https://travis-ci.org/cloudera/livy/builds/237064535

Personally, I don't consider Livy as part of the presentation. It should be on the other side of the fence, but it can't because, as you stated, it lacks encrypted traffic, between Zeppelin and Livy.

Re: ​How to prevent Livy to use random ports?

Thanks @Constantin Stanca for your comments on this

Re: ​How to prevent Livy to use random ports?

Expert Contributor

This PR is still in review https://github.com/cloudera/livy/pull/334/

Re: ​How to prevent Livy to use random ports?

Hi @jzhang

Thanks for your comment.

Does it mean we cannot currently prevent Livy to use random ports?

Re: ​How to prevent Livy to use random ports?

Expert Contributor

That's correct