I am building an HDP 2.6 cluster with separate VLANS for our presentation, functions and application. There is a firewall between these VLANS. Livy runs on the same machine as Zeppelin in the Presentation VLAN. The problem is that Livy uses random port numbers for the RFC launcher port. I cannot combine this behavior with the firewall because I would need to open every port between our VLANS which will create a security threat.
A logical solution would be to put Livy in the Application VLAN but this would create a security threat because encrypted traffic between Livy and Zeppelin is not supported.
Another solution would be to place Livy & Zeppelin in the application VLAN but this would create a different security threat, allowing end-users to enter our application layer.
Any solution for this is appreciated.
By default Livy runs on port 8998 (which can be changed with the
livy.server.port config option).
You have the option to put a proxy in front of Livy. That proxy will always use the same port. Think this as a load balancer. Ideally, you would use Knox configured for Livy Server. However, some work is still to be done: https://issues.apache.org/jira/browse/KNOX-843. Until then, HAProxy could help. HAProxy would be in the same VLAN with Livy and would expose a single IP:port for entry.
My response was addressing the concern of a single entry point then redirected to a randomly generated port for the RPC Server, port range that can be added in firewall exclusion list with one liner. I understood that part, just did not address it :)
Regarding those randomly generated ports, there are a few JIRA tickets for them:
Look at the first link for some possible solutions, including a patch.
It seems that build 1335 handles this situation: https://travis-ci.org/cloudera/livy/builds/237064535
Personally, I don't consider Livy as part of the presentation. It should be on the other side of the fence, but it can't because, as you stated, it lacks encrypted traffic, between Zeppelin and Livy.