Knox + Zeppelin websocket problems

[#1]

I am attempting to configure Zeppelin behind Knox gateway running within an HDP 3.1.0 cluster with HDF 3.3.1 installed, all administered by Ambari 2.7.3. I am following the docs here: https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/configuring-proxy-knox/sec_configuring_prox.... As per the docs, I have set gateway.websocket.feature.enabled=true in Ambari's Knox configs, and when I access Zeppelin at the proxied URL I get the UI, saying the websocket is disconnected, and in the knox logs I see this:

2019-01-08 20:26:46,217 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - Received request: GET /zeppelin/ws
2019-01-08 20:26:46,217 DEBUG knox.gateway (UrlRewriteProcessor.java:rewrite(163)) - Rewrote URL: https://KNOXHOST:8443/gateway/CLUSTERNAME/zeppelin/ws, direction: IN via explicit rule: ZEPPELINWS/zeppelin/ws/inbound to URL: ws://ZEPPELINHOST:9995/ws/ws
2019-01-08 20:26:46,218 DEBUG knox.gateway (DefaultDispatch.java:executeOutboundRequest(121)) - Dispatch request: GET ws://ZEPPELINHOST:9995/ws/ws
2019-01-08 20:26:46,219 WARN  knox.gateway (DefaultDispatch.java:executeOutboundRequest(147)) - Connection exception dispatching request: ws://ZEPPELINHOST:9995/ws/ws org.apache.http.conn.UnsupportedSchemeException: ws protocol is not supported

Seeing the extra "ws" appended to the redirect URL, I add a pre-shortened URL to empty ZEPPELINWS service definition in the knox descriptor, to override the bad one:

ws://ZEPPELINHOST:9995

Then save descriptor/redeploy the topology and also restart knox, then I see essentially the same error, but with the correct URL now in the knox logs:

2019-01-08 20:41:12,796 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - Received request: GET /zeppelin/ws
2019-01-08 20:41:12,798 DEBUG knox.gateway (UrlRewriteProcessor.java:rewrite(163)) - Rewrote URL: https://KNOXHOST:8443/gateway/CLUSTERNAME/zeppelin/ws, direction: IN via explicit rule: ZEPPELINWS/zeppelin/ws/inbound to URL: ws://ZEPPELINHOST:9995/ws
2019-01-08 20:41:12,800 DEBUG knox.gateway (DefaultDispatch.java:executeOutboundRequest(121)) - Dispatch request: GET ws://ZEPPELINHOST:9995/ws
2019-01-08 20:41:12,802 WARN  knox.gateway (DefaultDispatch.java:executeOutboundRequest(147)) - Connection exception dispatching request: ws://ZEPPELINHOST:9995/ws org.apache.http.conn.UnsupportedSchemeException: ws protocol is not supported

Any ideas what my problem might be? I have been banging my head on this for almost a full day and not making any progress, so thanks in advance for any assistance...

-Doug

[#2 · Accepted solution]

Are you accessing Knox via a load balancer? I've seen something similar where there was a load balancer in front of Knox that didn't support websockets.

The documentation you are following might be incorrect too. The role name I think should be ZEPPELINWS based on https://knox.apache.org/books/knox-1-2-0/user-guide.html#Zeppelin+UI

[#3]

Aha, that might just be it... I do have NGINX in front of Knox, but just proxying it, not load-balancing. I bet you are right, and if I configure NGINX for web sockets it will work. I just got thrown-off my those Knox errors and didn't even think to check NGINX logs, herpderp! Yes, the role is ZEPPELINWS. I am in the middle of swapping-out the self-signed certs for corporate ones, but once I get that taken care of will do the NGINX cfg changes and if works will accept your answer. THANKS for the excellent reply!!!

Cheers,

Doug

[#4]

You were exactly correct, thanks again!

For others who might be using an NGINX proxy in front of Knox, the bolded below is what I had to add to me NGINX cfg for the proxy URL:

server {

listen 443 ssl;

server_name PROXIED_URL;

ssl on;

ssl_protocols TLSv1.1 TLSv1.2;

ssl_certificate /etc/ssl/private/CORPORATE_CERT.crt;

ssl_certificate_key /etc/ssl/private/CORPORATE_CERT.key;

location / {

proxy_pass https://KNOX_HOST:8443;

proxy_http_version 1.1;

proxy_set_header Upgrade $http_upgrade;

proxy_set_header Connection "upgrade";

proxy_read_timeout 86400;

}

}