Zeppelin websocket message size limitations when proxied by Knox

[#1]

Hello,

I have setup Knox to proxy an HDP 3.1.0/HDF 3.3.0 cluster, but am having problems with Zeppelin...when I try to open a "large" notebook (no idea how big they actually are, since I cannot see them), like the Getting Started/Apache Spark in 5 Minutes notebook, Zeppelin hangs, and I see these errors in the Zeppelin logs:

({qtp1757676444-273} NotebookServer.java[onClose]:371) - Closed connection to 10.16.8.102 : 44880. (1006) EOF: Broken pipe

The associated upstream exceptions in the Knox logs are:

ERROR gateway.websockets (ProxyWebSocketAdapter.java:cleanupOnError(171)) - Error: org.eclipse.jetty.websocket.api.MessageTooLargeException: Resulting message size [73,728] is too large for configured max of [65,536]

Does anyone know can this message size limitation be configured/overcome? I've been googling-around for an hour now and made zero progress. I saw this Knox issue, which was apparently fixed in the 0.12.0 release, but maybe not? https://issues.apache.org/jira/browse/KNOX-836

Thanks in advance for any assistance/advice...

Cheers,

Doug

[#2]

Hello Doug,

Try setting message size using the following properties

gateway.websocket.max.text.size

gateway.websocket.max.binary.size

gateway.websocket.max.text.buffer.size

gateway.websocket.max.binary.buffer.size

gateway.websocket.input.buffer.size

[#3]

Hi Sandeep,

Thanks for your suggestion...I added all those properties to the Custom gateway-site section for Knox service in Ambari, with the same value I have specified for Zeppelin's zeppelin.websocket.max.text.message.size property (i.e., 3072000):

Then I restarted Knox, and re-logged-in to Zeppelin and tried to open the Getting Started/Apache Spark in 5 Minutes notebook, but received the same errors in gateway.log:

2019-01-21 15:43:11,856 ERROR gateway.websockets (ProxyWebSocketAdapter.java:cleanupOnError(171)) - Error: org.eclipse.jetty.websocket.api.MessageTooLargeException: Resulting message size [73,728] is too large for configured max of [65,536]

Did I not put those properties in the right place, or might I be missing something else?

Thanks,

Doug

[#4]

Exact Same Problem here. Any advice please / permanent fix ?

I had to manually extract the note.json file from the server and reupload it / import it, after renaming the notebook and changing the ID to be able to access to it again. That is very annoying.

[#5]

There appears to be a bug with the new way Knox is creating topologies. As far as I can tell none of the Knox parameters for gateway.websocket stuff is actually getting applied because nowhere in Knox is the value ever set to 65536. The defaults for the important stuff are integer max value which is a lot higher. The only place with a default value of 65536 is in the Jetty source code so somehow parameters aren't being applied.

[#6]

I submitted KNOX-1828 for this issue and have created a pull request for a patch that appears to work.