Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Zeppelin websocket message size limitations when proxied by Knox

avatar
Contributor

Hello,

I have setup Knox to proxy an HDP 3.1.0/HDF 3.3.0 cluster, but am having problems with Zeppelin...when I try to open a "large" notebook (no idea how big they actually are, since I cannot see them), like the Getting Started/Apache Spark in 5 Minutes notebook, Zeppelin hangs, and I see these errors in the Zeppelin logs:

({qtp1757676444-273} NotebookServer.java[onClose]:371) - Closed connection to 10.16.8.102 : 44880. (1006) EOF: Broken pipe

The associated upstream exceptions in the Knox logs are:

ERROR gateway.websockets (ProxyWebSocketAdapter.java:cleanupOnError(171)) - Error: org.eclipse.jetty.websocket.api.MessageTooLargeException: Resulting message size [73,728] is too large for configured max of [65,536]

Does anyone know can this message size limitation be configured/overcome? I've been googling-around for an hour now and made zero progress. I saw this Knox issue, which was apparently fixed in the 0.12.0 release, but maybe not? https://issues.apache.org/jira/browse/KNOX-836

Thanks in advance for any assistance/advice...

Cheers,

Doug

5 REPLIES 5

avatar
Rising Star

Hello Doug,

Try setting message size using the following properties

gateway.websocket.max.text.size

gateway.websocket.max.binary.size

gateway.websocket.max.text.buffer.size

gateway.websocket.max.binary.buffer.size

gateway.websocket.input.buffer.size

avatar
Contributor

Hi Sandeep,

Thanks for your suggestion...I added all those properties to the Custom gateway-site section for Knox service in Ambari, with the same value I have specified for Zeppelin's zeppelin.websocket.max.text.message.size property (i.e., 3072000):

96769-knox-message-size-properties.png

Then I restarted Knox, and re-logged-in to Zeppelin and tried to open the Getting Started/Apache Spark in 5 Minutes notebook, but received the same errors in gateway.log:

2019-01-21 15:43:11,856 ERROR gateway.websockets (ProxyWebSocketAdapter.java:cleanupOnError(171)) - Error: org.eclipse.jetty.websocket.api.MessageTooLargeException: Resulting message size [73,728] is too large for configured max of [65,536]

Did I not put those properties in the right place, or might I be missing something else?

Thanks,

Doug

avatar
Contributor

Exact Same Problem here. Any advice please / permanent fix ?

I had to manually extract the note.json file from the server and reupload it / import it, after renaming the notebook and changing the ID to be able to access to it again. That is very annoying.

avatar
Expert Contributor

There appears to be a bug with the new way Knox is creating topologies. As far as I can tell none of the Knox parameters for gateway.websocket stuff is actually getting applied because nowhere in Knox is the value ever set to 65536. The defaults for the important stuff are integer max value which is a lot higher. The only place with a default value of 65536 is in the Jetty source code so somehow parameters aren't being applied.

avatar
Expert Contributor

I submitted KNOX-1828 for this issue and have created a pull request for a patch that appears to work.