Which compression is used in Site-to-Site (Remote Process Group)

[#1]

1. Which compression algorithm is used when a remote port communication is set up?
2. Can it be customized?
3. Does it work on a FlowFile level or the batch that s2s protocol negotiated for transmission?

[#2 · Accepted solution]

Site-to-Site uses deflate at level 1 and compresses data in blocks/buffers. With site-to-site a series of 1..N flowfiles are sent at once and ack'd as a group. It is not configurable at this time. Keep in mind of course you can of course compress before sending to s2s and decompress after receiving from s2s using the CompressContent processor.

Do you feel there would be a good bit of value in letting the compression of s2s be configurable? If so would that be for cases like where snappy makes sense because it is certain types of text data?

Thanks

Joe

[#3]

Yes, Joe, I had something like snappy in my mind as a good middle ground between size and performance.

As a minimum, a compression level property should be exposed to the operator to balance an existing compression protocol between speeed/cpu load and network traffic volume.