Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Which compression is used in Site-to-Site (Remote Process Group)

SOLVED Go to solution
Highlighted

Which compression is used in Site-to-Site (Remote Process Group)

  1. Which compression algorithm is used when a remote port communication is set up?
  2. Can it be customized?
  3. Does it work on a FlowFile level or the batch that s2s protocol negotiated for transmission?

320-remote-port-connection-status.png

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Which compression is used in Site-to-Site (Remote Process Group)

Site-to-Site uses deflate at level 1 and compresses data in blocks/buffers. With site-to-site a series of 1..N flowfiles are sent at once and ack'd as a group. It is not configurable at this time. Keep in mind of course you can of course compress before sending to s2s and decompress after receiving from s2s using the CompressContent processor.

Do you feel there would be a good bit of value in letting the compression of s2s be configurable? If so would that be for cases like where snappy makes sense because it is certain types of text data?

Thanks

Joe

2 REPLIES 2

Re: Which compression is used in Site-to-Site (Remote Process Group)

Site-to-Site uses deflate at level 1 and compresses data in blocks/buffers. With site-to-site a series of 1..N flowfiles are sent at once and ack'd as a group. It is not configurable at this time. Keep in mind of course you can of course compress before sending to s2s and decompress after receiving from s2s using the CompressContent processor.

Do you feel there would be a good bit of value in letting the compression of s2s be configurable? If so would that be for cases like where snappy makes sense because it is certain types of text data?

Thanks

Joe

Re: Which compression is used in Site-to-Site (Remote Process Group)

Yes, Joe, I had something like snappy in my mind as a good middle ground between size and performance.

As a minimum, a compression level property should be exposed to the operator to balance an existing compression protocol between speeed/cpu load and network traffic volume.