Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

listenudp listentcp newline gets added in betwenn messages

Solved Go to solution
Highlighted

listenudp listentcp newline gets added in betwenn messages

Contributor

Hi All,

Thanks a lot to this aweosome community.

We have a listenTCP and listenUDP processors listening for events. We have set "Max Batch Size" to 20000 increase throughput.

However sometime in between messages a new line chacrater is added in a flowfile.

Is it because of Batching Message delimiter? I guess no beause it is happening in between the messages in a flowfile.

any suggestions?

Thanks

Dheeru

1 ACCEPTED SOLUTION

Accepted Solutions

Re: listenudp listentcp newline gets added in betwenn messages

Can you change whatever is sending the data to ListenUDP to not send a new-line at the end of the message?

If not, how about ReplaceText to replace \n\n with \n?

5 REPLIES 5

Re: listenudp listentcp newline gets added in betwenn messages

I don't think this could happen with ListenTCP, but with ListenUDP it could happen if the data being received already has a new-line at the end. For example, if you received these two messages "This is message 1\n" and "This is message 2\n" and then you used the batching delimiter of "\n" then you'd get "This is message 1\n\nThis is message 2\n".

Re: listenudp listentcp newline gets added in betwenn messages

Contributor

@Bryan Bende Thanks a lot for the response appreciate it, yes you re absolutely correct about, I have 4 listenudp merging on to 2 merge processors in serial and the puthdfs. I am batching the message for throughput in each of the listenUDP processors and the default matching delimiter is \n so sometimes "This is message 1\n\nThis is message 2\n" this happens.

What I am looking for is this pattern in hdfs

This is message 1

This is message 2

however right now

it is writing to hdfs as

This is message 1

\n

This is message 2

here the extraline takes the additional memory. Any way we can avoid it?

Thanks

Dheery

Re: listenudp listentcp newline gets added in betwenn messages

Can you change whatever is sending the data to ListenUDP to not send a new-line at the end of the message?

If not, how about ReplaceText to replace \n\n with \n?

Re: listenudp listentcp newline gets added in betwenn messages

Contributor

@Bryan Bende Thanks for the response, unfortunately I will not be able to change from the source side, but it looks like I will have to use replaceText processor

Thank you appreciate it.

Re: listenudp listentcp newline gets added in betwenn messages

You're welcome... one more option is to use ExecuteScript to run a simple processor that reads a flow file line by line and only writes out the lines with length > 0.

Don't have an account?
Coming from Hortonworks? Activate your account here