consuming from kafka topics using NiFi consumekafka processor results in multi line message being distirbuted across flowfiles

[#1]

Hi,

Thanks a lot to this awesome community.

I am listening on a tcp port and sending it publishkafka processor for publishing. The original message I send is attached below

The nifi template is also shown in picture below.

When I consume using consume kafka the multi message are split. See the image.

I am using nifi1.2 and using kafka version 10

Please help

Thanks a lot

[#2]

Is there anyone else who has faced this issue?

[#3]

Sorry is not clear by the description what the issue is? It would help if you can clarify what is the problem you are facing and if there is any error stack please also add it.

[#4]

@Felix Albani apologies for not making it clear.

I am publising a multi line message using publish kafka processor the message is something like below:

when i consume it using consume kafka messages are broken apart. any help Thanks

<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='Microsoft-Windows-Security-Auditing' Guid='{54567625-5078-4554-A5BA-3E3B0328C30D}'/><EventID>4672</EventID><Version>0</Version><Level>0</Level><Task>12548</Task><Opcode>0</Opcode><Keywords>0x8020000000000000</Keywords><TimeCreated SystemTime='2008-05-23T18:33:23.071073900Z'/><EventRecordID>510676945</EventRecordID><Correlation/><Execution ProcessID='652' ThreadID='4792'/><Channel>Security</Channel><Computer>prdserver.mycomapny.org</Computer><Security/></System><EventData><Data Name='SubjectUserSid'>S-1-5-21-442726818-4567565561-3997648070-3159</Data><Data Name='SubjectUserName'>app</Data><Data Name='SubjectDomainName'>ADmycomINT</Data><Data Name='SubjectLogonId'>0abfbd2cd</Data><Data Name='PrivilegeList'>SeSecurityPrivilege
			SeBackupPrivilege
			SeRestorePrivilege
			SeTakeOwnershipPrivilege
			SeDebugPrivilege
			SeSystemEnvironmentPrivilege
			SeLoadDriverPrivilege
			SeImpersonatePrivilege</Data></EventData></Event><br>

[#5]

@dhieru singh

What is the values of the property "Message Demarcator"?

[#6]

@Wynner

Thanks a lot for looking at the question.

I have not set the message demaractor on both publish and consume kafka processors.

Also when I describe my kafka topics it is "Topic:test PartitionCount:1 ReplicationFactor:1"

Also the number of concurrent tasks for Consume kafka and publish kafka is 1.

Thanks a lot

Dhieru

[#7 · Accepted solution]

@dhieru singh

Try setting the demarcator on the Consume processor.

[#8]

@Wynner it got fixed, however when data itself contains a new line character then it is giving a problem, should I change the message demaracter from new line(shift+enter) to something else ?

[#9]

@dhieru singh

You should use a demarcator that keeps your messages the way you want them.

[#10]

Kafka Consumer Demarcator could be one solution.Use Shift+enter and try persisting into HDFS.