question Re: consuming from kafka topics using NiFi consumekafka processor results in multi line message being distirbuted across flowfiles in Support Questions

consuming from kafka topics using NiFi consumekafka processor results in multi line message being distirbuted across flowfiles

dhieru — Sun, 18 Aug 2019 05:25:37 GMT

Hi,

Thanks a lot to this awesome community.

I am listening on a tcp port and sending it publishkafka processor for publishing. The original message I send is attached below

The nifi template is also shown in picture below.

When I consume using consume kafka the multi message are split. See the image.

I am using nifi1.2 and using kafka version 10

Please help

Thanks a lot

Re: consuming from kafka topics using NiFi consumekafka processor results in multi line message being distirbuted across flowfiles

dhieru — Thu, 31 May 2018 21:33:45 GMT

Is there anyone else who has faced this issue?

Re: consuming from kafka topics using NiFi consumekafka processor results in multi line message being distirbuted across flowfiles

falbani — Fri, 01 Jun 2018 21:55:52 GMT

Sorry is not clear by the description what the issue is? It would help if you can clarify what is the problem you are facing and if there is any error stack please also add it.

Re: consuming from kafka topics using NiFi consumekafka processor results in multi line message being distirbuted across flowfiles

dhieru — Sat, 02 Jun 2018 01:47:14 GMT

@Felix Albani apologies for not making it clear.

I am publising a multi line message using publish kafka processor the message is something like below:

when i consume it using consume kafka messages are broken apart. any help Thanks

<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='Microsoft-Windows-Security-Auditing' Guid='{54567625-5078-4554-A5BA-3E3B0328C30D}'/><EventID>4672</EventID><Version>0</Version><Level>0</Level><Task>12548</Task><Opcode>0</Opcode><Keywords>0x8020000000000000</Keywords><TimeCreated SystemTime='2008-05-23T18:33:23.071073900Z'/><EventRecordID>510676945</EventRecordID><Correlation/><Execution ProcessID='652' ThreadID='4792'/><Channel>Security</Channel><Computer>prdserver.mycomapny.org</Computer><Security/></System><EventData><Data Name='SubjectUserSid'>S-1-5-21-442726818-4567565561-3997648070-3159</Data><Data Name='SubjectUserName'>app</Data><Data Name='SubjectDomainName'>ADmycomINT</Data><Data Name='SubjectLogonId'>0abfbd2cd</Data><Data Name='PrivilegeList'>SeSecurityPrivilege
			SeBackupPrivilege
			SeRestorePrivilege
			SeTakeOwnershipPrivilege
			SeDebugPrivilege
			SeSystemEnvironmentPrivilege
			SeLoadDriverPrivilege
			SeImpersonatePrivilege</Data></EventData></Event><br>

Re: consuming from kafka topics using NiFi consumekafka processor results in multi line message being distirbuted across flowfiles

Wynner — Mon, 04 Jun 2018 00:14:24 GMT

@dhieru singh

What is the values of the property "Message Demarcator"?

Re: consuming from kafka topics using NiFi consumekafka processor results in multi line message being distirbuted across flowfiles

dhieru — Mon, 04 Jun 2018 03:37:48 GMT

@Wynner

Thanks a lot for looking at the question.

I have not set the message demaractor on both publish and consume kafka processors.

Also when I describe my kafka topics it is "Topic:test PartitionCount:1 ReplicationFactor:1"

Also the number of concurrent tasks for Consume kafka and publish kafka is 1.

Thanks a lot

Dhieru

Re: consuming from kafka topics using NiFi consumekafka processor results in multi line message being distirbuted across flowfiles

Wynner — Mon, 04 Jun 2018 20:11:02 GMT

@dhieru singh

Try setting the demarcator on the Consume processor.

Re: consuming from kafka topics using NiFi consumekafka processor results in multi line message being distirbuted across flowfiles

dhieru — Thu, 07 Jun 2018 05:37:43 GMT

@Wynner it got fixed, however when data itself contains a new line character then it is giving a problem, should I change the message demaracter from new line(shift+enter) to something else ?

Re: consuming from kafka topics using NiFi consumekafka processor results in multi line message being distirbuted across flowfiles

Wynner — Thu, 07 Jun 2018 06:11:33 GMT

@dhieru singh

You should use a demarcator that keeps your messages the way you want them.

Re: consuming from kafka topics using NiFi consumekafka processor results in multi line message being distirbuted across flowfiles

sunil_tc — Fri, 21 Jun 2019 19:11:44 GMT

Kafka Consumer Demarcator could be one solution.Use Shift+enter and try persisting into HDFS.