Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

consuming from kafka topics using NiFi consumekafka processor results in multi line message being distirbuted across flowfiles

Solved Go to solution
Highlighted

consuming from kafka topics using NiFi consumekafka processor results in multi line message being distirbuted across flowfiles

Contributor

Hi,

Thanks a lot to this awesome community.

I am listening on a tcp port and sending it publishkafka processor for publishing. The original message I send is attached below

The nifi template is also shown in picture below.

When I consume using consume kafka the multi message are split. See the image.

I am using nifi1.2 and using kafka version 10

Please help

77441-nifi1.png

77442-consume1.png

Thanks a lot


originalmessage.png
1 ACCEPTED SOLUTION

Accepted Solutions

Re: consuming from kafka topics using NiFi consumekafka processor results in multi line message being distirbuted across flowfiles

@dhieru singh

Try setting the demarcator on the Consume processor.

9 REPLIES 9

Re: consuming from kafka topics using NiFi consumekafka processor results in multi line message being distirbuted across flowfiles

Contributor

Is there anyone else who has faced this issue?

Re: consuming from kafka topics using NiFi consumekafka processor results in multi line message being distirbuted across flowfiles

Sorry is not clear by the description what the issue is? It would help if you can clarify what is the problem you are facing and if there is any error stack please also add it.

Re: consuming from kafka topics using NiFi consumekafka processor results in multi line message being distirbuted across flowfiles

Contributor

@Felix Albani apologies for not making it clear.

I am publising a multi line message using publish kafka processor the message is something like below:

when i consume it using consume kafka messages are broken apart. any help Thanks

<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='Microsoft-Windows-Security-Auditing' Guid='{54567625-5078-4554-A5BA-3E3B0328C30D}'/><EventID>4672</EventID><Version>0</Version><Level>0</Level><Task>12548</Task><Opcode>0</Opcode><Keywords>0x8020000000000000</Keywords><TimeCreated SystemTime='2008-05-23T18:33:23.071073900Z'/><EventRecordID>510676945</EventRecordID><Correlation/><Execution ProcessID='652' ThreadID='4792'/><Channel>Security</Channel><Computer>prdserver.mycomapny.org</Computer><Security/></System><EventData><Data Name='SubjectUserSid'>S-1-5-21-442726818-4567565561-3997648070-3159</Data><Data Name='SubjectUserName'>app</Data><Data Name='SubjectDomainName'>ADmycomINT</Data><Data Name='SubjectLogonId'>0abfbd2cd</Data><Data Name='PrivilegeList'>SeSecurityPrivilege
			SeBackupPrivilege
			SeRestorePrivilege
			SeTakeOwnershipPrivilege
			SeDebugPrivilege
			SeSystemEnvironmentPrivilege
			SeLoadDriverPrivilege
			SeImpersonatePrivilege</Data></EventData></Event><br>

Re: consuming from kafka topics using NiFi consumekafka processor results in multi line message being distirbuted across flowfiles

@dhieru singh

What is the values of the property "Message Demarcator"?

Re: consuming from kafka topics using NiFi consumekafka processor results in multi line message being distirbuted across flowfiles

Contributor
@Wynner

Thanks a lot for looking at the question.

I have not set the message demaractor on both publish and consume kafka processors.

Also when I describe my kafka topics it is "Topic:test PartitionCount:1 ReplicationFactor:1"

Also the number of concurrent tasks for Consume kafka and publish kafka is 1.

Thanks a lot

Dhieru

Re: consuming from kafka topics using NiFi consumekafka processor results in multi line message being distirbuted across flowfiles

@dhieru singh

Try setting the demarcator on the Consume processor.

Re: consuming from kafka topics using NiFi consumekafka processor results in multi line message being distirbuted across flowfiles

Contributor

@Wynner it got fixed, however when data itself contains a new line character then it is giving a problem, should I change the message demaracter from new line(shift+enter) to something else ?

Re: consuming from kafka topics using NiFi consumekafka processor results in multi line message being distirbuted across flowfiles

@dhieru singh

You should use a demarcator that keeps your messages the way you want them.

Re: consuming from kafka topics using NiFi consumekafka processor results in multi line message being distirbuted across flowfiles

New Contributor

Kafka Consumer Demarcator could be one solution.Use Shift+enter and try persisting into HDFS.