- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
consuming from kafka topics using NiFi consumekafka processor results in multi line message being distirbuted across flowfiles
- Labels:
-
Apache Kafka
-
Apache NiFi
Created on ‎05-23-2018 10:51 PM - edited ‎08-17-2019 10:25 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
Thanks a lot to this awesome community.
I am listening on a tcp port and sending it publishkafka processor for publishing. The original message I send is attached below
The nifi template is also shown in picture below.
When I consume using consume kafka the multi message are split. See the image.
I am using nifi1.2 and using kafka version 10
Please help
Thanks a lot
Created ‎06-04-2018 01:11 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Created ‎05-31-2018 02:33 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Is there anyone else who has faced this issue?
Created ‎06-01-2018 02:55 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Sorry is not clear by the description what the issue is? It would help if you can clarify what is the problem you are facing and if there is any error stack please also add it.
Created ‎06-01-2018 06:47 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Felix Albani apologies for not making it clear.
I am publising a multi line message using publish kafka processor the message is something like below:
when i consume it using consume kafka messages are broken apart. any help Thanks
<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='Microsoft-Windows-Security-Auditing' Guid='{54567625-5078-4554-A5BA-3E3B0328C30D}'/><EventID>4672</EventID><Version>0</Version><Level>0</Level><Task>12548</Task><Opcode>0</Opcode><Keywords>0x8020000000000000</Keywords><TimeCreated SystemTime='2008-05-23T18:33:23.071073900Z'/><EventRecordID>510676945</EventRecordID><Correlation/><Execution ProcessID='652' ThreadID='4792'/><Channel>Security</Channel><Computer>prdserver.mycomapny.org</Computer><Security/></System><EventData><Data Name='SubjectUserSid'>S-1-5-21-442726818-4567565561-3997648070-3159</Data><Data Name='SubjectUserName'>app</Data><Data Name='SubjectDomainName'>ADmycomINT</Data><Data Name='SubjectLogonId'>0abfbd2cd</Data><Data Name='PrivilegeList'>SeSecurityPrivilege SeBackupPrivilege SeRestorePrivilege SeTakeOwnershipPrivilege SeDebugPrivilege SeSystemEnvironmentPrivilege SeLoadDriverPrivilege SeImpersonatePrivilege</Data></EventData></Event><br>
Created ‎06-03-2018 05:14 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
What is the values of the property "Message Demarcator"?
Created ‎06-03-2018 08:37 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks a lot for looking at the question.
I have not set the message demaractor on both publish and consume kafka processors.
Also when I describe my kafka topics it is "Topic:test PartitionCount:1 ReplicationFactor:1"
Also the number of concurrent tasks for Consume kafka and publish kafka is 1.
Thanks a lot
Dhieru
Created ‎06-04-2018 01:11 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Try setting the demarcator on the Consume processor.
Created ‎06-06-2018 10:37 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Wynner it got fixed, however when data itself contains a new line character then it is giving a problem, should I change the message demaracter from new line(shift+enter) to something else ?
Created ‎06-06-2018 11:11 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You should use a demarcator that keeps your messages the way you want them.
Created ‎06-21-2019 12:11 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Kafka Consumer Demarcator could be one solution.Use Shift+enter and try persisting into HDFS.
