About mark_lin

mark_lin · ‎05-08-2018

This seems to be a nifi 1.4/1.5 issue. This issue is reproducible in both nifi 1.4 and 1.5. However, it works well with nifi 1.6 arvo recordreader. Thanks, Mark

mark_lin · ‎05-08-2018

Hi, My Nifi ConsumeKafakRecord_0_11 throws the following error Failed to parse message from Kafka using the configured Record Reader. Will route message as its own FlowFile to the 'parse.failure' relationship: org.apache.nifi.schema.access.SchemaNotFoundException: Could not find a schema with identifier 488 EVN: Nifi 1.4 with kafka connect and confluent schema registry service and the backend database to monitor is mysql When: This error occurred only recently when we changed to use nifi EL to specify kafka broker urls and topics in our ConsumeKafakRecord_0_11. The kafka topic is created through kafka connect with schema registry enabled. The issue here is the topic that throws this error has 3 versions in schema registry and that identifier 488 is in version 2 while ConsumeKafakRecord_0_11 issued to call to schema registry with "latest" that is latest version 3 which has a higher identifier 529. The question is how to fix this issue? I am trying to understand how ConsumeKafakRecord_0_11 determines to use 488(version 2) instead of 529(version 3). Thanks, Mark

mark_lin · ‎03-28-2018

My back pressure threshold for number of objects is default 10,000 while the data size is 1G. However, I saw the number of objects in the queue went far beyond 10.1000 though the data size was beyond 1G. Does this mean back pressure is applied only if both number of objects and data size reach the thresholds respectively? Another question. Suppose a kafka consumer processor is connected to a split processor. If the split processor reaches back pressure thresholds, will the kafka consumer stop consuming messages? Thanks, Mark

mark_lin · ‎03-23-2018

Hi Matt, The MonitorActivity processor is exceptionally useful. I will use it to monitor the overall health of my nifi processor groups. Thanks, Mark

mark_lin · ‎03-23-2018

Thanks Matt, Rahul, and Abdelkrim. Our design is changed a little bit to prevent duplicated emails from being sent out. Failed flowfiles are routed to a kafka publisher and then again routed to a updateAttribute which will serve as a holding queue. A separate processor group that is composed of a kafka consumer and a customized putMail processor that will send out the same type of error messages only once within a configured time period. We need this customized putMail processor because network or disk issue takes time to get fixed and we will receive too many duplicated emails without it. Thanks, Mark

mark_lin · ‎03-22-2018

For error handling purpose, I need a dummy processor to queue up flowfiles for reprocessing. Suppose I have a mongodb put processor that the write to mongodb. But the persistence fails due to network or disk issues. In this case, the flowfiles from the failure relationship is routed to a putMail processor which sends messages. Now I want to route all flowfiles from the putMail processor to a dummy processor and the dummy processor is routed by back to back putMongo processor. The dummy processor stays as stopped so as to queue up all the flowfiles. After the email is received, operation team will fix the mongodb issue. As this time, the dummy processor will be restarted to route the message back to mongodb put processor. Does nifi have a built-in dummy processor? Thanks, Mark

mark_lin · ‎03-22-2018

Thanks Rahul. Using separate topic means good isolation. Directly using Nifi is a good option. Mark

mark_lin · ‎03-21-2018

It was noticed today the nifi re-elected the coordinator node although the previous coordinator node stayed connected based on nifi app log. What is the logic for kicking off coordinator reelection? Thanks, Mark

mark_lin · ‎03-21-2018

Here is the scenario. I need two confluent jdbc source connectors to set up in kafka connectors. Both connectors will source data from the same table. The first connector will be setup in timestamp|incrementing mode and the second connector will be set up in bulk mode and will be used on demand in cases like complete reload due to data corruption. Is it possible and desirable to have both connectors to share the same topic? Thanks, Mark

mark_lin · ‎03-20-2018

I got the same "issue" and that is exactly the case for multiple threads. Thanks, Mark

Online	Offline
Last Visited	‎05-08-2018 04:08 PM

Member Since	‎02-09-2018 01:06 AM
Last Visited	‎05-08-2018 04:08 PM
Posts	18

Cloudera Community

Cloudera Community

Re: Nifi ConsumeKafakRecord_0_11 Throws Error "Cou...

Nifi ConsumeKafakRecord_0_11 Throws Error "Could n...

nifi back pressure threshholds

Re: Dumy processor for error handling

Re: Dumy processor for error handling

Dumy processor for error handling

Re: Share topics for different kafka connectors

What could cause nifi primary/coordinator node cha...

Share topics for different kafka connectors

Re: Does any one know why or under what circumstan...