Re: Nifi ConsumeKafakRecord_0_11 Throws Error "Cou...

mark_lin · ‎05-08-2018

This seems to be a nifi 1.4/1.5 issue. This issue is reproducible in both nifi 1.4 and 1.5. However, it works well with nifi 1.6 arvo recordreader. Thanks, Mark

mark_lin · ‎05-08-2018

Hi, My Nifi ConsumeKafakRecord_0_11 throws the following error Failed to parse message from Kafka using the configured Record Reader. Will route message as its own FlowFile to the 'parse.failure' relationship: org.apache.nifi.schema.access.SchemaNotFoundException: Could not find a schema with identifier 488 EVN: Nifi 1.4 with kafka connect and confluent schema registry service and the backend database to monitor is mysql When: This error occurred only recently when we changed to use nifi EL to specify kafka broker urls and topics in our ConsumeKafakRecord_0_11. The kafka topic is created through kafka connect with schema registry enabled. The issue here is the topic that throws this error has 3 versions in schema registry and that identifier 488 is in version 2 while ConsumeKafakRecord_0_11 issued to call to schema registry with "latest" that is latest version 3 which has a higher identifier 529. The question is how to fix this issue? I am trying to understand how ConsumeKafakRecord_0_11 determines to use 488(version 2) instead of 529(version 3). Thanks, Mark

mark_lin · ‎03-29-2018

Our kafka connect got issue after restart(kafka, registry, zookeeper) and we saw the following messages appeared again and again in kafka connect log. The #group_id# is the group_id in the connect-distributed.properties for connect. INFO Discovered coordinator ip:port (id: 2147483647 rack: null) for group #group_id#. (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:597) INFO Marking the coordinator 10.103.7.219:9092 (id: 2147483647 rack: null) dead for #group_id# Any idea of the root cause? Thanks, Mark

mark_lin · ‎03-28-2018

My back pressure threshold for number of objects is default 10,000 while the data size is 1G. However, I saw the number of objects in the queue went far beyond 10.1000 though the data size was beyond 1G. Does this mean back pressure is applied only if both number of objects and data size reach the thresholds respectively? Another question. Suppose a kafka consumer processor is connected to a split processor. If the split processor reaches back pressure thresholds, will the kafka consumer stop consuming messages? Thanks, Mark

mark_lin · ‎03-27-2018

@Bryan Bende I checked the Bulletin Board and I can see the messages there. My SiteToSiteBulletinReportingTask in controller setting is running. However, there is no message that gets into input port and then get published to kafka. It worked before. After my restarting nifi, it does not send out messages anymore. Thanks, Mark

mark_lin · ‎03-26-2018

It is noticed that the following rule is true only for nifi 1.5. For nifi 1.4, it accepts only one url instead of a list of urls delimited by comma for the destination field. For nifi 1.4, some events are not sent to the input port because we can configure only a single address. Any workaround for this issue? Destination URL - The URL of the destination NiFi instance to send data to, should be a comma-separated list of address in the format of http(s)://host:port/nifi. Mark

mark_lin · ‎03-23-2018

Hi Matt, The MonitorActivity processor is exceptionally useful. I will use it to monitor the overall health of my nifi processor groups. Thanks, Mark

mark_lin · ‎03-23-2018

Thanks Matt, Rahul, and Abdelkrim. Our design is changed a little bit to prevent duplicated emails from being sent out. Failed flowfiles are routed to a kafka publisher and then again routed to a updateAttribute which will serve as a holding queue. A separate processor group that is composed of a kafka consumer and a customized putMail processor that will send out the same type of error messages only once within a configured time period. We need this customized putMail processor because network or disk issue takes time to get fixed and we will receive too many duplicated emails without it. Thanks, Mark

mark_lin · ‎03-22-2018

For error handling purpose, I need a dummy processor to queue up flowfiles for reprocessing. Suppose I have a mongodb put processor that the write to mongodb. But the persistence fails due to network or disk issues. In this case, the flowfiles from the failure relationship is routed to a putMail processor which sends messages. Now I want to route all flowfiles from the putMail processor to a dummy processor and the dummy processor is routed by back to back putMongo processor. The dummy processor stays as stopped so as to queue up all the flowfiles. After the email is received, operation team will fix the mongodb issue. As this time, the dummy processor will be restarted to route the message back to mongodb put processor. Does nifi have a built-in dummy processor? Thanks, Mark

mark_lin · ‎03-22-2018

Thanks Rahul. Using separate topic means good isolation. Directly using Nifi is a good option. Mark

mark_lin · ‎03-21-2018

It was noticed today the nifi re-elected the coordinator node although the previous coordinator node stayed connected based on nifi app log. What is the logic for kicking off coordinator reelection? Thanks, Mark

mark_lin · ‎03-21-2018

Here is the scenario. I need two confluent jdbc source connectors to set up in kafka connectors. Both connectors will source data from the same table. The first connector will be setup in timestamp|incrementing mode and the second connector will be set up in bulk mode and will be used on demand in cases like complete reload due to data corruption. Is it possible and desirable to have both connectors to share the same topic? Thanks, Mark

mark_lin · ‎03-20-2018

I got the same "issue" and that is exactly the case for multiple threads. Thanks, Mark

mark_lin · ‎03-20-2018

Hi Matt, Thank you so much for the excellent, detailed explanation. Mark

mark_lin · ‎03-19-2018

It is recommended to configure nifi repository(content, flowfile, provenance) directories to point to mounted folders in linux environment. Suppose I have a nifi cluster with multiple nodes, should all nodes point to the same mount for each node should point to different mount? Take for example with shared mount for all all nodes of 3 nodes cluster. shared mount /data/nifi/node1 /data/nifi/node2 /data/nifi/node3 separate mounts /data1/nifi /data2/nifi /data3/nifi Which is the better way? Thanks, Mark

mark_lin · ‎03-12-2018

In nifi 1.4, the SiteToSiteBulletinReportingTask has two properties: destination url and instance url? Our nifi is deployed as a cluster. 1.what is the instance url? is that the nifi rest url? 2.How is the instance url different from an destination url? Any way to find the instance url from the nifi.properties? Thanks, Mark

mark_lin · ‎02-09-2018

Thanks Tim, Matt and All. Figured out the root cause. Our cluster has 3 nodes instead of two nodes that is my wrong assumption. By comparing the cluster nifi.properties against the standalone nifi.properties, I found we actually have 3 nodes in the cluster. After deploying it to the third node, everything works fine. Thank for your help. Mark

mark_lin · ‎02-09-2018

I created a custom processor and it is deployed successfully to standalone 1.4 and 1.5 nifi. It works very well in standalone. I copied the same processor to a cluster of two servers and restarted both servers. The log shows no errors and I can see that the new processor is unpacked to the work dir. However, I can not find the new processor from nifi ui. Is there any other setup required to make it available through nifi ui in a cluster? Thanks, Mark , I have a custom processor that is written in 1.6 snapshot. It is successfully deployed to standalone nifi instance of 1.5 and 1.5. All works fine in 1.5 and 1.4 standalone instances. However, the same .nar simply does not work after it is deployed to a nifi 1.4 cluster(copied the nar to all servers) and restarted all servers. There is no erro in log. After restart, we are not able to locate the new customer processor through filter. Is there any extra setup for a cluster deployment of a custom processor? Thanks, Mark

Online	Offline
Last Visited	‎05-08-2018 04:08 PM

Date Registered	‎02-09-2018 01:06 AM
Last Visited	‎05-08-2018 04:08 PM
Total Messages Posted	18

Re: Nifi ConsumeKafakRecord_0_11 Throws Error "Cou...

Nifi ConsumeKafakRecord_0_11 Throws Error "Could n...

kafka connect issue:Marking the coordinator ip:por...

nifi back pressure threshholds

Re: What is instance url in SiteToSiteBulletinRepo...

Re: What is instance url in SiteToSiteBulletinRepo...

Re: Dumy processor for error handling

Re: Dumy processor for error handling

Dumy processor for error handling

Re: Share topics for different kafka connectors

What could cause nifi primary/coordinator node cha...

Share topics for different kafka connectors

Re: Does any one know why or under what circumstan...

Re: nifi repositories directory configurations

nifi repositories directory configurations

What is instance url in SiteToSiteBulletinReportin...

Re: nifi cluster custom processor not work,nifi cu...

nifi cluster custom processor not work,nifi custom...