Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
mark_lin
mark_lin
New Contributor

Re: Nifi ConsumeKafakRecord_0_11 Throws Error "Cou...

by New Contributor mark_lin in Support Questions
‎05-08-2018 03:42 PM
‎05-08-2018 03:42 PM
This seems to be a nifi 1.4/1.5 issue. This issue is reproducible in both nifi 1.4 and 1.5. However, it works well with nifi 1.6 arvo recordreader. Thanks, Mark ... View more

Nifi ConsumeKafakRecord_0_11 Throws Error "Could n...

by New Contributor mark_lin in Support Questions
‎05-08-2018 01:28 PM
‎05-08-2018 01:28 PM
Hi, My Nifi ConsumeKafakRecord_0_11 throws the following error Failed to parse message from Kafka using the configured Record Reader. Will route message as its own FlowFile to the 'parse.failure' relationship: org.apache.nifi.schema.access.SchemaNotFoundException: Could not find a schema with identifier 488 EVN: Nifi 1.4 with kafka connect and confluent schema registry service and the backend database to monitor is mysql When: This error occurred only recently when we changed to use nifi EL to specify kafka broker urls and topics in our ConsumeKafakRecord_0_11. The kafka topic is created through kafka connect with schema registry enabled. The issue here is the topic that throws this error has 3 versions in schema registry and that identifier 488 is in version 2 while ConsumeKafakRecord_0_11 issued to call to schema registry with "latest" that is latest version 3 which has a higher identifier 529. The question is how to fix this issue? I am trying to understand how ConsumeKafakRecord_0_11 determines to use 488(version 2) instead of 529(version 3). Thanks, Mark ... View more

kafka connect issue:Marking the coordinator ip:por...

by New Contributor mark_lin in Support Questions
‎03-29-2018 05:30 PM
‎03-29-2018 05:30 PM
Our kafka connect got issue after restart(kafka, registry, zookeeper) and we saw the following messages appeared again and again in kafka connect log. The #group_id# is the group_id in the connect-distributed.properties for connect. INFO Discovered coordinator ip:port (id: 2147483647 rack: null) for group #group_id#. (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:597) INFO Marking the coordinator 10.103.7.219:9092 (id: 2147483647 rack: null) dead for #group_id# Any idea of the root cause? Thanks, Mark ... View more
Labels:

nifi back pressure threshholds

by New Contributor mark_lin in Support Questions
‎03-28-2018 04:03 PM
‎03-28-2018 04:03 PM
My back pressure threshold for number of objects is default 10,000 while the data size is 1G. However, I saw the number of objects in the queue went far beyond 10.1000 though the data size was beyond 1G. Does this mean back pressure is applied only if both number of objects and data size reach the thresholds respectively? Another question. Suppose a kafka consumer processor is connected to a split processor. If the split processor reaches back pressure thresholds, will the kafka consumer stop consuming messages? Thanks, Mark ... View more
Labels:

Re: What is instance url in SiteToSiteBulletinRepo...

by New Contributor mark_lin in Support Questions
‎03-27-2018 06:17 PM
‎03-27-2018 06:17 PM
@Bryan Bende I checked the Bulletin Board and I can see the messages there. My SiteToSiteBulletinReportingTask in controller setting is running. However, there is no message that gets into input port and then get published to kafka. It worked before. After my restarting nifi, it does not send out messages anymore. Thanks, Mark ... View more

Re: What is instance url in SiteToSiteBulletinRepo...

by New Contributor mark_lin in Support Questions
‎03-26-2018 08:53 PM
‎03-26-2018 08:53 PM
It is noticed that the following rule is true only for nifi 1.5. For nifi 1.4, it accepts only one url instead of a list of urls delimited by comma for the destination field. For nifi 1.4, some events are not sent to the input port because we can configure only a single address. Any workaround for this issue? Destination URL - The URL of the destination NiFi instance to send data to, should be a comma-separated list of address in the format of http(s)://host:port/nifi. Mark ... View more

Re: Dumy processor for error handling

by New Contributor mark_lin in Support Questions
‎03-23-2018 06:51 PM
‎03-23-2018 06:51 PM
Hi Matt, The MonitorActivity processor is exceptionally useful. I will use it to monitor the overall health of my nifi processor groups. Thanks, Mark ... View more

Re: Dumy processor for error handling

by New Contributor mark_lin in Support Questions
‎03-23-2018 01:20 PM
‎03-23-2018 01:20 PM
Thanks Matt, Rahul, and Abdelkrim. Our design is changed a little bit to prevent duplicated emails from being sent out. Failed flowfiles are routed to a kafka publisher and then again routed to a updateAttribute which will serve as a holding queue. A separate processor group that is composed of a kafka consumer and a customized putMail processor that will send out the same type of error messages only once within a configured time period. We need this customized putMail processor because network or disk issue takes time to get fixed and we will receive too many duplicated emails without it. Thanks, Mark ... View more

Dumy processor for error handling

by New Contributor mark_lin in Support Questions
‎03-22-2018 08:36 PM
‎03-22-2018 08:36 PM
For error handling purpose, I need a dummy processor to queue up flowfiles for reprocessing. Suppose I have a mongodb put processor that the write to mongodb. But the persistence fails due to network or disk issues. In this case, the flowfiles from the failure relationship is routed to a putMail processor which sends messages. Now I want to route all flowfiles from the putMail processor to a dummy processor and the dummy processor is routed by back to back putMongo processor. The dummy processor stays as stopped so as to queue up all the flowfiles. After the email is received, operation team will fix the mongodb issue. As this time, the dummy processor will be restarted to route the message back to mongodb put processor. Does nifi have a built-in dummy processor? Thanks, Mark ... View more
Labels:

Re: Share topics for different kafka connectors

by New Contributor mark_lin in Support Questions
‎03-22-2018 12:07 PM
‎03-22-2018 12:07 PM
Thanks Rahul. Using separate topic means good isolation. Directly using Nifi is a good option. Mark ... View more

What could cause nifi primary/coordinator node cha...

by New Contributor mark_lin in Support Questions
‎03-21-2018 06:05 PM
‎03-21-2018 06:05 PM
It was noticed today the nifi re-elected the coordinator node although the previous coordinator node stayed connected based on nifi app log. What is the logic for kicking off coordinator reelection? Thanks, Mark ... View more

Share topics for different kafka connectors

by New Contributor mark_lin in Support Questions
‎03-21-2018 01:41 PM
‎03-21-2018 01:41 PM
Here is the scenario. I need two confluent jdbc source connectors to set up in kafka connectors. Both connectors will source data from the same table. The first connector will be setup in timestamp|incrementing mode and the second connector will be set up in bulk mode and will be used on demand in cases like complete reload due to data corruption. Is it possible and desirable to have both connectors to share the same topic? Thanks, Mark ... View more
Labels:

Re: Does any one know why or under what circumstan...

by New Contributor mark_lin in Support Questions
‎03-20-2018 01:29 PM
‎03-20-2018 01:29 PM
I got the same "issue" and that is exactly the case for multiple threads. Thanks, Mark ... View more

Re: nifi repositories directory configurations

by New Contributor mark_lin in Support Questions
‎03-20-2018 12:53 PM
‎03-20-2018 12:53 PM
Hi Matt, Thank you so much for the excellent, detailed explanation. Mark ... View more

nifi repositories directory configurations

by New Contributor mark_lin in Support Questions
‎03-19-2018 06:03 PM
‎03-19-2018 06:03 PM
It is recommended to configure nifi repository(content, flowfile, provenance) directories to point to mounted folders in linux environment. Suppose I have a nifi cluster with multiple nodes, should all nodes point to the same mount for each node should point to different mount? Take for example with shared mount for all all nodes of 3 nodes cluster. shared mount /data/nifi/node1 /data/nifi/node2 /data/nifi/node3 separate mounts /data1/nifi /data2/nifi /data3/nifi Which is the better way? Thanks, Mark ... View more
Labels:

What is instance url in SiteToSiteBulletinReportin...

by New Contributor mark_lin in Support Questions
‎03-12-2018 01:26 PM
‎03-12-2018 01:26 PM
In nifi 1.4, the SiteToSiteBulletinReportingTask has two properties: destination url and instance url? Our nifi is deployed as a cluster. 1.what is the instance url? is that the nifi rest url? 2.How is the instance url different from an destination url? Any way to find the instance url from the nifi.properties? Thanks, Mark ... View more
Labels:

Re: nifi cluster custom processor not work,nifi cu...

by New Contributor mark_lin in Support Questions
‎02-09-2018 02:46 PM
‎02-09-2018 02:46 PM
Thanks Tim, Matt and All. Figured out the root cause. Our cluster has 3 nodes instead of two nodes that is my wrong assumption. By comparing the cluster nifi.properties against the standalone nifi.properties, I found we actually have 3 nodes in the cluster. After deploying it to the third node, everything works fine. Thank for your help. Mark ... View more

nifi cluster custom processor not work,nifi custom...

by New Contributor mark_lin in Support Questions
‎02-09-2018 12:01 PM
‎02-09-2018 12:01 PM
I created a custom processor and it is deployed successfully to standalone 1.4 and 1.5 nifi. It works very well in standalone. I copied the same processor to a cluster of two servers and restarted both servers. The log shows no errors and I can see that the new processor is unpacked to the work dir. However, I can not find the new processor from nifi ui. Is there any other setup required to make it available through nifi ui in a cluster? Thanks, Mark , I have a custom processor that is written in 1.6 snapshot. It is successfully deployed to standalone nifi instance of 1.5 and 1.5. All works fine in 1.5 and 1.4 standalone instances. However, the same .nar simply does not work after it is deployed to a nifi 1.4 cluster(copied the nar to all servers) and restarted all servers. There is no erro in log. After restart, we are not able to locate the new customer processor through filter. Is there any extra setup for a cluster deployment of a custom processor? Thanks, Mark ... View more
Labels:
Contact Me
Online
Offline
Last Visited
‎05-08-2018 04:08 PM
Public Statistics
Date Registered ‎02-09-2018 01:06 AM
Last Visited ‎05-08-2018 04:08 PM
Total Messages Posted 18