About srilakshmi

srilakshmi · ‎03-27-2023

Hi @MattWho Thanks for the reply. I know that "Best Effort" and "Guarantee single node" delivery setting in the PublishKafka processor have nothing to do with the NiFi nodes in the NiFi cluster. This has to do with the nodes in the destination Kafka setup. My query is when i set the options as "Guarantee single node" and there are 3 kafka nodes. So when one of the kafka node is down, the publishkafka processor will try with other two kafka nodes, is my understanding right? Thanks

srilakshmi · ‎03-22-2023

Hi, I have query on "Delivery guarantee" parameter in publishKafka processor in NIFI. I have query on below two option supported for the "Delivery guarantee" 1. Best effort 2. Guarantee single node delivery I know that the "best effort" does not wait for the ack/response from kafka, but in case of publish failure, when either of the above two option is set, will NIFI try to publish among the available nodes? Like in my setup its the cluster of three kafka nodes. If one of the node is down or if we observe connection timeout issue with one of the node, will NIFI tries to publish with other available nodes? Thanks

srilakshmi · ‎03-22-2023

Hi @MattWho I am not able to find those 3 flow files in provenance lineage, because the test was performed on March 20th and the oldest data i could find is from March 22nd in data provenance lineage. As per NIFI UI, the version is "Apache NIFI Version 1.9.0" , seems like we need to upgrade to 1.16 as the version is older(1.9.0)

srilakshmi · ‎03-22-2023

Hi @MattWho I am not able to found those 3 flow files yet. Yes i am auto terminating the success relationship out of logAttribute processor, i will check regarding DROP events and let you know. And also let me check on the NIFI version we are using. Thanks for the reply

srilakshmi · ‎03-21-2023

Hi @MattWho Thanks for the reply. The log message which are missing are the custom logs which we added using "logAttribute" processor in NIFI. we are using NIFI for kafka, when kafka publish is success or failure we log some message and this done using the logAttribute processor. These log messages are important to us to identify if all the messages are successfully published to kafka or not. we are observing that few of the custom logs added using logAttribute processor are missing. Please let me know the possible reason for these log messages missing. And please note that this issue is seen only during HA testing, like bringing down one NIFI instance and bringing it up back. Thanks.

srilakshmi · ‎03-20-2023

Hi, I have two NIFI instance running and i am sending data. Both the nodes receives the data in random order. when i kill one of NIFI instances(first instance), all the data is processed by other NIFI instance(Second instance). And again when i bring back the other NIFI instance which was killed , the data is distributed among them in random order. Here there is no data loss observed, but we are observing some log messages are missing in the nifi-app.log as part of this restart. For example, during this testing if i have sent 100 messages, i see only log messages for 97 messages. Is this expected in NIFI or is there a way we can fix this issue? Is it related to gracefully shutting down the NIFI? Any help would be appreciated. Thanks

srilakshmi · ‎03-14-2023

Hi , Currently i am using NIFI to consume and publish data using kafka and i have a requirement to perform retry only for topicAuthorizationException thrown by NIFI processor (publishKafka processor). Like similar to RouteonAttribute, is there any processor or any log attribute through which i can parse or read this topicAuthorizationException error , so that i can perform retry? when kafka topic does not exist, NIFI throws below exception, the same is seen in nifi-app.log, i need to know how can we read this error in NIFI org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [test_topic]

srilakshmi · ‎01-16-2023

Hi, I wanted to create log files in nifi based on the processor group, could someone help me on the configuration that needs to be done in logback.xml file? Like currently i have 3 different processor groups and i wanted to create 3 different log files for each of them.

Online	Offline
Last Visited	‎10-30-2023 11:42 AM

Member Since	‎01-16-2023 11:11 PM
Last Visited	‎10-30-2023 11:42 AM
Posts	11

Cloudera Community

Re: Query on Delivery guarantee parameter in Publi...

Query on Delivery guarantee parameter in PublishKa...

Re: Log messages are missing with NIFI HA testing

Re: Log messages are missing with NIFI HA testing

Re: Log messages are missing with NIFI HA testing

Log messages are missing with NIFI HA testing

Routing based on exception thrown by NIFI processo...

Creating log files in nifi based on processor grou...