Member since
11-03-2023
27
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1754 | 11-08-2023 09:29 AM |
09-23-2024
01:16 AM
1 Kudo
Hi @MattWho Thanks for your response . I also don't understand the overhead of ingesting the same messages twice in your NiFi My requirement is to send data to different end point , so that they can perform different operations on the data. Why not have have a single ConsumeKafka ingesting the messages from the topic and then routing the success relationship from the ConsumeKafka twice (once to InvokeHTTP A and once to InvokeHTTP B)? For me one flow is like one vendor , like this i will be having multiple number of vendors , everyone will have there separate end points. Keeping all in one flow is not possible . So I am creating separate data flow and separate retry logic for them. This above issue is with only 1 vendor , they require same data (consumed from same kafka topic ) to be pushed to 2 separate endpoints . But I am not able to handle the retry logic for them. Why publish failed or retry FlowFile messages to an external topic R just so they can be consumed back in to your NiFi? yes i want them to be consumed again to nifi . All failed requests iam publishing to retry topic and this is being handled in retry flow . With this iam able to keep my main flow without and failed requests and new requests which does not have any error will get pushed to end point successfully It would be more efficient to just keep them in NiFi and create a retry loop on each InvokeHTTP. NiFi even offers retry handling directly on the relationships with in the processor configuration if i add a retry loop to invoke http and the endpoint is down for a longer time , too many requests will get queued in nifi . If you must write the message out to just one topic R, you'll need to append something to the message that indicates what InvokeHTTP (A or B) failure or retry resulted in it being written to Topic R. Then have a single Retry dataflow that consume from Topic R, extracts that A or B identifier from message so that it can be routed to the correct invokeHTTP. Just seems like a lot of unnecessary overhead. Please help me with the retry logic . Data is going in same retry topic how can i differentiate between the data , whether it failed from data flow 1 or from data flow 2.
... View more
08-27-2024
05:52 AM
Hi , In My nifi flow i have two dataFlows i want to same same data to two different HTTP endPoints , for both data flow data is being consumed from same kaflka topic (TOPIC A) both flows has different kafka groupID and pushing data to different HPPT end points (endPoint 1 and endPoint 2). If there is any failure data is being produced into retryTopic R which is common for both data flow . There is separate retryData flow for both configured with seperate HttpEndPoints , but both consuming from same retryTopic . What issue coming is , due to same retry topic , dataFlow A can consume data of dataFlowB . How should i avoid this situation ? retryTopic used should be same for both. Here is a sample for the dataFlow .
... View more
Labels:
- Labels:
-
Apache NiFi
12-27-2023
10:46 PM
My "error" topic is on different Kafka cluster . So i wanted my data which is failing due to connection error should move to failure and then to error topic . Is there any way to do so ??
... View more
12-26-2023
09:05 AM
In my flow iam publishing data to kafka topic , if there is failure , that request should go to failure relation where iam publishing that data to error topic , but in my case when there is not connectivity with kafka or kafka broker value is not correct request is getting queue in PublishKafka processor not moving to failure relation . This queue will effect my other requests any even my error handling logic .Please suggest some solution Attaching the image of error and my nifi flow-
... View more
Labels:
- Labels:
-
Apache NiFi
12-21-2023
10:40 PM
nifi version iam using is old Powered by Apache NiFi - Version 1.8.0.3.3.0.0-165 , i have developed the same solution using retryFlowFile processor , but its working in my other environment due to old nifi version , so i need to have same implementation in that too . @MattWho even retry relation is not there in this version. i was trying to achieve this by expression language , but that is also not working.
... View more
12-21-2023
11:53 AM
I want to retry my flow file if it has some error for 24 hours , retry should will happen every hours . if the file is there in the flow for more then 24 hours it should be removed .
iam trying achieve same by using update attribute processors , routeOnAttribute processor and taking help of expression language . Still iam not able to achieve it.
these are the expression I am using to check elapsed time and received Time of flow file in routeOnAttribute processor but its going to unmatched relation every time -
these values are for testing pursose -
elapsed_time = ${now():toNumber():minus(${queuedtimestamp:toNumber()}):divide(60000):gt(10)}
receivedTime= ${now():toNumber():minus(${receivedTime:toDate("yyyy-MM-dd HH:mm:ss"):toNumber()}):gt(180000)}
NOTE: retryFlowFile processor in not available as my nifi version is old
Please suggest better approach to do so .
... View more
Labels:
- Labels:
-
Apache NiFi
12-20-2023
01:32 AM
control rate processor is nor working
... View more
12-19-2023
12:54 PM
Hi , iam retrying a flow file 3 times before moving moving it to failure relationship , with each retry i want to have 10 seconds gap in between them , so 2nd retry will happen after 10 seconds . iam not able to understand how to use wait processor for this . What should be the configurations of that processor . iam using older version of nifi so i can not use retryFlowFile processor .
... View more
Labels:
- Labels:
-
Apache NiFi
12-14-2023
10:14 PM
Hi , is there any idle connection timeout for ConsumeKafka and PublishKafka processors in NIFI. Do these processors go to idle state if no data in incoming ?? if yes then what is the time in which they will go to idle state in case of no data is coming ??? and is there any way to set this time ourself ???
... View more
Labels:
- Labels:
-
Apache NiFi
12-12-2023
11:06 PM
disconnect and reconnect again this functionality i want to achieve with Publish kafka processor . ConsumerKafka will consumer continuously .
... View more