Created on 08-17-2017 03:41 PM - edited 08-17-2019 07:27 PM
Hi,
I am trying to publish to a kafka topic on a kafka broker. I am receving the error such as
relationship not specified; rolling back session: {} org.apache.nifi.processor.exception.FlowFileHandlingException: StandardFlowFileRecord[uuid=be3d1509-fdc4-4235-a760-c8740a05883b,claim=,offset=0,,size=0] transfer relationship not specified at org.apache.nifi.controller.repository.StandardProcessSession.checkpoint(StandardProcessSession.java:248) at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:318) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1118) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
Created 08-21-2017 04:46 PM
Will you post the configuration of the GenerateFlowFile processor?
Are you seeing the same error now?
Created 08-17-2017 06:17 PM
What does the config of the PublishKafka processor look like?
Created on 08-19-2017 04:30 AM - edited 08-17-2019 07:26 PM
kafkaconfig2.pngHi @Wynner Please find attached image
Created 08-19-2017 08:57 PM
What about the Scheduling and Properties tabs?
Created on 08-19-2017 09:01 PM - edited 08-17-2019 07:26 PM
Created on 08-19-2017 09:17 PM - edited 08-17-2019 07:26 PM
Created 08-20-2017 08:11 PM
So, there is a Kafka Broker running on the same system as NiFi?
Is this an HDF cluster?
Is 9092 the correct port?
Created 08-20-2017 08:16 PM
Thanks for the response @Wynner
Yes this is a kafka broker running on the same system.
No it is not a HDF cluster
yes 9092 is the correct port
Thanks
Created 08-20-2017 08:41 PM
Try putting the actual host name instead of localhost for the Kafka Broker property.
Created 08-21-2017 04:13 PM
Thanks a lot for your patience.
I did try putting in the hostname instead of IP address but no success
Created 08-21-2017 04:29 PM
I was able to replicate the error when I tried to write 0 byte files.
Use the GenerateFlowFile processor to create ten 100 byte files, instead of zero byte files
Created 08-21-2017 04:35 PM
I really appreciate your help.
I tried using the GenerateFlowFileProcessor to create non zero bytes files, started it, it was successful hence it got queued on success. However it is the still same news.
Dhieru
Created 08-21-2017 04:40 PM
Did you clear all of the zero byte files from queue on the connection first?
Created 08-21-2017 04:43 PM
Yes I did empty the queue.
Created 08-21-2017 04:46 PM
Will you post the configuration of the GenerateFlowFile processor?
Are you seeing the same error now?
Created 08-21-2017 05:07 PM
Okay, so you are now getting a different error. Try setting the run schedule to 30 seconds so you don't generate so many flow files and fill up the queue.
It is probably trying to write all of the files at once and hitting the timeout.
Clear the queue, set the run schedule to 30 seconds on the GenerateFlowFile processor and then turn on the PublishKafka processor.
Created on 08-21-2017 04:57 PM - edited 08-17-2019 07:26 PM
@Wynner Thanks The generate flow file processor works however there is error on the PublishKafka side says Timeout exception,
Created 08-21-2017 05:13 PM
Thanks for the help it is hihgly appreciated
Created 08-21-2017 05:22 PM
Are you able to Publish messages to the Kafka broker now?
Created 08-21-2017 05:41 PM
yes @Wynner
I created a new topic and published it to that, it is working, no other changes except for creating a new topic on kafka broker. I generated a 100 B 10 files and published it, it worked like charm. Now I will move on from this PoC to a more solid use case. Thanks for you help and patience Highly appreciated.
Created on 08-25-2017 07:24 AM - edited 08-17-2019 07:26 PM
I had the same problem with PublishKafka_0_10 processor and I found that there is an open issue in JIRA
https://issues.apache.org/jira/browse/NIFI-3957
Meanwhile I solved the problem controlling the size of the FlowFile before with RouteOnAttribute