question Re: NiFi PublishKafka_0_10 processor in Support Questions

NiFi PublishKafka_0_10 processor

dhieru — Sun, 18 Aug 2019 02:27:05 GMT

Hi,

I am trying to publish to a kafka topic on a kafka broker. I am receving the error such as

relationship not specified; rolling back session: {}
org.apache.nifi.processor.exception.FlowFileHandlingException: StandardFlowFileRecord[uuid=be3d1509-fdc4-4235-a760-c8740a05883b,claim=,offset=0,,size=0] transfer relationship not specified
       at org.apache.nifi.controller.repository.StandardProcessSession.checkpoint(StandardProcessSession.java:248)
       at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:318)
       at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
       at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1118)
       at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
       at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
       at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       at java.lang.Thread.run(Thread.java:745)

Re: NiFi PublishKafka_0_10 processor

Wynner — Fri, 18 Aug 2017 01:17:04 GMT

@dhieru singh

What does the config of the PublishKafka processor look like?

Re: NiFi PublishKafka_0_10 processor

dhieru — Sun, 18 Aug 2019 02:26:57 GMT

kafkaconfig2.pngHi @Wynner Please find attached image

Re: NiFi PublishKafka_0_10 processor

Wynner — Sun, 20 Aug 2017 03:57:29 GMT

@dhieru singh

What about the Scheduling and Properties tabs?

Re: NiFi PublishKafka_0_10 processor

dhieru — Sun, 18 Aug 2019 02:26:49 GMT

Hi @Wynner

Thanks for your response. Here is the properties tab

Re: NiFi PublishKafka_0_10 processor

dhieru — Sun, 18 Aug 2019 02:26:41 GMT

@Wynner

Here is the scheduling tab.

Re: NiFi PublishKafka_0_10 processor

Wynner — Mon, 21 Aug 2017 03:11:50 GMT

@dhieru singh

So, there is a Kafka Broker running on the same system as NiFi?

Is this an HDF cluster?

Is 9092 the correct port?

Re: NiFi PublishKafka_0_10 processor

dhieru — Mon, 21 Aug 2017 03:16:40 GMT

Thanks for the response @Wynner

Yes this is a kafka broker running on the same system.

No it is not a HDF cluster

yes 9092 is the correct port

Thanks

Re: NiFi PublishKafka_0_10 processor

Wynner — Mon, 21 Aug 2017 03:41:30 GMT

@dhieru singh

Try putting the actual host name instead of localhost for the Kafka Broker property.

Re: NiFi PublishKafka_0_10 processor

dhieru — Mon, 21 Aug 2017 23:13:41 GMT

@Wynner

Thanks a lot for your patience.

I did try putting in the hostname instead of IP address but no success

Re: NiFi PublishKafka_0_10 processor

Wynner — Mon, 21 Aug 2017 23:29:11 GMT

@dhieru singh

I was able to replicate the error when I tried to write 0 byte files.

Use the GenerateFlowFile processor to create ten 100 byte files, instead of zero byte files

Re: NiFi PublishKafka_0_10 processor

dhieru — Mon, 21 Aug 2017 23:35:17 GMT

@Wynner

I really appreciate your help.

I tried using the GenerateFlowFileProcessor to create non zero bytes files, started it, it was successful hence it got queued on success. However it is the still same news.

Dhieru

Re: NiFi PublishKafka_0_10 processor

Wynner — Mon, 21 Aug 2017 23:40:47 GMT

@dhieru singh

Did you clear all of the zero byte files from queue on the connection first?

Re: NiFi PublishKafka_0_10 processor

dhieru — Mon, 21 Aug 2017 23:43:53 GMT

@Wynner

Yes I did empty the queue.

Re: NiFi PublishKafka_0_10 processor

Wynner — Mon, 21 Aug 2017 23:46:30 GMT

@dhieru singh

Will you post the configuration of the GenerateFlowFile processor?

Are you seeing the same error now?

Re: NiFi PublishKafka_0_10 processor

dhieru — Sun, 18 Aug 2019 02:26:33 GMT

@Wynner Thanks The generate flow file processor works however there is error on the PublishKafka side says Timeout exception,

Re: NiFi PublishKafka_0_10 processor

Wynner — Tue, 22 Aug 2017 00:07:06 GMT

@dhieru singh

Okay, so you are now getting a different error. Try setting the run schedule to 30 seconds so you don't generate so many flow files and fill up the queue.

It is probably trying to write all of the files at once and hitting the timeout.

Clear the queue, set the run schedule to 30 seconds on the GenerateFlowFile processor and then turn on the PublishKafka processor.

Re: NiFi PublishKafka_0_10 processor

dhieru — Tue, 22 Aug 2017 00:13:04 GMT

@Wynner

Thanks for the help it is hihgly appreciated

Re: NiFi PublishKafka_0_10 processor

Wynner — Tue, 22 Aug 2017 00:22:17 GMT

@dhieru singh

Are you able to Publish messages to the Kafka broker now?

Re: NiFi PublishKafka_0_10 processor

dhieru — Tue, 22 Aug 2017 00:41:16 GMT

yes @Wynner

I created a new topic and published it to that, it is working, no other changes except for creating a new topic on kafka broker. I generated a 100 B 10 files and published it, it worked like charm. Now I will move on from this PoC to a more solid use case. Thanks for you help and patience Highly appreciated.