Nifi PublishKafka silent failures

[#1]

I'm trying to construct a process group that produces to a Kafka topic. It's failing as all records are routed to the failure queue, but I get no errors from the PublishKafka processor to help me diagnose the problem.

Digging into the PublishKafka process it seems that it doesn't produce any output whether on success or failure, despite having bulletin level set to DEBUG.

If I substitute for a PutFile processor set to DEBUG then I get bulletins for each written file. So this looks like an issue specific to PublishKafka?

[#2]

Hello @agriff 

Welcome to the community. 

I find it very odd that no ERROR logging is being produced when your PublishKafka processor is routing FlowFiles to the Failure relationship.  Keep in mind that NiFi does not control the logging in the third party libraries that a given processor type may use.  So it is possible that the Kafka client library for the specific PublishKafka version has not DEBUG logging.  This is not specific to only publishKafka, but can be the case for any processor component that is dependent on a third party client library to which the open source community has no ability to modify.

The "PublishKafka" with no version number in its name is the oldest of all the client versions.  It was the first built and was deprecated some time ago because of its version age.  If you Kafka server is newer then 0.8, you'll want to be using a different version of this processor.  There are so many version of the Kafka based processors because of version client/server incompatibility between versions.

1. What version of Kafka are you publishing to?
2. What version of Apache NiFi are you using?
3. Which PublishKafka processor are you using?  There are numerous that use different Kafka Client library versions. You'll want to use the appropriate one that aligns with your Kafka server version?
   
4. Changing the "bulletin level" within the processor has not affect on the log level for that processor class in the nifi-app.log.  To set this processor class to DEBUG in the nifi-app.log, you'll need to modify the logback.xml in the NiFi conf directory:
   1. Example logger line you would add to logback.xml ith rest of existing loggers:

<logger name="org.apache.nifi.processors.kafka.pubsub.PublishKafka" level="DEBUG"/>​

• The class name will vary by processor:
  

  org.apache.nifi.processors.kafka.pubsub.PublishKafka

  org.apache.nifi.processors.kafka.pubsub.PublishKafka_0_10

  org.apache.nifi.processors.kafka.pubsub.PublishKafka_0_11

  org.apache.nifi.processors.kafka.pubsub.PublishKafka_1_0

  org.apache.nifi.processors.kafka.pubsub.PublishKafka_2_0

  org.apache.nifi.processors.kafka.pubsub.PublishKafka_2_6

  org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_0_10

  org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_0_11

  org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_1_0

  org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_2_0

  org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_2_6

• Sharing your dataflow and processor configuration might also be helpful to your query.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#3]

I'm publishing to Kafka 2.5 and running Nifi 2.4. There doesn't seem to be a choice in Nifi 2 for Kafka libraries - one size fits all?

Anyway, not exactly sure what I did but suddenly I started getting errors reported to bulletin board and was able to diagnose and fix the underlying issue. It was a schema issue - missing fields without default value.

I still don't get bulletins for successfully produced logs to Kafka, but from what you're saying that's just a limitation of the 3rd party libraries used?

[#4]

@agriff 
I did not know that you were using the Apache NiFi 2.x release.  
The component list I provided is from the Apache NiFi 1.x release.

NiFi 2.x switched from having numerous client version Kafka based processors to single Kafka based processors that now use a KafkaConnectionService controller service  component to define the kafka client version.  In Apache NiFi the only connection service included is for theKafka 3 Client.  The Kafka client 3 I understand to be backwards compatible to Kafka 2.6, but sounds like you are having success with using it for Kafka 2.5.
Glad to hear you were able to resolve yoru underlying schema issue.

Setting Bulletins level on a processor has absolutely nothing to do with log levels written to the nifi-app.log.  It only controls what level bulletins are created within the NiFi UI.  To change logging within the NiFi logs, you will need to modify the logback.xml configuration file found in the NiFi conf directory. 

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#5]

@agriff Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.