I want to read, with Consume_Kafka_2_0, all the headers that come with the message, but without knowing their names first.
In NiFi ConsumeKafka:
- Headers to Add as Attributes (Regex): knowheader1|knownHeader|knownHeaderN --> no problem
- Headers to Add as Attributes (Regex): .* --> at some point "Exception when processing data from kafka so will close the lease org.apache.nifi.processors.kafka.pubsub.ConsumerPool$SimpleConsumerLease@xxxxxxx due to null: java.lang.NullPointerException
The above error seems due to the fact that some messages come with null headers, however I don't know how to save all the custom headers that come with the message and at the same time avoid the above error.
Do you know how to solve this problem?
Not sure if this fully answers your question, but I believe that in recent versions of Kafka (including 2.0) Null headers are not supported anymore.
Also, you see a null pointer exception, but that is more a general indication that something went wrong, and does not directly indicate that the header is null.
I would recommend you to find out what the header actually is (something null, or perhaps something complex) and search further from there.
If you just want to write a message from Kafka to a file, is there a reason you can't just use ConsumeKafka -> PutFile?
The error is because you only put two Upsers lines inside of the if (flowFile != None): and if flow file is null you are still calling transfer on line 18 which is producing the error.