Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Topic XXX not present in metadata after 0000ms

avatar
Contributor

I deployed NiFi and Kafka cluster of cdp public cloud  on the azure.

Hae_0-1701059072599.png

And  I am trying to publish data to kafka(kafka-poc) user nifi(datahub-poc).

I am facing "Topic XXX not present in metadata after 0000ms" when I set 'Use transaction=false' 

 

Hae_2-1701059556476.png

And I am facing "Timeout expired after 0000milliseconds while awaiting  InitProducerId" after "Invaid transition attempted from state INITIALIZING to state ABORTING_TRANSACTION"  error when i set 'Use transaction=true'

Hae_5-1701060554469.png

 

 

 

Hae_4-1701059869254.png

 

Anything should I try to solve this problems?

 

 

 

 

 

 

1 ACCEPTED SOLUTION

avatar

@Hae I am not exactly sure of your settings or the issue with transactions, but I can share what is working for me.   In my CDP Public Cloud nifi flows i use some required settings to PublishKafka.


SSL Context Service:
Default NiFi SSL Context Service
Security Protocol: SASL_SSL
SASL Mechanism: PLAIN
Username: CDP Workload User Name
Password: CDP Workload User Password
 
Make sure you user has access to everything of course and you have set the workload password. Additionally, be patient on first processor run, sometimes takes minute or so of startup time to see SUCCESS.  Then give refresh on SMM to make sure the kafka topic is created and has the expected data.  

View solution in original post

2 REPLIES 2

avatar

@Hae I am not exactly sure of your settings or the issue with transactions, but I can share what is working for me.   In my CDP Public Cloud nifi flows i use some required settings to PublishKafka.


SSL Context Service:
Default NiFi SSL Context Service
Security Protocol: SASL_SSL
SASL Mechanism: PLAIN
Username: CDP Workload User Name
Password: CDP Workload User Password
 
Make sure you user has access to everything of course and you have set the workload password. Additionally, be patient on first processor run, sometimes takes minute or so of startup time to see SUCCESS.  Then give refresh on SMM to make sure the kafka topic is created and has the expected data.  

avatar
Contributor

It works. 

 

However  processor was spit "record has failed the validation on broker and hence will be rejected" with fail when if "complete,delete" set clean up policy on SMM. 

It works after change cleanup policy to "delete"

 

 

Thanks