Hdfs to confluent kafka

[#1]

Hello community,

I want to get the from hdfs to confluent cloud kafka is there any suggestion how to do that.

[#2]

@Israr Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our Kafka experts @ChethanYM @paras  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

[#3]

anyone can help ?

[#4]

Hi! @Babasaheb @rki_ Do you have any insights here? Thanks!

[#5 · Accepted solution]

Hello @Israr ,

Cloudera provides the easiest ways and options to configure such setups through Cloudera Manager (CM) and Streams Messaging Manager (SMM) [1]

[1] https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/monitoring-kafka-connect/topics/smm-creating-...

You can configure this pipeline with HDFS or Stateless NiFi Source and Sink connectors [2] & [3]. 

[2]  Public Cloud: https://docs.cloudera.com/runtime/7.2.18/kafka-connect/topics/kafka-connect-connector-nifi-stateless...

[3] PvC: https://docs.cloudera.com/cdp-private-cloud-base/latest/kafka-connect/topics/kafka-connect-connector...

HDFS: https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/kafka-connect/kafka-connect.pdf

If you found this response assisted with your query, please take a moment to log in and click on KUDOS 🙂 & ”Accept as Solution" below this post.

Thank you.

[#6]

i have install smm on my cdp priavate cluster connector option is not showing , is any other service also need i am using cdp 7.4.4 or i have chnage cdp version

[#7]

Hello @Israr ,

You should verify that both Kafka and Kafka Connect are running and in a healthy state.

Thank you. 

[#8]

Yes, It is running i am able to create the topic