Skip to main content
Israr
Israr
Gold ⭐️
September 18, 2024
Solved

Hdfs to confluent kafka

Hello community,

I want to get the from hdfs to confluent cloud kafka is there any suggestion how to do that.

Babasaheb
BabasahebAnswer
Gold ⭐️
September 24, 2024

Hello @Israr ,

Cloudera provides the easiest ways and options to configure such setups through Cloudera Manager (CM) and Streams Messaging Manager (SMM) [1]

[1] https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/monitoring-kafka-connect/topics/smm-creating-a-connector.html

You can configure this pipeline with HDFS or Stateless NiFi Source and Sink connectors [2] & [3]. 

[2]  Public Cloud: https://docs.cloudera.com/runtime/7.2.18/kafka-connect/topics/kafka-connect-connector-nifi-stateless.html

[3] PvC: https://docs.cloudera.com/cdp-private-cloud-base/latest/kafka-connect/topics/kafka-connect-connector-nifi-stateless.html

HDFS: https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/kafka-connect/kafka-connect.pdf

If you found this response assisted with your query, please take a moment to log in and click on KUDOS 🙂 & ”Accept as Solution" below this post.

Thank you.

View in thread

7 replies

DianaTorres
Community Manager
DianaTorres
Community Manager
September 18, 2024

@Israr Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our Kafka experts @ChethanYM @paras  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Translate
Regards, Diana Torres, Senior Community Moderator
Israr
IsrarAuthor
Gold ⭐️
September 21, 2024

anyone can help ?

Translate
DianaTorres
Community Manager
DianaTorres
Community Manager
September 23, 2024

Hi! @Babasaheb @rki_ Do you have any insights here? Thanks!

Translate
Regards, Diana Torres, Senior Community Moderator
Babasaheb
Gold ⭐️
BabasahebAnswer
Gold ⭐️
September 24, 2024

Hello @Israr ,

Cloudera provides the easiest ways and options to configure such setups through Cloudera Manager (CM) and Streams Messaging Manager (SMM) [1]

[1] https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/monitoring-kafka-connect/topics/smm-creating-a-connector.html

You can configure this pipeline with HDFS or Stateless NiFi Source and Sink connectors [2] & [3]. 

[2]  Public Cloud: https://docs.cloudera.com/runtime/7.2.18/kafka-connect/topics/kafka-connect-connector-nifi-stateless.html

[3] PvC: https://docs.cloudera.com/cdp-private-cloud-base/latest/kafka-connect/topics/kafka-connect-connector-nifi-stateless.html

HDFS: https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/kafka-connect/kafka-connect.pdf

If you found this response assisted with your query, please take a moment to log in and click on KUDOS 🙂 & ”Accept as Solution" below this post.

Thank you.

Translate
    Israr
    IsrarAuthor
    Gold ⭐️
    September 24, 2024

    Israr_0-1727198765742.png

    i have install smm on my cdp priavate cluster connector option is not showing , is any other service also need i am using cdp 7.4.4 or i have chnage cdp version

    Translate
      Babasaheb
      Gold ⭐️
      Babasaheb
      Gold ⭐️
      September 25, 2024

      Hello @Israr ,

      You should verify that both Kafka and Kafka Connect are running and in a healthy state.

      Thank you. 

       

      Translate
        Israr
        IsrarAuthor
        Gold ⭐️
        September 25, 2024

        Yes, It is running i am able to create the topic 

        Translate
        Terms & ConditionsAccessibility statement