Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Can we integrate an external Kafka cluster with Atlas ?

Can we integrate an external Kafka cluster with Atlas ?

New Contributor

Hi Everyone,

Greetings !

I want to integrate Atlas to a Kafka Cluster , which is outside the Hortonworks Data Platform. Would you please be able to help let me know whether it's possible or is it mandatory that the Kafka cluster needs to exist within the HDP itself

Thanks,

Debu

1 REPLY 1
Highlighted

Re: Can we integrate an external Kafka cluster with Atlas ?

New Contributor

@Debabrata Ghosh, integrating external Kafka to Atlas should be possible, In addition In addition you will need to create and provide appropriate permissions in Kafka for topics ATLAS_HOOKS and ATLAS_ENTITIES which will be used by Atlas and Atlas hook services for respective users.

Additionally, kafka configurations will need to be put in place in Atlas and the Atlas hook services in the atlas-application.properties atlas.kafka.bootstrap.servers and atlas.kafka.zookeeper.connect.

Mainly, Kafka brokers configuration and Kafka-Zookeeper quorum should be added in the atlas-application.properties to get started with.

If Kafka cluster is kerberized then Atlas will need to use the same KDC as Kafka cluster.