Support Questions

Find answers, ask questions, and share your expertise

Can we integrate an external Kafka cluster with Atlas ?

New Contributor

Hi Everyone,

Greetings !

I want to integrate Atlas to a Kafka Cluster , which is outside the Hortonworks Data Platform. Would you please be able to help let me know whether it's possible or is it mandatory that the Kafka cluster needs to exist within the HDP itself




Cloudera Employee

@Debabrata Ghosh, integrating external Kafka to Atlas should be possible, In addition In addition you will need to create and provide appropriate permissions in Kafka for topics ATLAS_HOOKS and ATLAS_ENTITIES which will be used by Atlas and Atlas hook services for respective users.

Additionally, kafka configurations will need to be put in place in Atlas and the Atlas hook services in the atlas.kafka.bootstrap.servers and atlas.kafka.zookeeper.connect.

Mainly, Kafka brokers configuration and Kafka-Zookeeper quorum should be added in the to get started with.

If Kafka cluster is kerberized then Atlas will need to use the same KDC as Kafka cluster.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.