I want to integrate Atlas to a Kafka Cluster , which is outside the Hortonworks Data Platform. Would you please be able to help let me know whether it's possible or is it mandatory that the Kafka cluster needs to exist within the HDP itself
@Debabrata Ghosh, integrating external Kafka to Atlas should be possible, In addition In addition you will need to create and provide appropriate permissions in Kafka for topics ATLAS_HOOKS and ATLAS_ENTITIES which will be used by Atlas and Atlas hook services for respective users.
Additionally, kafka configurations will need to be put in place in Atlas and the Atlas hook services in the atlas-application.properties atlas.kafka.bootstrap.servers and atlas.kafka.zookeeper.connect.
Mainly, Kafka brokers configuration and Kafka-Zookeeper quorum should be added in the atlas-application.properties to get started with.
If Kafka cluster is kerberized then Atlas will need to use the same KDC as Kafka cluster.