Dedicated Kafka Cluster: Does Sentry need HDFS?

[#1]

Hi, we have installed a "Kafka only" cluster on CDH 6.3 along with ZooKeeper, SMM, Sentry and SchemaRegistry.  During the cluster install, Sentry (from what I can remember) was dependent on HDFS being present.  Now that the cluster is online, does anyone know if it's safe to remove HDFS?

Thank you, 
Mike

[#2 · Accepted solution]

Hi @mikejeezy ,

You shouldn't remove the HDFS component but you can stop the HDFS service in the scenario that you describe. 

Please refer to the documentation here: Configuring Kafka to Use Sentry Authorization

Sentry requires that your cluster include HDFS. After you install and start Sentry with the correct configuration, you can stop the HDFS service. 

Regards,

Steve

[#3]

Hey @StevenOD if we stop the service... do you know if there is a way to keep it from starting when we restart the cluster vi CM?

[#4]

Hi @DataMike ,

I had a chat with some of my colleagues about this and it seems there is no easy way of stopping the HDFS starting when the cluster restarts. You might be able to do something via the Cloudera Manager API - but that probably quite complicated.

If it's any consolidation, this is fixed in the next generation of technology from Cloudera i.e. CDP. When you upgrade to CDP Sentry is replaced with Ranger and this HDFS dependency for Kafka no longer exists.

Regards,

Steve

[#5]

Thanks for checking @StevenOD  (sorry for posting that last post under a different account - forgot who I was logged in as).