question Dedicated Kafka Cluster: Does Sentry need HDFS? in Support Questions

Dedicated Kafka Cluster: Does Sentry need HDFS?

mikejeezy — Thu, 09 Apr 2020 15:49:23 GMT

Hi, we have installed a "Kafka only" cluster on CDH 6.3 along with ZooKeeper, SMM, Sentry and SchemaRegistry. During the cluster install, Sentry (from what I can remember) was dependent on HDFS being present. Now that the cluster is online, does anyone know if it's safe to remove HDFS?

Thank you,
Mike

Re: Dedicated Kafka Cluster: Does Sentry need HDFS?

StevenOD — Sat, 11 Apr 2020 00:12:21 GMT

Hi @mikejeezy ,

You shouldn't remove the HDFS component but you can stop the HDFS service in the scenario that you describe.

Please refer to the documentation here: Configuring Kafka to Use Sentry Authorization

Sentry requires that your cluster include HDFS. After you install and start Sentry with the correct configuration, you can stop the HDFS service.

Regards,

Steve

Re: Dedicated Kafka Cluster: Does Sentry need HDFS?

DataMike — Mon, 13 Apr 2020 15:11:24 GMT

Hey @StevenOD if we stop the service... do you know if there is a way to keep it from starting when we restart the cluster vi CM?

Re: Dedicated Kafka Cluster: Does Sentry need HDFS?

StevenOD — Thu, 16 Apr 2020 07:42:52 GMT

Hi @DataMike ,

I had a chat with some of my colleagues about this and it seems there is no easy way of stopping the HDFS starting when the cluster restarts. You might be able to do something via the Cloudera Manager API - but that probably quite complicated.

If it's any consolidation, this is fixed in the next generation of technology from Cloudera i.e. CDP. When you upgrade to CDP Sentry is replaced with Ranger and this HDFS dependency for Kafka no longer exists.

Regards,

Steve

Re: Dedicated Kafka Cluster: Does Sentry need HDFS?

mikejeezy — Thu, 16 Apr 2020 14:39:50 GMT

Thanks for checking @StevenOD (sorry for posting that last post under a different account - forgot who I was logged in as).