Atlas : how to secure Kafka ?

[#1]

Hello,

I could access all topics on kafka without authentification.

My question : how I could secure access on kafka topics ?

Thks.

[#2]

@Smart Data

Atlas is more Governance related, security to a less extent.

You secure Kafka via Kerberos for authentication, and Ranger for authorization:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/index.html#bk_security

[#3]

@Graham Martin

Thks for your quick reply. Is there an alternative to Kerberos ? May be Apache Knox + LDAP ?

I went to the link: indeed, it explains only the use of Kerberos.

[#4 · Accepted solution]

@Smart Data

Ranger can be used to sync users with LDAP/AD. Credentials are stored in LDAP/AD, and Ranger configured to access.

Knox is used as a proxy, but more for REST API service calls, and some UIs. It is not meant to proxy high volume traffic like Kafka messages.

[#5]

@Graham Martin

Thks for your expanation. I am going to install & to use Kerberos.

[#6]

@@Smart Data

If you intend to run a secure Hadop cluster then there is no way you can avoid Kerberos. Below are the difference between knox and kerberos.

The Apache Knox Gateway is a system that provides a single point of authentication and access. It provides the following features:

• Single REST API Access Point
• Centralized authentication, authorization and auditing for Hadoop REST/HTTP services
• LDAP/AD Authentication, Service Authorization and Audit
• Eliminates SSH edge node risks
• Hides Network Topology

LAYERS OF DEFENSE FOR A HADOOP CLUSTER

• Perimeter Level Security – Network Security, Apache Knox (gateway)
• Authentication : Kerberos
• Authorization
• OS Security : encryption of data in network and HDFS

Apache Knox can also access a Hadoop cluster over HTTP or HTTPS

CURRENT FEATURES OF APACHE KNOX

• Authenticate : by LDAP or Cloud SSO Provider
• Provides services for HDFS, HCat, HBase, Oozie, Hive, YARN, and Storm
• HTTP access for Hive over JDBC support is available (ODBC driver Support- In Future)

Hope that helps to explain.

[#7]

@@Smart Data

If you intend to run a secure Hadop cluster then there is no way you can avoid Kerberos. Below are the difference between knox and kerberos.

The Apache Knox Gateway is a system that provides a single point of authentication and access. It provides the following features:

• Single REST API Access Point
• Centralized authentication, authorization and auditing for Hadoop REST/HTTP services
• LDAP/AD Authentication, Service Authorization and Audit
• Eliminates SSH edge node risks
• Hides Network Topology

LAYERS OF DEFENSE FOR A HADOOP CLUSTER

• Perimeter Level Security – Network Security, Apache Knox (gateway)
• Authentication : Kerberos
• Authorization
• OS Security : encryption of data in network and HDFS

Apache Knox can also access a Hadoop cluster over HTTP or HTTPS

CURRENT FEATURES OF APACHE KNOX

• Authenticate : by LDAP or Cloud SSO Provider
• Provides services for HDFS, HCat, HBase, Oozie, Hive, YARN, and Storm
• HTTP access for Hive over JDBC support is available (ODBC driver Support- In Future)

Hope that helps to explain.

[#8]

@Geoffrey Shelton Okot thks for this explanation.