Support Questions
Find answers, ask questions, and share your expertise

Atlas : how to secure Kafka ?

Contributor

Hello,

I could access all topics on kafka without authentification.

My question : how I could secure access on kafka topics ?

Thks.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Atlas : how to secure Kafka ?

Expert Contributor

@Smart Data

Ranger can be used to sync users with LDAP/AD. Credentials are stored in LDAP/AD, and Ranger configured to access.

Knox is used as a proxy, but more for REST API service calls, and some UIs. It is not meant to proxy high volume traffic like Kafka messages.

View solution in original post

7 REPLIES 7

Re: Atlas : how to secure Kafka ?

Expert Contributor

@Smart Data

Atlas is more Governance related, security to a less extent.

You secure Kafka via Kerberos for authentication, and Ranger for authorization:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/index.html#bk_security

Re: Atlas : how to secure Kafka ?

Contributor

@Graham Martin

Thks for your quick reply. Is there an alternative to Kerberos ? May be Apache Knox + LDAP ?

I went to the link: indeed, it explains only the use of Kerberos.

Re: Atlas : how to secure Kafka ?

Expert Contributor

@Smart Data

Ranger can be used to sync users with LDAP/AD. Credentials are stored in LDAP/AD, and Ranger configured to access.

Knox is used as a proxy, but more for REST API service calls, and some UIs. It is not meant to proxy high volume traffic like Kafka messages.

View solution in original post

Re: Atlas : how to secure Kafka ?

Contributor

@Graham Martin

Thks for your expanation. I am going to install & to use Kerberos.

Re: Atlas : how to secure Kafka ?

Mentor

@@Smart Data

If you intend to run a secure Hadop cluster then there is no way you can avoid Kerberos. Below are the difference between knox and kerberos.

The Apache Knox Gateway is a system that provides a single point of authentication and access. It provides the following features:

  • Single REST API Access Point
  • Centralized authentication, authorization and auditing for Hadoop REST/HTTP services
  • LDAP/AD Authentication, Service Authorization and Audit
  • Eliminates SSH edge node risks
  • Hides Network Topology

LAYERS OF DEFENSE FOR A HADOOP CLUSTER

  • Perimeter Level Security – Network Security, Apache Knox (gateway)
  • Authentication : Kerberos
  • Authorization
  • OS Security : encryption of data in network and HDFS

Apache Knox can also access a Hadoop cluster over HTTP or HTTPS

CURRENT FEATURES OF APACHE KNOX

  • Authenticate : by LDAP or Cloud SSO Provider
  • Provides services for HDFS, HCat, HBase, Oozie, Hive, YARN, and Storm
  • HTTP access for Hive over JDBC support is available (ODBC driver Support- In Future)

Hope that helps to explain.

Re: Atlas : how to secure Kafka ?

Mentor

@@Smart Data

If you intend to run a secure Hadop cluster then there is no way you can avoid Kerberos. Below are the difference between knox and kerberos.

The Apache Knox Gateway is a system that provides a single point of authentication and access. It provides the following features:

  • Single REST API Access Point
  • Centralized authentication, authorization and auditing for Hadoop REST/HTTP services
  • LDAP/AD Authentication, Service Authorization and Audit
  • Eliminates SSH edge node risks
  • Hides Network Topology

LAYERS OF DEFENSE FOR A HADOOP CLUSTER

  • Perimeter Level Security – Network Security, Apache Knox (gateway)
  • Authentication : Kerberos
  • Authorization
  • OS Security : encryption of data in network and HDFS

Apache Knox can also access a Hadoop cluster over HTTP or HTTPS

CURRENT FEATURES OF APACHE KNOX

  • Authenticate : by LDAP or Cloud SSO Provider
  • Provides services for HDFS, HCat, HBase, Oozie, Hive, YARN, and Storm
  • HTTP access for Hive over JDBC support is available (ODBC driver Support- In Future)

Hope that helps to explain.

Re: Atlas : how to secure Kafka ?

Contributor

@Geoffrey Shelton Okot thks for this explanation.