Flume with Secured Kafka Channel in HDP 2.5

[#1]

Caveat: This feature has been validated by HWX engineer manually but we don't officially support it at the moment.

Environment:

• HDP-2.5.3.0-37
• Ambari-2.4.2.0-136
• JDK1.8
• Kerberos enabled
• Ranger enabled
• Due to security limitations, we can only launch flume agent processes in Ambari.

STEP 1: Create/modify flume configuration file.

Ambari -> Flume -> Configs -> flume.conf

# Flume agent config

#### Global ####
demo.sources = logtcp logudp
demo.channels = kafka_channel
demo.sinks = sink

#### Sources ####
demo.sources.logtcp.type = multiport_syslogtcp
demo.sources.logtcp.ports = 9515
demo.sources.logtcp.host = 0.0.0.0
demo.sources.logtcp.keepFields = true
demo.sources.logtcp.selector.type=replicating
demo.sources.logtcp.channels= kafka_channel

demo.sources.logudp.type = syslogudp
demo.sources.logudp.port = 9515
demo.sources.logudp.host = 0.0.0.0
demo.sources.logudp.keepFields = true
demo.sources.logudp.selector.type=replicating
demo.sources.logudp.channels = kafka_channel

#### Sinks ####
demo.sinks.sink.type = logger
demo.sinks.sink.channel = kafka_channel

#### Channels ####
demo.channels.kafka_channel.type = org.apache.flume.channel.kafka.KafkaChannel
demo.channels.kafka_channel.kafka.bootstrap.servers = node1.vxu.com:6667,node2.vxu.com:6667,node3.vxu.com:6667
demo.channels.kafka_channel.kafka.topic = flume_topic
demo.channels.kafka_channel.kafka.producer.security.protocol = SASL_PLAINTEXT
demo.channels.kafka_channel.kafka.producer.sasl.mechanism = GSSAPI
demo.channels.kafka_channel.kafka.consumer.security.protocol = SASL_PLAINTEXT
demo.channels.kafka_channel.kafka.consumer.sasl.mechanism = GSSAPI

STEP 2: Add kafka jaas file(s)

Create a flume_kafka_jaas.conf in /etc/flume/conf/:

KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    serviceName="kafka"
    keyTab="/etc/security/keytabs/kafka.service.keytab"
    principal="kafka/[email protected]";
};

STEP 3: Modify flume-env template

Ambari -> Flume -> Configs -> Advanced flume-env -> flume-env template

...
# Enviroment variables can be set here. 
export JAVA_HOME={{java_home}} 
# Give Flume more memory and pre-allocate, enable remote monitoring via JMX 
export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote -Dflume.monitoring.type=http -Dflume.monitoring.port=34545 -Djava.security.auth.login.config=/etc/flume/conf/flume_kafka_jaas.conf" 
# Note that the Flume conf directory is always included in the classpath. 
# Add flume sink to classpath 
if [ -e "/usr/lib/flume/lib/ambari-metrics-flume-sink.jar" ]; then
  export FLUME_CLASSPATH=$FLUME_CLASSPATH:/usr/lib/flume/lib/ambari-metrics-flume-sink.jar
fi 
export HIVE_HOME={{flume_hive_home}} 
export HCAT_HOME={{flume_hcat_home}}

Note:

After changing the flume configs, you need to clear the /etc/flume/conf/demo directory and kill all previous flume agent processes. Otherwise, new configs may not take effect.

[#2]

Excellent write up Victor, this definitely helps