Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (2)
Contributor

Caveat: This feature has been validated by HWX engineer manually but we don't officially support it at the moment.

Environment:

  • HDP-2.5.3.0-37
  • Ambari-2.4.2.0-136
  • JDK1.8
  • Kerberos enabled
  • Ranger enabled
  • Due to security limitations, we can only launch flume agent processes in Ambari.

STEP 1: Create/modify flume configuration file.

Ambari -> Flume -> Configs -> flume.conf

# Flume agent config

#### Global ####
demo.sources = logtcp logudp
demo.channels = kafka_channel
demo.sinks = sink

#### Sources ####
demo.sources.logtcp.type = multiport_syslogtcp
demo.sources.logtcp.ports = 9515
demo.sources.logtcp.host = 0.0.0.0
demo.sources.logtcp.keepFields = true
demo.sources.logtcp.selector.type=replicating
demo.sources.logtcp.channels= kafka_channel

demo.sources.logudp.type = syslogudp
demo.sources.logudp.port = 9515
demo.sources.logudp.host = 0.0.0.0
demo.sources.logudp.keepFields = true
demo.sources.logudp.selector.type=replicating
demo.sources.logudp.channels = kafka_channel

#### Sinks ####
demo.sinks.sink.type = logger
demo.sinks.sink.channel = kafka_channel

#### Channels ####
demo.channels.kafka_channel.type = org.apache.flume.channel.kafka.KafkaChannel
demo.channels.kafka_channel.kafka.bootstrap.servers = node1.vxu.com:6667,node2.vxu.com:6667,node3.vxu.com:6667
demo.channels.kafka_channel.kafka.topic = flume_topic
demo.channels.kafka_channel.kafka.producer.security.protocol = SASL_PLAINTEXT
demo.channels.kafka_channel.kafka.producer.sasl.mechanism = GSSAPI
demo.channels.kafka_channel.kafka.consumer.security.protocol = SASL_PLAINTEXT
demo.channels.kafka_channel.kafka.consumer.sasl.mechanism = GSSAPI

STEP 2: Add kafka jaas file(s)

Create a flume_kafka_jaas.conf in /etc/flume/conf/:

KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    serviceName="kafka"
    keyTab="/etc/security/keytabs/kafka.service.keytab"
    principal="kafka/node1.vxu.com@VXU.COM";
};

STEP 3: Modify flume-env template

Ambari -> Flume -> Configs -> Advanced flume-env -> flume-env template

...
# Enviroment variables can be set here. 
export JAVA_HOME={{java_home}} 
# Give Flume more memory and pre-allocate, enable remote monitoring via JMX 
export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote -Dflume.monitoring.type=http -Dflume.monitoring.port=34545 -Djava.security.auth.login.config=/etc/flume/conf/flume_kafka_jaas.conf" 
# Note that the Flume conf directory is always included in the classpath. 
# Add flume sink to classpath 
if [ -e "/usr/lib/flume/lib/ambari-metrics-flume-sink.jar" ]; then
  export FLUME_CLASSPATH=$FLUME_CLASSPATH:/usr/lib/flume/lib/ambari-metrics-flume-sink.jar
fi 
export HIVE_HOME={{flume_hive_home}} 
export HCAT_HOME={{flume_hcat_home}}

Note:

After changing the flume configs, you need to clear the /etc/flume/conf/demo directory and kill all previous flume agent processes. Otherwise, new configs may not take effect.

1,525 Views
Comments
Not applicable

Excellent write up Victor, this definitely helps

Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
1 of 1
Last update:
‎02-28-2017 02:25 PM
Updated by:
Contributor vxu
 
Contributors
Top Kudoed Authors