Caveat: This feature has been validated by HWX engineer manually but we don't officially support it at the moment. Environment: HDP-2.5.3.0-37 Ambari-2.4.2.0-136 JDK1.8 Kerberos enabled Ranger enabled Due to security limitations, we can only launch flume agent processes in Ambari. STEP 1: Create/modify flume configuration file. Ambari -> Flume -> Configs -> flume.conf # Flume agent config #### Global #### demo.sources = logtcp logudp demo.channels = kafka_channel demo.sinks = sink #### Sources #### demo.sources.logtcp.type = multiport_syslogtcp demo.sources.logtcp.ports = 9515 demo.sources.logtcp.host = 0.0.0.0 demo.sources.logtcp.keepFields = true demo.sources.logtcp.selector.type=replicating demo.sources.logtcp.channels= kafka_channel demo.sources.logudp.type = syslogudp demo.sources.logudp.port = 9515 demo.sources.logudp.host = 0.0.0.0 demo.sources.logudp.keepFields = true demo.sources.logudp.selector.type=replicating demo.sources.logudp.channels = kafka_channel #### Sinks #### demo.sinks.sink.type = logger demo.sinks.sink.channel = kafka_channel #### Channels #### demo.channels.kafka_channel.type = org.apache.flume.channel.kafka.KafkaChannel demo.channels.kafka_channel.kafka.bootstrap.servers = node1.vxu.com:6667,node2.vxu.com:6667,node3.vxu.com:6667 demo.channels.kafka_channel.kafka.topic = flume_topic demo.channels.kafka_channel.kafka.producer.security.protocol = SASL_PLAINTEXT demo.channels.kafka_channel.kafka.producer.sasl.mechanism = GSSAPI demo.channels.kafka_channel.kafka.consumer.security.protocol = SASL_PLAINTEXT demo.channels.kafka_channel.kafka.consumer.sasl.mechanism = GSSAPI STEP 2: Add kafka jaas file(s) Create a flume_kafka_jaas.conf in /etc/flume/conf/: KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true serviceName="kafka" keyTab="/etc/security/keytabs/kafka.service.keytab" principal="kafka/node1.vxu.com@VXU.COM"; }; STEP 3: Modify flume-env template Ambari -> Flume -> Configs -> Advanced flume-env -> flume-env template ... # Enviroment variables can be set here. export JAVA_HOME={{java_home}} # Give Flume more memory and pre-allocate, enable remote monitoring via JMX export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote -Dflume.monitoring.type=http -Dflume.monitoring.port=34545 -Djava.security.auth.login.config=/etc/flume/conf/flume_kafka_jaas.conf" # Note that the Flume conf directory is always included in the classpath. # Add flume sink to classpath if [ -e "/usr/lib/flume/lib/ambari-metrics-flume-sink.jar" ]; then export FLUME_CLASSPATH=$FLUME_CLASSPATH:/usr/lib/flume/lib/ambari-metrics-flume-sink.jar fi export HIVE_HOME={{flume_hive_home}} export HCAT_HOME={{flume_hcat_home}} Note: After changing the flume configs, you need to clear the /etc/flume/conf/demo directory and kill all previous flume agent processes. Otherwise, new configs may not take effect. ... View more

QUESTION: How are the Ambari-2.2.1.1 local accounts protected, such as "admin"? What about all the various component's configuration data managed by Ambari-2.2.1.1? ANSWER: Ambari local account credentials These are stored in the Ambari database as the SHA256 hash of the (randomly salted) password. Service configuration password properties These are stored in the Ambari database in blobs of JSON-formatted data in plaintext. When returned via API calls, the properties marked as passwords are masked and not displayed as plaintext. When sent to the agents, they are stored in plaintext in the command.json files stored in /var/lib/ambari-agent/data (readable only by root and the user that executes ambari-agent). Ambari-specific database and ldap credentials These are stored in plaintext in the ambari.properities file by default but can be encrypted via ambari-server setup-security. If encrypted, they are stored in a Java Keystore implementation (JCEKS) which uses 3DES in CBC mode with PKCS #5 padding to encrypt its keys. The master key for this keystore is either stored in plaintext on the Ambari server host, or query for when Ambari is started. ... View more