Posts: 18
Registered: ‎04-10-2014
Accepted Solution

Twitter stream using flume on a securized cluster

Hello, I am trying to configure my twitter agent for flume on a kerberized cluster.


I followed the security manual, adding :

agentName.sinks.sinkName.hdfs.kerberosPrincipal = flume/
agentName.sinks.sinkName.hdfs.kerberosKeytab = /etc/flume-ng/conf/flume.keytab


with my own values.

As Kerberos principal I created both flume@HADDOP.COM and flume/_HOST@HADOOP.COM


kadmin.local: ktadd -k /etc/flume-ng/conf/flume.keytab flume/
Entry for principal flume/ with kvno 2, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/flume-ng/conf/flume.keytab.
Entry for principal flume/ with kvno 2, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/etc/flume-ng/conf/flume.keytab


[root@evl2400469 ~]# kinit -p flume/
Password for flume/
[root@evl2400469 ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: flume/


[root@evl2400469 ~]# ls -l /etc/flume-ng/conf/
total 16
-rw-r--r-- 1 root root 0 Mar 28 08:14 flume.conf
-rw-r--r-- 1 root root 1661 Mar 28 08:14
-rw-r--r-- 1 root root 1197 Mar 28 08:14
-rw-r----- 1 root root 234 Jun 19 16:18 flume.keytab
-rw-r--r-- 1 root root 3074 Mar 28 08:14


Did I miss something in the configuration ?


I have this error:

Sink HDFS has been removed due to an error during configuration
java.lang.IllegalArgumentException: The keyTab file: /etc/flume-ng/conf/flume.keytab is nonexistent or can't read. Please specify a readable keytab file for Kerberos auth.
	at org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(
	at org.apache.flume.sink.hdfs.HDFSEventSink.configure(
	at org.apache.flume.conf.Configurables.configure(
	at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(
	at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(
	at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$
	at java.util.concurrent.Executors$
	at java.util.concurrent.FutureTask.runAndReset(
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(
	at java.util.concurrent.ScheduledThreadPoolExecutor$
	at java.util.concurrent.ThreadPoolExecutor.runWorker(
	at java.util.concurrent.ThreadPoolExecutor$

Thanks for helping me :)


Lefevre Kevin
Cloudera Employee
Posts: 68
Registered: ‎04-08-2014

Re: Twitter stream using flume on a securized cluster

Sounds like the user Flume is running as does not have UNIX permissions to read the keytab file. You have it chmod 0440 owned by root, so unless Flume is running as the root user, it will not be able to read that file.

Posts: 18
Registered: ‎04-10-2014

Re: Twitter stream using flume on a securized cluster

Yep, the keytab created had not the correct permission, I forgot it !

Lefevre Kevin
New solutions