Twitter stream using flume on a securized cluster

[#1]

Hello, I am trying to configure my twitter agent for flume on a kerberized cluster.

I followed the security manual, adding :

agentName.sinks.sinkName.hdfs.kerberosPrincipal = flume/fully.qualified.domain.name@YOUR-REALM.COM
agentName.sinks.sinkName.hdfs.kerberosKeytab = /etc/flume-ng/conf/flume.keytab

with my own values.

As Kerberos principal I created both flume@HADDOP.COM and flume/_HOST@HADOOP.COM

kadmin.local: ktadd -k /etc/flume-ng/conf/flume.keytab flume/evl2400469.eu.verio.net@HADOOP.COM
Entry for principal flume/evl2400469.eu.verio.net@HADOOP.COM with kvno 2, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/flume-ng/conf/flume.keytab.
Entry for principal flume/evl2400469.eu.verio.net@HADOOP.COM with kvno 2, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/etc/flume-ng/conf/flume.keytab

[root@evl2400469 ~]# kinit -p flume/evl2400469.eu.verio.net@HADOOP.COM
Password for flume/evl2400469.eu.verio.net@HADOOP.COM:
[root@evl2400469 ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: flume/evl2400469.eu.verio.net@HADOOP.COM

[root@evl2400469 ~]# ls -l /etc/flume-ng/conf/
total 16
-rw-r--r-- 1 root root 0 Mar 28 08:14 flume.conf
-rw-r--r-- 1 root root 1661 Mar 28 08:14 flume-conf.properties.template
-rw-r--r-- 1 root root 1197 Mar 28 08:14 flume-env.sh.template
-rw-r----- 1 root root 234 Jun 19 16:18 flume.keytab
-rw-r--r-- 1 root root 3074 Mar 28 08:14 log4j.properties

Did I miss something in the configuration ?

I have this error:

Sink HDFS has been removed due to an error during configuration
java.lang.IllegalArgumentException: The keyTab file: /etc/flume-ng/conf/flume.keytab is nonexistent or can't read. Please specify a readable keytab file for Kerberos auth.
	at org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.java:542)
	at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:247)
	at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
	at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:418)
	at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
	at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)

Thanks for helping me 🙂

[#2 · Accepted solution]

Sounds like the user Flume is running as does not have UNIX permissions to read the keytab file. You have it chmod 0440 owned by root, so unless Flume is running as the root user, it will not be able to read that file.

[#3]

Yep, the keytab created had not the correct permission, I forgot it !