Twitter stream using flume on a securized cluster

[#1]

Hello, I am trying to configure my twitter agent for flume on a kerberized cluster.

I followed the security manual, adding :

agentName.sinks.sinkName.hdfs.kerberosPrincipal = flume/[email protected]
agentName.sinks.sinkName.hdfs.kerberosKeytab = /etc/flume-ng/conf/flume.keytab

with my own values.

As Kerberos principal I created both [email protected] and flume/[email protected]

kadmin.local: ktadd -k /etc/flume-ng/conf/flume.keytab flume/[email protected]
Entry for principal flume/[email protected] with kvno 2, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/flume-ng/conf/flume.keytab.
Entry for principal flume/[email protected] with kvno 2, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/etc/flume-ng/conf/flume.keytab

[root@evl2400469 ~]# kinit -p flume/[email protected]
Password for flume/[email protected]:
[root@evl2400469 ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: flume/[email protected]

[root@evl2400469 ~]# ls -l /etc/flume-ng/conf/
total 16
-rw-r--r-- 1 root root 0 Mar 28 08:14 flume.conf
-rw-r--r-- 1 root root 1661 Mar 28 08:14 flume-conf.properties.template
-rw-r--r-- 1 root root 1197 Mar 28 08:14 flume-env.sh.template
-rw-r----- 1 root root 234 Jun 19 16:18 flume.keytab
-rw-r--r-- 1 root root 3074 Mar 28 08:14 log4j.properties

Did I miss something in the configuration ?

I have this error:

Sink HDFS has been removed due to an error during configuration
java.lang.IllegalArgumentException: The keyTab file: /etc/flume-ng/conf/flume.keytab is nonexistent or can't read. Please specify a readable keytab file for Kerberos auth.
	at org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.java:542)
	at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:247)
	at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
	at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:418)
	at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
	at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)

Thanks for helping me 🙂

[#2 · Accepted solution]

Sounds like the user Flume is running as does not have UNIX permissions to read the keytab file. You have it chmod 0440 owned by root, so unless Flume is running as the root user, it will not be able to read that file.

[#3]

Yep, the keytab created had not the correct permission, I forgot it !