Reply
Highlighted
Explorer
Posts: 18
Registered: ‎04-10-2014
Accepted Solution

Twitter stream using flume on a securized cluster

Hello, I am trying to configure my twitter agent for flume on a kerberized cluster.

 

I followed the security manual, adding :

agentName.sinks.sinkName.hdfs.kerberosPrincipal = flume/fully.qualified.domain.name@YOUR-REALM.COM
agentName.sinks.sinkName.hdfs.kerberosKeytab = /etc/flume-ng/conf/flume.keytab

 

with my own values.

As Kerberos principal I created both flume@HADDOP.COM and flume/_HOST@HADOOP.COM

 

kadmin.local: ktadd -k /etc/flume-ng/conf/flume.keytab flume/evl2400469.eu.verio.net@HADOOP.COM
Entry for principal flume/evl2400469.eu.verio.net@HADOOP.COM with kvno 2, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/flume-ng/conf/flume.keytab.
Entry for principal flume/evl2400469.eu.verio.net@HADOOP.COM with kvno 2, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/etc/flume-ng/conf/flume.keytab

 

[root@evl2400469 ~]# kinit -p flume/evl2400469.eu.verio.net@HADOOP.COM
Password for flume/evl2400469.eu.verio.net@HADOOP.COM:
[root@evl2400469 ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: flume/evl2400469.eu.verio.net@HADOOP.COM

 

[root@evl2400469 ~]# ls -l /etc/flume-ng/conf/
total 16
-rw-r--r-- 1 root root 0 Mar 28 08:14 flume.conf
-rw-r--r-- 1 root root 1661 Mar 28 08:14 flume-conf.properties.template
-rw-r--r-- 1 root root 1197 Mar 28 08:14 flume-env.sh.template
-rw-r----- 1 root root 234 Jun 19 16:18 flume.keytab
-rw-r--r-- 1 root root 3074 Mar 28 08:14 log4j.properties

 

Did I miss something in the configuration ?

 

I have this error:

Sink HDFS has been removed due to an error during configuration
java.lang.IllegalArgumentException: The keyTab file: /etc/flume-ng/conf/flume.keytab is nonexistent or can't read. Please specify a readable keytab file for Kerberos auth.
	at org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.java:542)
	at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:247)
	at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
	at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:418)
	at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
	at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)

Thanks for helping me :)

 

--
Lefevre Kevin
Cloudera Employee
Posts: 70
Registered: ‎04-08-2014

Re: Twitter stream using flume on a securized cluster

Sounds like the user Flume is running as does not have UNIX permissions to read the keytab file. You have it chmod 0440 owned by root, so unless Flume is running as the root user, it will not be able to read that file.

Explorer
Posts: 18
Registered: ‎04-10-2014

Re: Twitter stream using flume on a securized cluster

Yep, the keytab created had not the correct permission, I forgot it !

--
Lefevre Kevin
Announcements
New solutions