New Contributor
Posts: 1
Registered: ‎01-10-2017

Flume to Hive sink in secure hadoop



I'm trying to use flume for transport data from kafka into hive table, but whatever I do still get well know error:


GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]


I created keytab for flume user (via CM) and i can kinit to it without a problem. Sink is configured as follow:


gt_flume.sinks.sink2.kerberos = true
gt_flume.sinks.sink2.hive.kerberosPrincipal = $KERBEROS_PRINCIPAL
gt_flume.sinks.sink2.hive.kerberosKeytab = $KERBEROS_KEYTAB
gt_flume.sinks.sink2.type = hive
gt_flume.sinks.sink2.hive.metastore = thrift://host.domain:9083
gt_flume.sinks.sink2.hive.database =  hive_test
gt_flume.sinks.sink2.hive.table = kafka_test
gt_flume.sinks.sink2.serializer = DELIMITED
gt_flume.sinks.sink2.serializer.fieldnames = value1,value2 = channel1
gt_flume.sinks.sink2.idleTimeout = 0
gt_flume.sinks.sink2.callTimeout = 10000


I tried also add this, without success:

gt_flume.sinks.sink2.client-principal = $KERBEROS_PRINCIPAL
gt_flume.sinks.sink2.client-keytab = $KERBEROS_KEYTAB


Other variables set:



Writing directlly to hdfs works well, to HBASE table also, only hive fails. I found one jira:

My hadoop environment wa installed in september 2015 (CDH 5.4) and I am not sure if it is already included in my flume version.


Any ideas? Shoud i finally upgrade CDH :) ?




New Contributor
Posts: 2
Registered: ‎01-26-2017

Re: Flume to Hive sink in secure hadoop

Did you ev er get this working?  I'm running to the same issue....

New Contributor
Posts: 1
Registered: ‎06-04-2019

Re: Flume to Hive sink in secure hadoop

[ Edited ]

does flume really support kerberos for hive sink?

Cloudera Employee
Posts: 277
Registered: ‎01-09-2014

Re: Flume to Hive sink in secure hadoop

The flume hive sink isn't supported on the CDH platform:

it would be recommended to just use the hdfs sink, and then configure hive to use the partitions/directories that are created by flume.