Reply
New Contributor
Posts: 1
Registered: ‎01-10-2017

Flume to Hive sink in secure hadoop

Hi,

 

I'm trying to use flume for transport data from kafka into hive table, but whatever I do still get well know error:

 

GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

 

I created keytab for flume user (via CM) and i can kinit to it without a problem. Sink is configured as follow:

 

gt_flume.sinks.sink2.kerberos = true
gt_flume.sinks.sink2.hive.kerberosPrincipal = $KERBEROS_PRINCIPAL
gt_flume.sinks.sink2.hive.kerberosKeytab = $KERBEROS_KEYTAB
gt_flume.sinks.sink2.type = hive
gt_flume.sinks.sink2.hive.metastore = thrift://host.domain:9083
gt_flume.sinks.sink2.hive.database =  hive_test
gt_flume.sinks.sink2.hive.table = kafka_test
gt_flume.sinks.sink2.serializer = DELIMITED
gt_flume.sinks.sink2.serializer.fieldnames = value1,value2
gt_flume.sinks.sink2.channel = channel1
gt_flume.sinks.sink2.idleTimeout = 0
gt_flume.sinks.sink2.callTimeout = 10000

 

I tried also add this, without success:

gt_flume.sinks.sink2.client-principal = $KERBEROS_PRINCIPAL
gt_flume.sinks.sink2.client-keytab = $KERBEROS_KEYTAB

 

Other variables set:

HIVE_HOME=/opt/cloudera/parcels/CDH-5.4.8-1.cdh5.4.8.p0.4/lib
HCAT_HOME=/opt/cloudera/parcels/CDH-5.4.8-1.cdh5.4.8.p0.4/lib/hive-hcatalog/
FLUME_CLASSPATH=/opt/cloudera/parcels/CDH-5.4.8-1.cdh5.4.8.p0.4/lib/hive/lib/hive-cli.jar

 

Writing directlly to hdfs works well, to HBASE table also, only hive fails. I found one jira: https://issues.apache.org/jira/browse/FLUME-2433

My hadoop environment wa installed in september 2015 (CDH 5.4) and I am not sure if it is already included in my flume version.

 

Any ideas? Shoud i finally upgrade CDH :) ?

 

Thanks,

Grzegorz

Highlighted
New Contributor
Posts: 2
Registered: ‎01-26-2017

Re: Flume to Hive sink in secure hadoop

Did you ev er get this working?  I'm running to the same issue....

Announcements