Reply
New Contributor
Posts: 1
Registered: ‎01-10-2017

Flume to Hive sink in secure hadoop

Hi,

 

I'm trying to use flume for transport data from kafka into hive table, but whatever I do still get well know error:

 

GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

 

I created keytab for flume user (via CM) and i can kinit to it without a problem. Sink is configured as follow:

 

gt_flume.sinks.sink2.kerberos = true
gt_flume.sinks.sink2.hive.kerberosPrincipal = $KERBEROS_PRINCIPAL
gt_flume.sinks.sink2.hive.kerberosKeytab = $KERBEROS_KEYTAB
gt_flume.sinks.sink2.type = hive
gt_flume.sinks.sink2.hive.metastore = thrift://host.domain:9083
gt_flume.sinks.sink2.hive.database =  hive_test
gt_flume.sinks.sink2.hive.table = kafka_test
gt_flume.sinks.sink2.serializer = DELIMITED
gt_flume.sinks.sink2.serializer.fieldnames = value1,value2
gt_flume.sinks.sink2.channel = channel1
gt_flume.sinks.sink2.idleTimeout = 0
gt_flume.sinks.sink2.callTimeout = 10000

 

I tried also add this, without success:

gt_flume.sinks.sink2.client-principal = $KERBEROS_PRINCIPAL
gt_flume.sinks.sink2.client-keytab = $KERBEROS_KEYTAB

 

Other variables set:

HIVE_HOME=/opt/cloudera/parcels/CDH-5.4.8-1.cdh5.4.8.p0.4/lib
HCAT_HOME=/opt/cloudera/parcels/CDH-5.4.8-1.cdh5.4.8.p0.4/lib/hive-hcatalog/
FLUME_CLASSPATH=/opt/cloudera/parcels/CDH-5.4.8-1.cdh5.4.8.p0.4/lib/hive/lib/hive-cli.jar

 

Writing directlly to hdfs works well, to HBASE table also, only hive fails. I found one jira: https://issues.apache.org/jira/browse/FLUME-2433

My hadoop environment wa installed in september 2015 (CDH 5.4) and I am not sure if it is already included in my flume version.

 

Any ideas? Shoud i finally upgrade CDH :) ?

 

Thanks,

Grzegorz

New Contributor
Posts: 2
Registered: ‎01-26-2017

Re: Flume to Hive sink in secure hadoop

Did you ev er get this working?  I'm running to the same issue....

New Contributor
Posts: 1
Registered: ‎06-04-2019

Re: Flume to Hive sink in secure hadoop

[ Edited ]

does flume really support kerberos for hive sink?

Cloudera Employee
Posts: 277
Registered: ‎01-09-2014

Re: Flume to Hive sink in secure hadoop

The flume hive sink isn't supported on the CDH platform:

https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_rn_flume_ki.html#topic_1_...

it would be recommended to just use the hdfs sink, and then configure hive to use the partitions/directories that are created by flume.

-pd