Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Flume to Hive sink in secure hadoop

Flume to Hive sink in secure hadoop

New Contributor

Hi,

 

I'm trying to use flume for transport data from kafka into hive table, but whatever I do still get well know error:

 

GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

 

I created keytab for flume user (via CM) and i can kinit to it without a problem. Sink is configured as follow:

 

gt_flume.sinks.sink2.kerberos = true
gt_flume.sinks.sink2.hive.kerberosPrincipal = $KERBEROS_PRINCIPAL
gt_flume.sinks.sink2.hive.kerberosKeytab = $KERBEROS_KEYTAB
gt_flume.sinks.sink2.type = hive
gt_flume.sinks.sink2.hive.metastore = thrift://host.domain:9083
gt_flume.sinks.sink2.hive.database =  hive_test
gt_flume.sinks.sink2.hive.table = kafka_test
gt_flume.sinks.sink2.serializer = DELIMITED
gt_flume.sinks.sink2.serializer.fieldnames = value1,value2
gt_flume.sinks.sink2.channel = channel1
gt_flume.sinks.sink2.idleTimeout = 0
gt_flume.sinks.sink2.callTimeout = 10000

 

I tried also add this, without success:

gt_flume.sinks.sink2.client-principal = $KERBEROS_PRINCIPAL
gt_flume.sinks.sink2.client-keytab = $KERBEROS_KEYTAB

 

Other variables set:

HIVE_HOME=/opt/cloudera/parcels/CDH-5.4.8-1.cdh5.4.8.p0.4/lib
HCAT_HOME=/opt/cloudera/parcels/CDH-5.4.8-1.cdh5.4.8.p0.4/lib/hive-hcatalog/
FLUME_CLASSPATH=/opt/cloudera/parcels/CDH-5.4.8-1.cdh5.4.8.p0.4/lib/hive/lib/hive-cli.jar

 

Writing directlly to hdfs works well, to HBASE table also, only hive fails. I found one jira: https://issues.apache.org/jira/browse/FLUME-2433

My hadoop environment wa installed in september 2015 (CDH 5.4) and I am not sure if it is already included in my flume version.

 

Any ideas? Shoud i finally upgrade CDH :) ?

 

Thanks,

Grzegorz

3 REPLIES 3

Re: Flume to Hive sink in secure hadoop

New Contributor

Did you ev er get this working?  I'm running to the same issue....

Re: Flume to Hive sink in secure hadoop

New Contributor

does flume really support kerberos for hive sink?

Re: Flume to Hive sink in secure hadoop

Super Collaborator
The flume hive sink isn't supported on the CDH platform:

https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_rn_flume_ki.html#topic_1_...

it would be recommended to just use the hdfs sink, and then configure hive to use the partitions/directories that are created by flume.

-pd