Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to use Flume with Secured HBase?

How to use Flume with Secured HBase?

New Contributor

Tried to write data to Kerberised HBase by Flume.

Setup agent according to Flume docs, created principal and key tab for the user.

Kan get ticket from the command line, but Flume permanently gives error:

Java config name: null

Native config name: /etc/krb5.conf

Loaded from native config

Java config name: null

Native config name: /etc/krb5.conf

Loaded from native config

>>> KdcAccessibility: reset

>>> KdcAccessibility: reset

>>> KeyTabInputStream, readName(): CS.INTERSET.COM

>>> KeyTabInputStream, readName(): interset

>>> KeyTab: load() entry length: 78; type: 18

Looking for keys for: interset@CS.INTERSET.COM

Found unsupported keytype (18) for interset@CS.INTERSET.COM

17/10/04 19:44:46 ERROR lifecycle.LifecycleSupervisor: Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@56563488 counterGroup:{ name:null counters:{} } } - Exception follows.

org.apache.flume.FlumeException: Failed to login to HBase using provided credentials.

at org.apache.flume.sink.hbase.HBaseSink.start(HBaseSink.java:141)

at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)

at org.apache.flume.SinkRunner.start(SinkRunner.java:79)

at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.io.IOException: Login failure for interset@CS.INTERSET.COM from keytab /tmp/interset.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user

Here is my agent configuration:

#cat agent.conf

a1.sources = source1

a1.sinks = sink1

a1.channels = channel1

# Describe/configure the source

a1.sources.source1.type = seq

a1.sources.source1.channel = channel1

# Use a channel which buffers events in memory

a1.channels.channel1.type = memory

a1.channels.c1.capacity = 1000

a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel

a1.sources.source1.channels = channel1

a1.sinks.sink1.channel = channel1

# Describe the sink

a1.sinks.sink1.type = hbase

a1.sinks.sink1.table = test

a1.sinks.sink1.columnFamily = field1

a1.sinks.sink1.serializer = org.apache.flume.sink.hbase.RegexHbaseEventSerializer

a1.sinks.sink1.kerberosPrincipal = interset@CS.INTERSET.COM

a1.sinks.sink1.kerberosKeytab = /tmp/interset.keytab

a1.sinks.sink1.zookeeperQuorum = ip-172-30-4-33.ec2.internal:2181

a1.sinks.sink1.znodeParent = /hbase-secure

a1.sinks.sink1.channel = channel1

Manual ticket collection works fine:

$ kinit -kt /tmp/interset.keytab interset@CS.INTERSET.COM

$ klist -e

Ticket cache: FILE:/tmp/krb5cc_1001

Default principal: interset@CS.INTERSET.COM

Valid starting Expires Service principal

10/04/2017 19:56:24 10/05/2017 19:56:24 krbtgt/CS.INTERSET.COM@CS.INTERSET.COM

Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96

$

Any idea how to solve this?

2 REPLIES 2
Highlighted

Re: How to use Flume with Secured HBase?

Super Collaborator

Hi @Leonid Fedotov,

Looks thats an IO exception (Caused by: java.io.IOException:) can you please ensure to have at least read permission on the keytab file (/tmp/interset.keytab )for the user who is running the flume(for testing you may set "chmod +r /tmp/interset.keytab" or change the owner or group).

if the issue still presists,

Please ensure that you have the latest abase-site.xml file copied to the flume class path, and ensure to check the property hbase.security.authentication to "kerberos" highlighted here

To force the flume to use the same kerberos configuration file which you are using while on command line, set the following property in flume-env.sh

JAVA_OPTS="$JAVA_OPTS -Djava.security.krb5.conf=/tmp/krb5.conf"

PS : For security reasons it is not recommend to have the keytabs in /tmp as some one can have a copy of that cab be lead to identity misuse, so please have them in safe location with appropriate user and group permissions.

Hope that helps!!

Highlighted

Re: How to use Flume with Secured HBase?

Super Guru
@Leonid Fedotov

In your /etc/krb5.conf file, can you please check under [libdefaults ] your supported encryption types. Do they include one of the following (from your klist -e):

Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96

Don't have an account?
Coming from Hortonworks? Activate your account here