kafka-sentry -lr not working

[#1]

ERROR transport.SentryTransportPool: Failed to obtain transport for xxx.us.xxx.com:8038: null
19/08/08 07:12:27 WARN security.UserGroupInformation: PriviledgedActionException as:xxxx@xxx.xxx.COM (auth:KERBEROS) cause:org.apache.thrift.transport.TTransportException: Peer indicated failure: Problem with callback handler
19/08/08 07:12:27 ERROR transport.SentryTransportPool: Failed to obtain transport for xxxx.us.xxx.com:8038: null
19/08/08 07:12:30 ERROR tools.SentryShellKafka: Peer indicated failure: Problem with callback handler
org.apache.sentry.SentryUserException: Peer indicated failure: Problem with callback handler
at org.apache.sentry.core.common.transport.RetryClientInvocationHandler.connect(RetryClientInvocationHandler.java:166)
at org.apache.sentry.core.common.transport.RetryClientInvocationHandler.invokeImpl(RetryClientInvocationHandler.java:90)
at org.apache.sentry.core.common.transport.SentryClientInvocationHandler.invoke(SentryClientInvocationHandler.java:41)
at com.sun.proxy.$Proxy6.listAllRoles(Unknown Source)
at org.apache.sentry.provider.db.generic.tools.command.ListRolesCmd.execute(ListRolesCmd.java:43)
at org.apache.sentry.provider.db.generic.tools.SentryShellKafka.run(SentryShellKafka.java:83)
at org.apache.sentry.provider.db.tools.SentryShellCommon.executeShell(SentryShellCommon.java:262)
at org.apache.sentry.provider.db.generic.tools.SentryShellKafka.main(SentryShellKafka.java:96)
Caused by: org.apache.thrift.transport.TTransportException: Peer indicated failure: Problem with callback handler
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:199)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:307)
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at org.apache.sentry.core.common.transport.SentryTransportFactory$UgiSaslClientTransport.baseOpen(SentryTransportFactory.java:183)
at org.apache.sentry.core.common.transport.SentryTransportFactory$UgiSaslClientTransport.access$100(SentryTransportFactory.java:141)
at org.apache.sentry.core.common.transport.SentryTransportFactory$UgiSaslClientTransport$1.run(SentryTransportFactory.java:168)
at org.apache.sentry.core.common.transport.SentryTransportFactory$UgiSaslClientTransport$1.run(SentryTransportFactory.java:166)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.sentry.core.common.transport.SentryTransportFactory$UgiSaslClientTransport.open(SentryTransportFactory.java:166)
at org.apache.sentry.core.common.transport.SentryTransportFactory.connectToServer(SentryTransportFactory.java:99)
at org.apache.sentry.core.common.transport.SentryTransportFactory.getTransport(SentryTransportFactory.java:86)
at org.apache.sentry.core.common.transport.SentryTransportPool$PoolFactory.create(SentryTransportPool.java:302)
at org.apache.sentry.core.common.transport.SentryTransportPool$PoolFactory.create(SentryTransportPool.java:271)
at org.apache.commons.pool2.BaseKeyedPooledObjectFactory.makeObject(BaseKeyedPooledObjectFactory.java:62)
at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:1041)
at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:380)
at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:279)
at org.apache.sentry.core.common.transport.SentryTransportPool.getTransport(SentryTransportPool.java:181)
at org.apache.sentry.provider.db.generic.service.thrift.SentryGenericServiceClientDefaultImpl.connect(SentryGenericServiceClientDefaultImpl.java:91)
at org.apache.sentry.core.common.transport.RetryClientInvocationHandler.connect(RetryClientInvocationHandler.java:141)
... 7 more
The operation failed. Message: Peer indicated failure: Problem with callback handler

i tried with all possible scenarios 

with super users, kafka all users and i tried with kafka super.users setting on cloudera,

also add all usernames in sentry.service.admin.group and sentry.service.allow.connect

[#2]

It seems that you Sentry service is not aware of the Kafka brokers and fails to connect to the service.

You should check the following things:

• Check the property "Sentry Service" in your Kafka service configuration and ensure that the correct Sentry service is selected.
• Ensure the node from where you're running the kafka-sentry command has a Kafka Gateway role assigned to it.

After making the needed corrections, restart the stale services, deploy client configuration and try again.

Regards,

André

[#3]

Hi Araujo,

still same error.

yes i verified that below are the my configs and yes i assigned Kafka Gateway role as well.

[#4]

Do you have multiple Kafka and Sentry services on your cluster? Just wondering due to the numbering of those services (KAFKA-4 and SENTRY-2)?

[#5 · Accepted solution]

yes we have multiple Kafka clusters,

Issue was fixed.

solution:-

my username added with Caps on "sentry.service.allow.connect"

earlier i added with all small.