Router error in Druid with Kerberos

[#1]

Hi All,

I have installed Druid (0.9.2) using Ambari (2.5.1.0) on a kerberized cluster. This error keeps popping up in the router.log file:

ERROR [CoordinatorRuleManager-Exec--0] io.druid.server.router.CoordinatorRuleManager - Exception while polling for rules java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.security.CompositeGroupsMapping not found

And I also saw this error in the overlord.log:

WARN org.apache.hadoop.security.authentication.server.AuthenticationFilter - AuthenticationToken ignored: org.apache.hadoop.security.authentication.util.SignerException: Invalid signature

Once the Druid is installed, does it require any other steps to get it work for a kerberized cluster?

TIA,

Kumar

[#2]

@Kumar Peddibhotla

You need to go through this druid setup in a kerberized environment if you haven't yet done so Druid-Kerberos.

After kerberisation check whether druid keytab was generated in /etc/security/keytabs/druid.keytab. If so then you will need to kinit the /etc/security/keytabs/druid.keytab. with the correct principal which you can obtain by running

# klist -kt /etc/security/keytabs/druid.keytab

The output could be something like this

$ klist -kt /etc/security/keytabs/druid.keytab 
Keytab name: FILE:/etc/security/keytabs/druid.keytab 
KVNO        Timestamp             Principal 
---- ----------------- -------------------------------
1          02/02/17 23:00:12      druid/FQDN@REALM 
1          02/02/17 23:00:12      druid/FQDN@REALM

To get a valid Kerberos ticket you will need to kinit kt $keytab $ principal

$ kinit -kt /etc/security/keytabs/druid.keytab druid/FQDN@REALM

Now the errors should disappear, please let me know if that helped

[#3]

Thank you, @Geoffrey Shelton Okot, for the reply. Yes, I have followed all of the steps, from the link you have provided, before, except for this: druid.hadoop.security.spnego.cookieSignatureSecret. I will try adding this and see if it will resolve the issue.

And the Druid principal looks slightly different in our cluster, druid@REALM.

[#4]

@Kumar Peddibhotla

Good I will wait for your feedback, the output of my REALM is just an example placeholder and for sure it won't match yours but the methodology is the same

[#5]

@Geoffrey Shelton Okot, it did not work. I have added all the missing configurations and did the kinit to the druid.headless.keytab on all the master nodes and still I am seeing the same error when I tried to run this command:

curl --negotiate -u:<USER> -b ~/cookies.txt -c ~/cookies.txt -X 'POST' -H 'Content-Type:application/json' -d @wikiticker-index.json <OVERLORD_HOST>:8090/druid/indexer/v1/task

[#6]

@Kumar Peddibhotla

Please can you paste here how you proceeded step by step so I can validate, please remember to obscure ONLY import inputs like IP, REALM ,HOSTNAME etc

It's important to see and hence be able to debug. Did you add Druid as a service with Ambari, how can I reproduce your errot?

[#7]

Yes, I have added the Druid service through Ambari. Initially, the Druid keytabs were missing and then I had to regenerate them and then the services came up with no other issues. But I saw the error message related to the missing CompositeGrousMapping class in the router.log file. I have even tried to copy the hadoop-common.jar to druid/lib, the error went way, but the services are not staying up.I have also followed the link you provided to make sure all the config settings are in there, it was missing druid.hadoop.security.spnego.cookieSignatureSecret, which I have added and restarted Druid.

Thanks a lot for all the help, @Geoffrey Shelton Okot

[#8 · Accepted solution]

The issue was resolved by disabling Kerberos authentication for Druid and also by fixing Broker host and Broker port values in the Superset console for the Druid cluster. Thank you, @Nishant Bangarwa, for all the help.