Created on 10-16-2017 06:51 PM - edited 09-16-2022 08:47 AM
Hi All,
I have installed Druid (0.9.2) using Ambari (2.5.1.0) on a kerberized cluster. This error keeps popping up in the router.log file:
ERROR [CoordinatorRuleManager-Exec--0] io.druid.server.router.CoordinatorRuleManager - Exception while polling for rules java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.security.CompositeGroupsMapping not found
And I also saw this error in the overlord.log:
WARN org.apache.hadoop.security.authentication.server.AuthenticationFilter - AuthenticationToken ignored: org.apache.hadoop.security.authentication.util.SignerException: Invalid signature
Once the Druid is installed, does it require any other steps to get it work for a kerberized cluster?
TIA,
Kumar
Created 10-24-2017 01:04 PM
The issue was resolved by disabling Kerberos authentication for Druid and also by fixing Broker host and Broker port values in the Superset console for the Druid cluster. Thank you, @Nishant Bangarwa, for all the help.
Created 10-16-2017 08:00 PM
You need to go through this druid setup in a kerberized environment if you haven't yet done so Druid-Kerberos.
After kerberisation check whether druid keytab was generated in /etc/security/keytabs/druid.keytab. If so then you will need to kinit the /etc/security/keytabs/druid.keytab. with the correct principal which you can obtain by running
# klist -kt /etc/security/keytabs/druid.keytab
The output could be something like this
$ klist -kt /etc/security/keytabs/druid.keytab Keytab name: FILE:/etc/security/keytabs/druid.keytab KVNO Timestamp Principal ---- ----------------- ------------------------------- 1 02/02/17 23:00:12 druid/FQDN@REALM 1 02/02/17 23:00:12 druid/FQDN@REALM
To get a valid Kerberos ticket you will need to kinit kt $keytab $ principal
$ kinit -kt /etc/security/keytabs/druid.keytab druid/FQDN@REALM
Now the errors should disappear, please let me know if that helped
Created 10-16-2017 08:28 PM
Thank you, @Geoffrey Shelton Okot, for the reply. Yes, I have followed all of the steps, from the link you have provided, before, except for this: druid.hadoop.security.spnego.cookieSignatureSecret. I will try adding this and see if it will resolve the issue.
And the Druid principal looks slightly different in our cluster, druid@REALM.
Created 10-17-2017 06:45 AM
Good I will wait for your feedback, the output of my REALM is just an example placeholder and for sure it won't match yours but the methodology is the same
Created 10-17-2017 12:10 PM
@Geoffrey Shelton Okot, it did not work. I have added all the missing configurations and did the kinit to the druid.headless.keytab on all the master nodes and still I am seeing the same error when I tried to run this command:
curl --negotiate -u:<USER> -b ~/cookies.txt -c ~/cookies.txt -X 'POST' -H 'Content-Type:application/json' -d @wikiticker-index.json <OVERLORD_HOST>:8090/druid/indexer/v1/task
Created 10-17-2017 02:18 PM
Please can you paste here how you proceeded step by step so I can validate, please remember to obscure ONLY import inputs like IP, REALM ,HOSTNAME etc
It's important to see and hence be able to debug. Did you add Druid as a service with Ambari, how can I reproduce your errot?
Created 10-17-2017 02:34 PM
Yes, I have added the Druid service through Ambari. Initially, the Druid keytabs were missing and then I had to regenerate them and then the services came up with no other issues. But I saw the error message related to the missing CompositeGrousMapping class in the router.log file. I have even tried to copy the hadoop-common.jar to druid/lib, the error went way, but the services are not staying up.I have also followed the link you provided to make sure all the config settings are in there, it was missing druid.hadoop.security.spnego.cookieSignatureSecret, which I have added and restarted Druid.
Thanks a lot for all the help, @Geoffrey Shelton Okot
Created 10-24-2017 01:04 PM
The issue was resolved by disabling Kerberos authentication for Druid and also by fixing Broker host and Broker port values in the Superset console for the Druid cluster. Thank you, @Nishant Bangarwa, for all the help.