Apache Drill - Authentication failed: Server requires authentication using [kerberos, plain]. Insufficient credentials?

[#1]

I've setup Apace drill 1.10.0 on RHEL 6.7, HDP 2.5.3, kerberos enabled.

I'm getting below error while running "drill-conf" as user "drill" which is configured in the "drill-override.conf" file and it has valid keytab/ticket.

drill@host:/opt/drill/bin>  drill-conf
Error: Failure in connecting to Drill: org.apache.drill.exec.rpc.NonTransientRpcException: javax.security.sasl.SaslException: Authentication failed: Server requires authentication using [kerberos, plain]. Insufficient credentials? [Caused by javax.security.sasl.SaslException: Server requires authentication using [kerberos, plain]. Insufficient credentials?] (state=,code=0)
java.sql.SQLException: Failure in connecting to Drill: org.apache.drill.exec.rpc.NonTransientRpcException: javax.security.sasl.SaslException: Authentication failed: Server requires authentication using [kerberos, plain]. Insufficient credentials? [Caused by javax.security.sasl.SaslException: Server requires authentication using [kerberos, plain]. Insufficient credentials?]
        at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:166)
        at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:72)
        at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
        at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143)
        at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
        at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
        at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
        at sqlline.Commands.connect(Commands.java:1083)
        at sqlline.Commands.connect(Commands.java:1015)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
        at sqlline.SqlLine.dispatch(SqlLine.java:742)
        at sqlline.SqlLine.initArgs(SqlLine.java:528)
        at sqlline.SqlLine.begin(SqlLine.java:596)
        at sqlline.SqlLine.start(SqlLine.java:375)
        at sqlline.SqlLine.main(SqlLine.java:268)
Caused by: org.apache.drill.exec.rpc.NonTransientRpcException: javax.security.sasl.SaslException: Authentication failed: Server requires authentication using [kerberos, plain]. Insufficient credentials? [Caused by javax.security.sasl.SaslException: Server requires authentication using [kerberos, plain]. Insufficient credentials?]
        at org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:157)
        at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:432)
        at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:379)
        at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:157)
        ... 18 more
Caused by: javax.security.sasl.SaslException: Authentication failed: Server requires authentication using [kerberos, plain]. Insufficient credentials? [Caused by javax.security.sasl.SaslException: Server requires authentication using [kerberos, plain]. Insufficient credentials?]
        at org.apache.drill.exec.rpc.user.UserClient$3.mapException(UserClient.java:204)
        at org.apache.drill.exec.rpc.user.UserClient$3.mapException(UserClient.java:197)
        at com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:85)
        at org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:155)
        ... 21 more
Caused by: javax.security.sasl.SaslException: Server requires authentication using [kerberos, plain]. Insufficient credentials?
        at org.apache.drill.exec.rpc.user.UserClient.getAuthenticatorFactory(UserClient.java:285)
        at org.apache.drill.exec.rpc.user.UserClient.authenticate(UserClient.java:216)
        ... 22 more
apache drill 1.10.0
"this isn't your grandfather's sql"

Error from "sqlline.log":

2017-04-07 14:18:22,564 [main] WARN  o.a.drill.exec.metrics.DrillMetrics - Removing old metric since name matched newly registered metric. Metric name: drill.allocator.root.used
2017-04-07 14:18:22,565 [main] WARN  o.a.drill.exec.metrics.DrillMetrics - Removing old metric since name matched newly registered metric. Metric name: drill.allocator.root.peak
2017-04-07 14:18:22,615 [main] INFO  o.a.drill.common.config.DrillConfig - Configuration and plugin file(s) identified in 50ms.
Base Configuration:
        - jar:file:/opt/drill/jars/drill-common-1.10.0.jar!/drill-default.conf

Intermediate Configuration and Plugin files, in order of precedence:
        - jar:file:/opt/drill/jars/drill-java-exec-1.10.0.jar!/drill-module.conf
        - jar:file:/opt/drill/jars/drill-common-1.10.0.jar!/drill-module.conf
        - jar:file:/opt/drill/jars/drill-storage-hive-core-1.10.0.jar!/drill-module.conf
        - jar:file:/opt/drill/jars/drill-memory-base-1.10.0.jar!/drill-module.conf
        - jar:file:/opt/drill/jars/drill-mongo-storage-1.10.0.jar!/drill-module.conf
        - jar:file:/opt/drill/jars/drill-jdbc-storage-1.10.0.jar!/drill-module.conf
        - jar:file:/opt/drill/jars/drill-logical-1.10.0.jar!/drill-module.conf
        - jar:file:/opt/drill/jars/drill-storage-hbase-1.10.0.jar!/drill-module.conf
        - jar:file:/opt/drill/jars/drill-gis-1.10.0.jar!/drill-module.conf
        - jar:file:/opt/drill/jars/drill-kudu-storage-1.10.0.jar!/drill-module.conf
        - jar:file:/opt/drill/jars/drill-hive-exec-shaded-1.10.0.jar!/drill-module.conf

Override File: file:/opt/drill/conf/drill-override.conf

2017-04-07 14:18:22,625 [main] WARN  o.a.drill.exec.metrics.DrillMetrics - Removing old metric since name matched newly registered metric. Metric name: drill.allocator.root.used
2017-04-07 14:18:22,625 [main] WARN  o.a.drill.exec.metrics.DrillMetrics - Removing old metric since name matched newly registered metric. Metric name: drill.allocator.root.peak
2017-04-07 14:18:22,671 [main] ERROR o.a.drill.exec.client.DrillClient - Connection to host1.com:31010 failed with error javax.security.sasl.SaslException: Authentication failed: Server requires authentication using [kerberos, plain]. Insufficient credentials? [Caused by javax.security.sasl.SaslException: Server requires authentication using [kerberos, plain]. Insufficient credentials?]. Not retrying anymore

The drill web console is working with with PAM authentication. i was able to login with local user "drill"

Here is my "drill-override.conf" file:

drill.exec: {
  cluster-id: "drillbits1",
  zk.connect: "host1:2181,host2:2181,host3:2181",
  security: {
          user.auth.enabled: true,
          user.auth.impl: "pam",
          user.auth.pam_profiles: [ "sudo", "login" ],
          packages += "org.apache.drill.exec.rpc.user.security",
          auth.mechanisms: ["KERBEROS","PLAIN"],
          auth.principal: "drill/labhdp@LAB.COM",
          auth.keytab: "/opt/drill/.keytab/drill.keytab"
        }
}

Please help me resolve the issue.

[#2]

Apache has good documentation for this, you need to make sure you have correct kerberos versions.

[#3]

I'm using Kerberos 1.10.3.

krb5-server-1.10.3-42.el6.x86_64
krb5-libs-1.10.3-42.el6.x86_64
krb5-workstation-1.10.3-42.el6.x86_64

[#4]

@Neeraj Sabharwal

Please see if you can help me.