question Beeline uses incorrect kerberos principal for zookeeper in Support Questions

question Beeline uses incorrect kerberos principal for zookeeper in Support Questions https://community.cloudera.com/t5/Support-Questions/Beeline-uses-incorrect-kerberos-principal-for-zookeeper/m-p/326010#M229768 I am working on a project to enable multiple hadoop clusters to be managed on the same machines. The premise is to customize the kerberos principal names to include the name of the target cluster e.g. instead of `userX/server.com@REALM.COM` it would be `userX-clusterX/server.com@REALM.COM`. I have systematically adjusted the configuration of a test cluster so all kerberos principal names (including service principals and smoke users) follow this naming convention. Here is a sample of the KDC's `list_principals` output: ```[root@master-01 log]# kadmin -r REALM.COM -p ****/**** -w **** "list_principals" HTTP/master-01.com@REALM.COM admin/admin@REALM.COM dn-mycluster/worker-03.com@REALM.COM hive-mycluster/master-01.com@REALM.COM jn-mycluster/master-01.com@REALM.COM krbtgt/REALM.COM@REALM.COM nn-mycluster/master-02.com@REALM.COM rangerlookup-mycluster/master-02.com@REALM.COM rm-mycluster/master-02.com@REALM.COM spark-mycluster/worker-03.com@REALM.COM zookeeper-mycluster/master-01.com@REALM.COM zookeeper-mycluster/master-02.com@REALM.COM ``` The cluster deployment works as expected for hadoop, yarn, zookeeper and ranger, but hive + beeline are failing to authenticate (though the installation finishes without issue). Beeline is unable to connect to hive because it attempts to connect to `zookeeper/server.com@REALM.COM` rather than `zookeeper-mycluster/server.com@REALM.COM`. I used the following commands to connect to the hiveserver2 via zookeeper: ``` su smoke_user kinit -kt ~/.ssh/smoke_user.principal.keytab smoke_user/master-02.com@REALM.COM /opt/hadoop/hive/bin/hive --config /etc/hive/conf.s2 --service beeline -u "jdbc:hive2://master-01.com:2181,master-02.com:2181,master-03.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;sslTrustStore=/etc/ssl/certs/truststore.jks;trustStorePassword=****" ``` ... and this is the error it throws: ``` Connecting to jdbc:hive2://master-01.com:2181,master-02.com:2181,master-03.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;sslTrustStore=/etc/ssl/certs/truststore.jks;trustStorePassword=**** Error: org.apache.hive.jdbc.ZooKeeperHiveClientException: Unable to read HiveServer2 configs from ZooKeeper (state=,code=0) ``` The /hiverserver2 znode has not been created but those of other services have been created (so I guess it's not a general problem with zookeeper but a specific one with hive + zookeeper). Suspecting a kerberos authentification problem, I saw in the `krb5kdc.log` that the incorrect zookeeper principal was being used by hive/beeline: ``` Oct 01 08:38:50 master-01.com krb5kdc[6796](info): TGS_REQ (4 etypes {18 17 16 23}) 192.168.32.15: LOOKING_UP_SERVER: authtime 0, hive-mycluster/master-03.com@REALM.COM for zookeeper/master-02.com@REALM.COM, Server not found in Kerberos database Oct 01 08:38:50 master-01.com krb5kdc[6796](info): TGS_REQ (4 etypes {18 17 16 23}) 192.168.32.15: LOOKING_UP_SERVER: authtime 0, hive-mycluster/master-03.com@REALM.COM for zookeeper/master-02.com@REALM.COM, Server not found in Kerberos database ``` The principal `zookeeper/master-02.com@REALM.COM` is being generated somewhere in the process of attempting to authenticate with the hiveserver2, but this principal is incorrect and follows a naming convention that this project has deviated from. It should match what is in the KDC for zookeeper: ``` [root@master-01 log]# kadmin -r REALM.COM -p ****/**** -w **** "list_principals" | grep zookeeper zookeeper-mycluster/master-01.com@REALM.COM zookeeper-mycluster/master-02.com@REALM.COM zookeeper-mycluster/master-03.com@REALM.COM ``` I have been pouring over the hive, zookeeper and kerberos configuration documentation without finding any parameter which would allow to set the zookeeper principal directly (would be nice if I'm wrong though). How can I force a specific zookeeper principal name to be used? Fri, 01 Oct 2021 13:59:22 GMT alxKd 2021-10-01T13:59:22Z