@ambari275  I have gone through the logs and here are my observations Error: WARNING: A HTTP GET method, public javax.ws.rs.core.Response org.apache.ambari.server.api.services.ExtensionsService.getExtensionVersions(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String), should not consume any entity. Solution: To fix the issue: # cat /etc/ambari-server/conf/ambari.properties | grep client.threadpool.size.max client.threadpool.size.max=25 The client.threadpool.size.max property indicates a number of parallel threads servicing client requests. To find the number of cores on the server, issue Linux command nproc # nproc 25 1) Edit /etc/ambari-server/conf/ambari.properties file and change the default value of client.threadpool.size.max to have the number of cores on your machine. client.threadpool.size.max=25 2) Restart ambari-server # ambari-server restart Error 2021-07-23 12:43:42,673 WARN [Stack Version Loading Thread] RepoVdfCallable:142 - Could not load version definition for HDP-3.0 identified by https://archive.cloudera.com/p/HDP/centos7/3.x/3.0.1.0/HDP-3.0.1.0-187.xml. Server returned HTTP response code: 401 for URL: https://archive.cloudera.com/p/HDP/centos7/3.x/3.0.1.0/HDP-3.0.1.0-187.xml java.io.IOException: Server returned HTTP response code: 401 for URL: https://archive.cloudera.com/p/HDP/centos7/3.x/3.0.1.0/HDP-3.0.1.0-187.xml Reason: 401 means "Unauthorized", so there must be something with your credentials this is purely an authorization issue. It seems your access to the HDP repos is an issue. Your krb5.conf should look something like this  # cat /etc/krb5.conf # Configuration snippets may be placed in this directory as well includedir /etc/krb5.conf.d/ [logging] default = FILE:/var/log/krb5libs.log kdc = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log [libdefaults] dns_lookup_realm = false ticket_lifetime = 24h renew_lifetime = 7d forwardable = true rdns = false default_realm = DOMAIN.COM default_ccache_name = KEYRING:persistent:%{uid} [realms] DOMAIN.COM = { kdc = [FQDN 10.1.1.150] admin_server =[FQDN 10.1.1.150] } [domain_realm] .domain.com = DOMAIN.COM domain.com = DOMAIN.COM Your /etc/host I think I remember once having issues with hostnames with - try using host1 for ESXI-host2 etc and please don't comment out the IPV6 entry it can cause network connectivity issue so please remove the be # on the second line x.x.x.x localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 x.x.x.x FQDN server x.x.x.x host1 x.x.x.x host2 x.x.x.x host3 Kerberos service uses DNS to resolve hostnames. Therefore, DNS must be enabled on all hosts. With DNS, the principal must contain the fully qualified domain name (FQDN) of each host. For example, if the hostname is host1, the DNS domain name is domain.com, and the realm name is DOMAIN.COM, then the principal name for the host would be host/host1.domain.com@DOMAIN.COM. The examples in this guide require that DNS is configured and that the FQDN is used for each host. Also, ensure ambari agents is installed on all hosts including the ambari-server! Ensure on all the hosts the hostname point to the Ambari server [server] hostname=<FQDN_oF_Ambari_server> url_port=8440 secured_url_port=8441 connect_retry_delay=10 max_reconnect_retry_delay=30 Please revert     ... View more

@USMAN_HAIDER  There is this step below did you perform that? Kerberos must be specified as the security mechanism for Hadoop infrastructure, starting with the HDFS service. Enable Cloudera Manager Server security for the cluster on an HDFS service. After you do so, the Cloudera Manager Server automatically enables Hadoop security on the MapReduce and YARN services associated with that HDFS service. In the Cloudera Manager Admin Console: Select Clusters > HDFS-n. 1.Click the Configuration tab. 2.Select HDFS-n for the Scope filter. 3.Select Security for the Category filter. 4.Scroll (or search) to find the Hadoop Secure Authentication property. 5.Click the Kerberos button to select Kerberos: Please revert             ... View more

@ambari275  You can set up the kerberos server anywhere on the network provided it can be accessed by the hosts in your cluster. I suspect there is d^something wrong with yor Ambari server. Can you share your /var/log/ambari-server/ambari-server.log I asked for a couple of files but you only shared the krb5.conf. I will need the rest of the files to be able to understand and determine what could be the issue. Can describe your setup? Number of Nodes,network, OS etc ... View more

@ambari275  From the onset, I see you left the defaults and I doubt whether that really maps to your cluster. Here is a list of outputs I need to validate $ hostname -f [Where you installed the kerberos server] /etc/hosts /var/kerberos/krb5kdc/kadm5.acl /var/kerberos/krb5kdc/kdc.conf On the Kerberos server can you run # kadmin.local Then list_principals q to quit The hostname -f output on the Kerberos server should replace kdc and admin_server in krb5.conf Here is an example OS: Centos 7 Cluster Realm HOTEL.COM My hosts entry is for a class C network so yours could be different but your host name must be resolved by DNS [root@test ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.0.153 test.hotel.com test [root@test ~]# hostname -f test.hotel.com [root@test ~]# cat /var/kerberos/krb5kdc/kadm5.acl */admin@HOTEL.COM * [root@test ~]# cat /etc/krb5.conf # Configuration snippets may be placed in this directory as well includedir /etc/krb5.conf.d/ [logging] default = FILE:/var/log/krb5libs.log kdc = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log [libdefaults] dns_lookup_realm = false ticket_lifetime = 24h renew_lifetime = 7d forwardable = true rdns = false default_realm = HOTEL.COM default_ccache_name = KEYRING:persistent:%{uid} [realms] HOTEL.COM = { kdc = test.hotel.com admin_server =test.hotel.com } [domain_realm] .hotel.com = HOTEL.COM hotel.com = HOTEL.COM [root@test ~]# cat /var/kerberos/krb5kdc/kdc.conf [kdcdefaults] kdc_ports = 88 kdc_tcp_ports = 88 [realms] HOTEL.COM = { #master_key_type = aes256-cts acl_file = /var/kerberos/krb5kdc/kadm5.acl dict_file = /usr/share/dict/words admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal } [realms] HOTEL.COM = { master_key_type = des-cbc-crc database_name = /var/kerberos/krb5kdc/principal admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab supported_enctypes = des-cbc-crc:normal des3-cbc-raw:normal des3-cbc-sha1:norm al des-cbc-crc:v4 des-cbc-crc:afs3 kadmind_port = 749 acl_file = /var/kerberos/krb5kdc/kadm5.acl dict_file = /usr/dict/words } Once you share the above then I could figure out where the issue could be. Happy hadooping     ... View more

@USMAN_HAIDER  When you create a new Principal in the slave KDC you should also have a  crontab  that will propagate it to the master #!/bin/sh #/var/kerberos/kdc-master-propogate.sh kdclist = "slave-kdc.customer.com" /sbin/kdb5_util dump /usr/local/var/krb5kdc/master_datatrans for kdc in $kdclist do /sbin/kprop -f /usr/local/var/krb5kdc/master_datatrans $kdc done This way the  principals will be sync'ed ... View more

@mike_bronson7    Are you using the default capacity schedule settings? No queues/leafs created?  Is what you shared the current seeting? ... View more

@srinivasp    I am wondering whether your Ranger policies are also in place.  Please explicitly give the correct permissions to the group/user in Ranger as the beeline authorization depends now on Ranger 🙂     Happy hadooping.  ... View more

@enirys  That's correct to successfully set up an HMS  HA you  MUST ensure the metadata DB should have followed the steps mention in this official document: Configuring High Availability for the Hive Metastore High Availability for Hive Metastore That's should help you sort of the stale metadata issue ... View more

@SparkNewbie  Bingo you are using the derby DB, which is only recommended for testing.   There are three modes for Hive Metastore deployment: Embedded Metastore Local Metastore Remote Metastore In Hive by default, metastore service runs in the same JVM as the Hive service. It uses embedded derby database stored on the local file system in this mode. Thus both metastore service and hive service runs in the same JVM by using embedded Derby Database. But, this mode also has its limitation that, as only one embedded Derby database can access the database files on disk at any one time, so only one Hive session could be open at a time. 21/07/07 23:07:56 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY Derby is an embedded relational database in Java program and used for online transaction processing and has a 3.5 MB disk-space footprint. Depending on your software HDP or Cloudera ensure the hive DB is plugged to an external Mysql database For CDH using Mysql  For HDP using mysql  Check your current hive UI backend metadata databases !! After installing MySQL then you should toggle hive config to point to the external Mysql database .. Once done your commands and the refresh should succeed Please let me know if you need help Happy hadooping   ... View more