Apologies, that is a wrong ticket. I should have changed it. I have updated it now. Previously, I had regenerated both keytabs and kerberos credentials many times but no luck. Also, after I manually kinit the kerberos ticket using kinit -k -t /var/run/cloudera-scm-agent/process/1546506889-hdfs-DATANODE/hdfs.keytab HTTP/host@EXAMPLE-REALM.COM,I was able to use curl command on the datanode web url (https://fqdn:9865) and got 200 ok response. However, it's seems like Cloudera isn't able to detect the credential for some reason. See response below.     [root@host 1546506889-hdfs-DATANODE]# curl -v -k --negotiate -u : https://host.com:9865 * About to connect() to host.com port 9865 (#0) * Trying xx.xx.xxx.xx... * Connected to host.com (xx.xx.xxx.xx) port 9865 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * skipping SSL peer certificate verification * SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 * Server certificate: * subject: CN=host.com,OU=Technology,O=xxxx plc,L=xl,ST=xl,C=GB * start date: Nov 14 14:41:16 2024 GMT * expire date: Nov 09 14:41:16 2025 GMT * common name: host.com * issuer: CN=host.com,OU=Technology,O=xxxx plc,L=xl,ST=xl,C=GB > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: host.com:9865 > Accept: */* > < HTTP/1.1 401 Authentication required < Connection: close < Pragma: no-cache < Strict_Transport_Security: max-age=0; includeSubDomains < X-Content-Type-Options: nosniff < X-FRAME-OPTIONS: SAMEORIGIN < X-XSS-Protection: 1; mode=block < Pragma: no-cache < Strict_Transport_Security: max-age=0; includeSubDomains < X-Content-Type-Options: nosniff < X-FRAME-OPTIONS: SAMEORIGIN < X-XSS-Protection: 1; mode=block < WWW-Authenticate: Negotiate < Set-Cookie: hadoop.auth=; Path=/; HttpOnly < Cache-Control: must-revalidate,no-cache,no-store < Content-Type: text/html;charset=iso-8859-1 < Content-Length: 447 < * Closing connection 0 * Issue another request to this URL: 'https://host.com:9865/' * About to connect() to host.com port 9865 (#1) * Trying xx.xx.xxx.xx... * Connected to host.com (xx.xx.xxx.xx) port 9865 (#1) * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * skipping SSL peer certificate verification * SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 * Server certificate: * subject: CN=host.com,OU=Technology,O=xxxx plc,L=xl,ST=xl,C=GB * start date: Nov 14 14:41:16 2024 GMT * expire date: Nov 09 14:41:16 2025 GMT * common name: host.com * issuer: CN=host.com,OU=Technology,O=xxxx plc,L=xl,ST=xl,C=GB * Server auth using GSS-Negotiate with user '' > GET / HTTP/1.1 > Authorization: Negotiate xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxg== > User-Agent: curl/7.29.0 > Host: host.com:9865 > Accept: */* > < HTTP/1.1 200 OK < Connection: close < Date: Mon, 18 Nov 2024 17:07:11 GMT < Cache-Control: no-cache < Expires: Mon, 18 Nov 2024 17:07:11 GMT < Date: Mon, 18 Nov 2024 17:07:11 GMT < Pragma: no-cache < Content-Type: text/html < Strict_Transport_Security: max-age=0; includeSubDomains < X-Content-Type-Options: nosniff < X-FRAME-OPTIONS: SAMEORIGIN < X-XSS-Protection: 1; mode=block < Expires: Mon, 18 Nov 2024 17:07:11 GMT < Date: Mon, 18 Nov 2024 17:07:11 GMT < Pragma: no-cache < Strict_Transport_Security: max-age=0; includeSubDomains < X-Content-Type-Options: nosniff < X-FRAME-OPTIONS: SAMEORIGIN < X-XSS-Protection: 1; mode=block < WWW-Authenticate: Negotiate xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx= < Set-Cookie: hadoop.auth="u=HTTP&p=HTTP/host.com.COM&t=kerberos&e=17xxxxxx7&s=CaYM+xxxxxxxxxxfBXleJ0K/ObFbrjALqy/R//g="; Path=/; HttpOnly < Last-Modified: Fri, 30 Aug 2024 16:14:30 GMT < Accept-Ranges: bytes < Content-Length: 1085 < <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta http-equiv="REFRESH" content="0;url=datanode.html" /> <title>Hadoop Administration</title> </head> * Closing connection 1 </html>[root@host 1546506889-hdfs-DATANODE]#     ... View more

I would appreciate any support from anyone ... View more

Thanks for getting back. The process_timestamp isn't there. It's not available on other running processes too. I had tried the work around, it didn't work, but I will give it another go. Another thing is the soft link for RegionServer process does not exist in /var/run/cloudera-scm-agent/supervisor/include directory. ... View more

      The issue was sorted after I reboot the host. I believe the reboot did the same things you mentioned above. I can start, datanode, impala daemon, and yarn. However, I am still unable to start hbase regionserver. I'm getting the following error. I believe it's something related to znode file not existing in the process directory.   + echo 'Adding HBoss JARs to HBase service classpath' + znode_cleanup regionserver + export 'HBASE_CLASSPATH=/opt/cloudera/cm/lib/plugins/event-publish-7.11.3-shaded.jar:/opt/cloudera/cm/lib/plugins/tt-instrumentation-7.11.3.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p3013.57035125/lib/hbase_filesystem/lib/*' + HBASE_CLASSPATH='/opt/cloudera/cm/lib/plugins/event-publish-7.11.3-shaded.jar:/opt/cloudera/cm/lib/plugins/tt-instrumentation-7.11.3.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p3013.57035125/lib/hbase_filesystem/lib/*' + exec /opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p3013.57035125/lib/hbase/../../bin/hbase --config /var/run/cloudera-scm-agent/process/1546503485-hbase-REGIONSERVER regionserver start ++ date + echo 'Tue 12 Nov 06:03:50 GMT 2024 Starting znode cleanup thread with HBASE_ZNODE_FILE=/var/run/cloudera-scm-agent/process/1546503485-hbase-REGIONSERVER/znode14618 for regionserver' ++ replace_pid -Djava.net.preferIPv4Stack=true ++ echo -Djava.net.preferIPv4Stack=true ++ sed 's#{{PID}}#14618#g' + HBASE_OPTS=-Djava.net.preferIPv4Stack=true + '[' jaas.conf '!=' '' ']' + export 'HBASE_OPTS=-Djava.security.auth.login.config=/var/run/cloudera-scm-agent/process/1546503485-hbase-REGIONSERVER/jaas.conf -Djava.net.preferIPv4Stack=true' + HBASE_OPTS='-Djava.security.auth.login.config=/var/run/cloudera-scm-agent/process/1546503485-hbase-REGIONSERVER/jaas.conf -Djava.net.preferIPv4Stack=true' + LOG_FILE=/var/run/cloudera-scm-agent/process/1546503485-hbase-REGIONSERVER/logs/znode_cleanup.log + set +x OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N /opt/cloudera/cm-agent/service/hbase/hbase.sh: line 234: kill: (14618) - No such process + RET=0 + '[' -f /var/run/cloudera-scm-agent/process/1546503485-hbase-REGIONSERVER/znode14618 ']' ++ date + echo 'Tue 12 Nov 06:03:56 GMT 2024 Znode file does not exist. No cleanup required.' + exit 0     Below is the agent log.   [12/Nov/2024 05:53:11 +0000] 1559 MainThread heartbeat_tracker INFO HB stats (seconds): num:43 LIFE_MIN:0.08 min:0.04 mean:0.06 max:0.11 LIFE_MAX:0.20 [12/Nov/2024 06:03:12 +0000] 1559 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.04 min:0.04 mean:0.07 max:0.11 LIFE_MAX:0.20 [12/Nov/2024 06:03:16 +0000] 1559 CP Server WorkerThread _cplogging INFO 127.0.0.1 - - [12/Nov/2024:06:03:16] "GET /heartbeat HTTP/1.1" 200 2 "" "python-requests/2.26.0" [12/Nov/2024 06:03:16 +0000] 1559 __run_queue process INFO [1546503323-hbase-REGIONSERVER] Updating process (remove). [12/Nov/2024 06:03:16 +0000] 1559 __run_queue process INFO [1546503323-hbase-REGIONSERVER] Deactivating process (skipped) [12/Nov/2024 06:03:16 +0000] 1559 __run_queue process INFO [1546503323-hbase-REGIONSERVER] stopping monitors [12/Nov/2024 06:03:16 +0000] 1559 __run_queue process INFO [1546503323-hbase-REGIONSERVER] Orphaning process [12/Nov/2024 06:03:16 +0000] 1559 __run_queue process ERROR Error creating marker /var/run/cloudera-scm-agent/process/1546503323-hbase-REGIONSERVER/process_timestamp Traceback (most recent call last): File "/opt/cloudera/cm-agent/lib/python3.8/site-packages/cmf/process.py", line 1302, in mark_orphan f = open(marker, 'w') FileNotFoundError: [Errno 2] No such file or directory: '/var/run/cloudera-scm-agent/process/1546503323-hbase-REGIONSERVER/process_timestamp' [12/Nov/2024 06:03:16 +0000] 1559 __run_queue util INFO Using specific audit plugin for process hbase-REGIONSERVER [12/Nov/2024 06:03:16 +0000] 1559 __run_queue util INFO Creating metadata plugin for process hbase-REGIONSERVER [12/Nov/2024 06:03:16 +0000] 1559 __run_queue util INFO Using specific metadata plugin for process hbase-REGIONSERVER [12/Nov/2024 06:03:16 +0000] 1559 __run_queue util INFO Using generic metadata plugin for process hbase-REGIONSERVER [12/Nov/2024 06:03:16 +0000] 1559 __run_queue util INFO Creating profile plugin for process hbase-REGIONSERVER [12/Nov/2024 06:03:16 +0000] 1559 __run_queue util INFO Using generic profile plugin for process hbase-REGIONSERVER [12/Nov/2024 06:03:16 +0000] 1559 __run_queue process INFO [1546503485-hbase-REGIONSERVER] Instantiating process [12/Nov/2024 06:03:16 +0000] 1559 __run_queue process INFO [1546503485-hbase-REGIONSERVER] Updating process: True {} [12/Nov/2024 06:03:16 +0000] 1559 __run_queue process INFO First time to activate the process [1546503485-hbase-REGIONSERVER]. [12/Nov/2024 06:03:16 +0000] 1559 __run_queue cgroups INFO Creating cgroup /sys/fs/cgroup/blkio/1546503485-hbase-REGIONSERVER [12/Nov/2024 06:03:16 +0000] 1559 __run_queue cgroups INFO Creating cgroup /sys/fs/cgroup/cpu,cpuacct/system.slice/cloudera-scm-agent.service/1546503485-hbase-REGIONSERVER [12/Nov/2024 06:03:16 +0000] 1559 __run_queue cgroups INFO Creating cgroup /sys/fs/cgroup/devices/1546503485-hbase-REGIONSERVER [12/Nov/2024 06:03:16 +0000] 1559 __run_queue agent INFO Created /var/run/cloudera-scm-agent/process/1546503485-hbase-REGIONSERVER [12/Nov/2024 06:03:16 +0000] 1559 __run_queue agent INFO Chowning /var/run/cloudera-scm-agent/process/1546503485-hbase-REGIONSERVER to hbase (39993) hbase (39993) [12/Nov/2024 06:03:16 +0000] 1559 __run_queue agent INFO Chmod'ing /var/run/cloudera-scm-agent/process/1546503485-hbase-REGIONSERVER to 0751 [12/Nov/2024 06:03:16 +0000] 1559 __run_queue agent INFO Created /var/run/cloudera-scm-agent/process/1546503485-hbase-REGIONSERVER/logs [12/Nov/2024 06:03:16 +0000] 1559 __run_queue agent INFO Chowning /var/run/cloudera-scm-agent/process/1546503485-hbase-REGIONSERVER/logs to hbase (39993) hbase (39993) [12/Nov/2024 06:03:16 +0000] 1559 __run_queue agent INFO Chmod'ing /var/run/cloudera-scm-agent/process/1546503485-hbase-REGIONSERVER/logs to 0751 [12/Nov/2024 06:03:16 +0000] 1559 __run_queue process INFO [1546503485-hbase-REGIONSERVER] Refreshing process files: None [12/Nov/2024 06:03:16 +0000] 1559 __run_queue process INFO /opt/cloudera/cmlib/postgresql-connector.jar doesn't exists! Trying to find /usr/share/java/postgresql-connector-java.jar [12/Nov/2024 06:03:16 +0000] 1559 __run_queue process INFO /usr/share/java/postgresql-connector-java.jar doesn't exists! Trying to find a postgres jar of the pattern /opt/cloudera/cmlib/postgres*.jar [12/Nov/2024 06:03:16 +0000] 1559 __run_queue parcel INFO prepare_environment begin: {'CDH': '7.1.7-1.cdh7.1.7.p3013.57035125', 'SPARK3': '3.2.3.3.2.7172000.0-334-1.p0.37609510'}, ['cdh'], ['hdfs-client-plugin', 'cdh-plugin', 'hbase-plugin'] [12/Nov/2024 06:03:16 +0000] 1559 __run_queue parcel INFO The following requested parcels are not available: {} [12/Nov/2024 06:03:16 +0000] 1559 __run_queue parcel INFO Obtained tags ['cdh', 'impala', 'sentry', 'solr', 'spark', 'kafka', 'kudu'] for parcel CDH [12/Nov/2024 06:03:16 +0000] 1559 __run_queue parcel INFO Obtained tags ['spark3'] for parcel SPARK3 [12/Nov/2024 06:03:16 +0000] 1559 __run_queue parcel_patch INFO Patched parcel in /opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p3013.57035125 for python3 compatibility. [12/Nov/2024 06:03:16 +0000] 1559 __run_queue parcel INFO prepare_environment end: {'CDH': '7.1.7-1.cdh7.1.7.p3013.57035125'} [12/Nov/2024 06:03:16 +0000] 1559 __run_queue __init__ INFO Extracted 19 files and 0 dirs to /var/run/cloudera-scm-agent/process/1546503485-hbase-REGIONSERVER. [12/Nov/2024 06:03:16 +0000] 1559 __run_queue throttling_logger INFO Added principal HTTP/host.com with keytab /var/run/cloudera-scm-agent/process/1546503485-hbase-REGIONSERVER/hbase.keytab as a candidate to kinit [12/Nov/2024 06:03:16 +0000] 1559 __run_queue process INFO [1546503485-hbase-REGIONSERVER] Evaluating resource: cpu [12/Nov/2024 06:03:16 +0000] 1559 __run_queue cgroups INFO Reconfiguring cgroup pseudofile /sys/fs/cgroup/cpu,cpuacct/system.slice/cloudera-scm-agent.service/1546503485-hbase-REGIONSERVER/cpu.shares with value 400 [12/Nov/2024 06:03:16 +0000] 1559 __run_queue cgroups INFO Reconfiguring cgroup pseudofile /sys/fs/cgroup/cpu,cpuacct/system.slice/cloudera-scm-agent.service/1546503485-hbase-REGIONSERVER/cpu.rt_runtime_us with value 1000 [12/Nov/2024 06:03:16 +0000] 1559 __run_queue process INFO [1546503485-hbase-REGIONSERVER] Evaluating resource: io [12/Nov/2024 06:03:16 +0000] 1559 __run_queue cgroups INFO Reconfiguring cgroup pseudofile /sys/fs/cgroup/blkio/1546503485-hbase-REGIONSERVER/blkio.weight with value 200 [12/Nov/2024 06:03:16 +0000] 1559 __run_queue process INFO [1546503485-hbase-REGIONSERVER] Evaluating resource: memory [12/Nov/2024 06:03:16 +0000] 1559 __run_queue process INFO [1546503485-hbase-REGIONSERVER] Evaluating resource: directory [12/Nov/2024 06:03:16 +0000] 1559 __run_queue process INFO [1546503485-hbase-REGIONSERVER] Evaluating resource: tcp_listen [12/Nov/2024 06:03:16 +0000] 1559 __run_queue process INFO reading limits: {'limit_fds': 32768, 'limit_memlock': None} [12/Nov/2024 06:03:16 +0000] 1559 __run_queue process INFO [1546503485-hbase-REGIONSERVER] Launching process. one-off False, command hbase/hbase.sh, args ['regionserver', 'start'] [12/Nov/2024 06:03:16 +0000] 1559 __run_queue supervisor WARNING Failed while getting process info. Retrying. (<Fault 10: 'BAD_NAME: 1546503485-hbase-REGIONSERVER'>) [12/Nov/2024 06:03:18 +0000] 1559 __run_queue supervisor INFO Triggering supervisord update. [12/Nov/2024 06:03:18 +0000] 1559 __run_queue process INFO Begin audit plugin refresh [12/Nov/2024 06:03:18 +0000] 1559 __run_queue process INFO Begin metadata plugin refresh [12/Nov/2024 06:03:18 +0000] 1559 __run_queue process INFO Begin profile plugin refresh [12/Nov/2024 06:03:18 +0000] 1559 __run_queue daemon INFO Instantiating generic monitor for service HBASE and role REGIONSERVER [12/Nov/2024 06:03:18 +0000] 1559 __run_queue process INFO Begin monitor refresh. [12/Nov/2024 06:03:18 +0000] 1559 __run_queue abstract_monitor INFO Refreshing GenericMonitor HBASE-REGIONSERVER for None [12/Nov/2024 06:03:18 +0000] 1559 __run_queue daemon INFO New monitor: (<cmf.monitor.generic.GenericMonitor object at 0x7f727379a2b0>,) [12/Nov/2024 06:03:18 +0000] 1559 __run_queue process INFO Daemon refresh complete for process 1546503485-hbase-REGIONSERVER. [12/Nov/2024 06:03:20 +0000] 1559 Profile-Plugin navigator_plugin INFO Pipelines updated for Profile Plugin: set() [12/Nov/2024 06:03:20 +0000] 1559 Audit-Plugin navigator_plugin INFO Pipelines updated for Audit Plugin: [] [12/Nov/2024 06:03:20 +0000] 1559 Metadata-Plugin navigator_plugin INFO Pipelines updated for Metadata Plugin: [] [12/Nov/2024 06:03:57 +0000] 1559 MainThread process INFO [1546503485-hbase-REGIONSERVER] Unregistered supervisor process FATAL [12/Nov/2024 06:03:57 +0000] 1559 MainThread cgroups INFO Destroying cgroup /sys/fs/cgroup/blkio/1546503485-hbase-REGIONSERVER [12/Nov/2024 06:03:57 +0000] 1559 MainThread cgroups INFO Destroying cgroup /sys/fs/cgroup/cpu,cpuacct/system.slice/cloudera-scm-agent.service/1546503485-hbase-REGIONSERVER [12/Nov/2024 06:03:57 +0000] 1559 MainThread cgroups INFO Destroying cgroup /sys/fs/cgroup/devices/1546503485-hbase-REGIONSERVER [12/Nov/2024 06:03:59 +0000] 1559 MainThread supervisor INFO Triggering supervisord update. [12/Nov/2024 06:03:59 +0000] 1559 MainThread throttling_logger INFO Removed keytab /var/run/cloudera-scm-agent/process/1546503485-hbase-REGIONSERVER/hbase.keytab as a candidate to kinit from [12/Nov/2024 06:04:12 +0000] 1559 __run_queue process INFO [1546503485-hbase-REGIONSERVER] Updating process: False {'run_generation': (1, 2), 'running': (True, False)} [12/Nov/2024 06:04:12 +0000] 1559 __run_queue process INFO [1546503485-hbase-REGIONSERVER] Deactivating process (skipped) [12/Nov/2024 06:04:12 +0000] 1559 __run_queue process INFO [1546503485-hbase-REGIONSERVER] stopping monitors [12/Nov/2024 06:04:15 +0000] 1559 Profile-Plugin navigator_plugin INFO stopping Profile Plugin for hbase-REGIONSERVER with count 0 pipelines names []. [12/Nov/2024 06:04:15 +0000] 1559 Audit-Plugin navigator_plugin INFO stopping Audit Plugin for hbase-REGIONSERVER with count 0 pipelines names []. [12/Nov/2024 06:04:15 +0000] 1559 Metadata-Plugin navigator_plugin INFO stopping Metadata Plugin for hbase-REGIONSERVER with count 0 pipelines names []. [12/Nov/2024 06:04:18 +0000] 1559 MonitorDaemon-Scheduler daemon INFO Monitor expired: ('GenericMonitor HBASE-REGIONSERVER for hbase-REGIONSERVER-78fd4f39bfc69a473cc5abed13e41dac',)   ... View more

Hi @upadhyayk04  Thanks for your response. I have tried to reply your message but no luck. Hope this one will get through. Firstly, the hosts are heart beating. Secondly, the /etc/krb5.conf seems to be the same for other working host (Hue server host in this case). The Web Server Status issue is the same across HDFS, Hbase, Yarn, and Impala. Thirdly, I had tried the manual kinit before but it still throw the same error. After trying manual kinit (kinit -k -t hdfs.keytab hdfs/host.my-default-realm.com) from the latest data node process, I ran the klist command (klist -e) and got the following. Valid starting Expires Service principal 10/11/24 23:43:47 11/11/24 09:43:47 krbtgt/my-default-realm.COM@my-default-realm.COM renew until 17/11/24 23:43:47, Etype (skey, tkt): arcfour-hmac, aes256-cts-hmac-sha1-96 Below is the configured Kerberos Encryption Types from the Cloudera Manager Console Below is part of the host /etc/krb5.conf content. [libdefaults] renew_lifetime = 604800 ticket_lifetime = 36000 udp_preference_limit = 1 permitted_enctypes = rc4-hmac aes256-cts aes128-cts default_tgs_enctypes = rc4-hmac aes256-cts aes128-cts default_tkt_enctypes = rc4-hmac aes256-cts aes128-cts default_realm = my-default-realm.com default_etypes = arcfour-hmac-md5 default_etypes_des = des-cbc-crc allow_weak_crypto = true forwardable = true default_keytab_name = /etc/opt/quest/vas/host.keytab [libvas] site-name-override = iNET-LDAP use-dns-srv = true use-tcp-only = true auth-helper-timeout = 60 Finally, the OS upgrade is not yet performed. We're still on RED Hat OL7. ... View more

I have been trying to upload the full log but it keeps getting removed. ... View more