this is resolved. after checking /var/log/ambari-metrics-collector/hbase-ams-master-XXXX.log found Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user. checked keytabs files under /etc/security/keytabs/ams-hbase.master.keytab after klist to this keytab came to know that its has invalid entry for hosts. klist -ekt ams-hbase.master.keytab Keytab name: FILE:ams-hbase.master.keytab KVNO Timestamp Principal ---- ----------------- -------------------------------------------------------- 2 07/04/16 03:09:57 host/kdchost1.EXAMPLE.COM@EXAMPLE.COM (aes256-cts-hmac-sha1-96) 2 07/04/16 03:09:57 host/kdchost1.EXAMPLE.COM@EXAMPLE.COM (aes128-cts-hmac-sha1-96) 2 07/04/16 03:09:57 host/kdchost1.EXAMPLE.COM@EXAMPLE.COM (des3-cbc-sha1) 2 07/04/16 03:09:57 host/kdchost1.EXAMPLE.COM@EXAMPLE.COM (arcfour-hmac) 2 07/04/16 03:09:57 host/kdchost1.EXAMPLE.COM@EXAMPLE.COM (aes256-cts-hmac-sha1-96) 2 07/04/16 03:09:57 host/kdchost1.EXAMPLE.COM@EXAMPLE.COM (aes128-cts-hmac-sha1-96) 2 07/04/16 03:09:57 host/kdchost1.EXAMPLE.COM@EXAMPLE.COM (des3-cbc-sha1) 2 07/04/16 03:09:57 host/kdchost1.EXAMPLE.COM@EXAMPLE.COM (arcfour-hmac) 2 07/13/16 11:11:54 kmsusr@EXAMPLE.COM (aes256-cts-hmac-sha1-96) 2 07/13/16 11:11:54 kmsusr@EXAMPLE.COM (aes128-cts-hmac-sha1-96) 2 07/13/16 11:11:54 kmsusr@EXAMPLE.COM (des3-cbc-sha1) 2 07/13/16 11:11:54 kmsusr@EXAMPLE.COM (arcfour-hmac) 3 07/13/16 11:14:14 kmsusr@EXAMPLE.COM (aes256-cts-hmac-sha1-96) 3 07/13/16 11:14:14 kmsusr@EXAMPLE.COM (aes128-cts-hmac-sha1-96) 3 07/13/16 11:14:14 kmsusr@EXAMPLE.COM (des3-cbc-sha1) 3 07/13/16 11:14:14 kmsusr@EXAMPLE.COM (arcfour-hmac) 4 07/13/16 11:15:46 kmsusr@EXAMPLE.COM (aes256-cts-hmac-sha1-96) 4 07/13/16 11:15:46 kmsusr@EXAMPLE.COM (aes128-cts-hmac-sha1-96) 4 07/13/16 11:15:46 kmsusr@EXAMPLE.COM (des3-cbc-sha1) 4 07/13/16 11:15:46 kmsusr@EXAMPLE.COM (arcfour-hmac) 2 09/21/16 11:11:23 amshbase/edgenode.EXAMPLE.COM@EXAMPLE.COM (aes256-cts-hmac-sha1-96) 2 09/21/16 11:11:23 amshbase/edgenode.EXAMPLE.COM@EXAMPLE.COM (aes128-cts-hmac-sha1-96) 2 09/21/16 11:11:23 amshbase/edgenode.EXAMPLE.COM@EXAMPLE.COM (des3-cbc-sha1) 2 09/21/16 11:11:23 amshbase/edgenode.EXAMPLE.COM@EXAMPLE.COM (arcfour-hmac) 3 09/21/16 11:13:40 amshbase/edgenode.EXAMPLE.COM@EXAMPLE.COM (aes256-cts-hmac-sha1-96) 3 09/21/16 11:13:40 amshbase/edgenode.EXAMPLE.COM@EXAMPLE.COM (aes128-cts-hmac-sha1-96) 3 09/21/16 11:13:40 amshbase/edgenode.EXAMPLE.COM@EXAMPLE.COM (des3-cbc-sha1) 3 09/21/16 11:13:40 amshbase/edgenode.EXAMPLE.COM@EXAMPLE.COM (arcfour-hmac) 4 09/21/16 11:15:07 amshbase/edgenode.EXAMPLE.COM@EXAMPLE.COM (aes256-cts-hmac-sha1-96) 4 09/21/16 11:15:07 amshbase/edgenode.EXAMPLE.COM@EXAMPLE.COM (aes128-cts-hmac-sha1-96) 4 09/21/16 11:15:07 amshbase/edgenode.EXAMPLE.COM@EXAMPLE.COM (des3-cbc-sha1) 4 09/21/16 11:15:07 amshbase/edgenode.EXAMPLE.COM@EXAMPLE.COM (arcfour-hmac) so deleted that keytab file and generated new keytab with valid entries and placed on the metrics-collector host under /etc/security/keytabs restarted metrics-collector service the service started. ... View more

@Balkrishna, The problem is with special characters in the KAFKA service metrics file. We use this file as a part of stack_advisor calculations for AMS split points. Following is a grep on non ASCII characters that reveals the problem :- grep --color='auto' -P -n "[\x80-\xFF]" /var/lib/ambari-server/resources/common-services/AMBARI_METRICS/0.1.0/package/files/service-metrics/KAFKA.txt This will show you the ascii characters present in the file as, Output:-43:��kafka.network.RequestMetrics.RequestsPerSec.request.OffsetFetch.count�� 45:��kafka.network.RequestMetrics.RequestsPerSec.request.OffsetCommit.count 47:kafka.network.RequestMetrics.RequestsPerSec.request.LeaderAndIsr.1MinuteRate�� Use /var/lib/ambari-server/resources/scripts/configs.sh to modify and get values from ambari-server as, /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p ADMIN_PASSWORD -port 8080 get localhost CLUSTER_NAME ams-site Check the value of "timeline.metrics.cluster.aggregate.splitpoints" and "timeline.metrics.host.aggregate.splitpoints", Look for special non-ascii characters example: "dfs.datanode.ReplaceBlockOpAvgTime,kafka.network.RequestMetrics.RequestsPerSec.request.JoinGroup.1MinuteRate ,master.Master.ProcessCallTime_num_ops,regionserver.Server.blockCacheEvictionCount" Here after 1MinuteRate there is space which will shown as a special character on browser through API call as, "http://AMBARI_SERVER_HOSTS:8080/api/v1/clusters/CLUSTER_NAME/configurations?type=ams-site" Take latest tag from it at the bottom of page. and put it on browser you will see the special character as, "timeline.metrics.cluster.aggregate.splitpoints":"dfs.datanode.ReplaceBlockOpAvgTime,kafka.network.RequestMetrics.RequestsPerSec.request.JoinGroup.1MinuteRate
,master.Master.ProcessCallTime_num_ops,regionserver.Server.blockCacheEvictionCount" To resolve this issue use, /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p ADMIN_PASSWORD -port 8080 set localhost CLUSTER_NAME ams-site timeline.metrics.cluster.aggregate.splitpoints dfs.datanode.ReplaceBlockOpAvgTime,kafka.network.RequestMetrics.RequestsPerSec.request.JoinGroup.1MinuteRate,master.Master.ProcessCallTime_num_ops,regionserver.Server.blockCacheEvictionCount To change second property paramater use:- /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p ADMIN_PASSWORD -port 8080 get localhost CLUSTER_NAME ams-site /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p ADMIN_PASSWORD -port 8080 set localhost CLUSTER_NAME ams-site timeline.metrics.host.aggregate.splitpoints EventTakeSuccessCount,cpu_idle,dfs.FSNamesystem.ExcessBlocks,dfs.datanode.ReadBlockOpNumOps,disk_total,jvm.JvmMetrics.LogError,kafka.controller.ControllerStats.LeaderElectionRateAndTimeMs.99percentile,kafka.network.RequestMetrics.RequestsPerSec.request.FetchFollower.5MinuteRate,kafka.network.RequestMetrics.RequestsPerSec.request.UpdateMetadata.1MinuteRate,kafka.server.BrokerTopicMetrics.FailedFetchRequestsPerSec.meanRate,master.AssignmentManger.ritCount,master.FileSystem.MetaHlogSplitTime_95th_percentile,mem_shared,proc_total,regionserver.Server.Append_median,regionserver.Server.Replay_95th_percentile,regionserver.Server.totalRequestCount,rpcdetailed.rpcdetailed.GetBlockLocationsAvgTime,write_bps ### Restart the Service ... View more