Grafana - initContextWithBasicAuth()] [E] Invalid Basic Auth Header: Invalid basic auth header

[#1]

I've installed Grafana through Ambari 2.4.1.0 on HDP 2.3.4. The datasource is "AMBARI_METRICS".

When I try to validate the datasource, I'm getting below error:

2016/09/28 17:05:56 [middleware.go:145 initContextWithBasicAuth()] [E] Invalid Basic Auth Header: Invalid basic auth header 2016/09/28 17:05:56 [I] Completed 10.128.211.106 admin "GET /api/datasources/proxy/1/ws/v1/timeline/metrics/metadata HTTP/1.1" 401 Unauthorized 39 bytes in 102us

datasource info:

Url = http://hostip:6188

access = proxy

http Auth = unchecked

with credentials= unchecked

-------

If I change the auth.basic=false, Grafana service fails to start. We have kerberos installed in our cluster.

[#2]

Hi P D,

Do you have SPNEGO enabled for Hadoop components on the cluster?

[#3]

Yes, SPENGO is enabled.

I do not see any useful logs with "Debug" logging.

root@myhost:conf> grep ';level' ams-grafana.ini

# ;level = Info

;level = Debug

[#4 · Accepted solution]

Hi PD,

This is a known bug in AMS.

The cause is that AMS uses its own instance of Yarn ATS as the application server, and has the cluster core-site in its conf directory (/etc/ambari-metrics-collector/conf). Since the core-site has spnego configs used by cluster Hadoop components, this ATS picks them up, and acts as a spnego enabled end point, even though AMS has no spnego support.

The workaround fix (https://issues.apache.org/jira/browse/AMBARI-18187) was recently checked into branch-2.4, and should be part of next Ambari release - 2.4.2. The long term solution of supporting SPNEGO in AMS is being worked on as part of https://issues.apache.org/jira/browse/AMBARI-14384, and is planned for Ambari-2.5.0.

[#5]

@Aravindan Vijayan

Thanks for your response.

Based on the JIRA the fix is replacing the "ams.py" script. Actually I've replaced the script in /var/lib/ambari-server/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/ and restarted Ambari Metrics. However I'm still facing the same issue.

ambari-metrics-monitor> grep -A3 'XmlConfig("core-site.xml"' /var/lib/ambari-server/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/ams.py
      XmlConfig("core-site.xml",
                conf_dir=params.ams_collector_conf_dir,
                configurations=params.config['configurations']['core-site'],
                configuration_attributes=params.config['configuration_attributes']['core-site'],
--
      XmlConfig("core-site.xml",
                conf_dir=params.hbase_conf_dir,
                configurations=params.config['configurations']['core-site'],
                configuration_attributes=params.config['configuration_attributes']['core-site'],
--
      XmlConfig("core-site.xml",
                conf_dir=params.ams_collector_conf_dir,
                configurations=truncated_core_site,
                configuration_attributes=params.config['configuration_attributes']['core-site'],
--
      XmlConfig("core-site.xml",
                conf_dir=params.hbase_conf_dir,
                configurations=truncated_core_site,
                configuration_attributes=params.config['configuration_attributes']['core-site'],

I've also noticed below error in ambari-metrics-monitor.out file.

2016-09-21 21:35:18,778 [INFO] emitter.py:91 - server: http://ip:6188/ws/v1/timeline/metrics 2016-09-21 21:35:18,780 [WARNING] emitter.py:74 - Error sending metrics to server. HTTP Error 401: Authentication required 2016-09-21 21:35:18,780 [WARNING] emitter.py:80 - Retrying after 5 ...

https://issues.apache.org/jira/browse/AMBARI-14384

Our metrics collector server is part of the cluster.

[#6]

Hi @P D,

After replacing the ams.py under /var/lib/ambari-server, you will have to restart ambari-server for all the agents to pick up the change. You should also see the ams.py in a similar path under /var/lib/ambari-agent. That will be get the new script on Ambari Server restart. Then, go ahead and restart Metrics collector component.

[#7]

@Aravindan Vijayan

Sorry i forgot mention, i did restart the ambari server and agent on all nodes.

Just now, I've restarted all metrics-collector and still facing the same error.

root@host:ambari-agent> grep -A3 'XmlConfig("core-site.xml"' /var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/ams.py
      XmlConfig("core-site.xml",
                conf_dir=params.ams_collector_conf_dir,
                configurations=params.config['configurations']['core-site'],
                configuration_attributes=params.config['configuration_attributes']['core-site'],
--
      XmlConfig("core-site.xml",
                conf_dir=params.hbase_conf_dir,
                configurations=params.config['configurations']['core-site'],
                configuration_attributes=params.config['configuration_attributes']['core-site'],
--
      XmlConfig("core-site.xml",
                conf_dir=params.ams_collector_conf_dir,
                configurations=truncated_core_site,
                configuration_attributes=params.config['configuration_attributes']['core-site'],
--
      XmlConfig("core-site.xml",
                conf_dir=params.hbase_conf_dir,
                configurations=truncated_core_site,
                configuration_attributes=params.config['configuration_attributes']['core-site'],

[#8]

Hi @P D,

In /etc/ambari-metrics-collector/conf/core-site.xml, do you see the following configs?

hadoop.http.filter.initializers

hadoop.http.authentication.type

[#9]

@Aravindan Vijayan

Yes. Actually our Ambari server is kerberoised too.

hadoop.http.filter.initializers = org.apache.hadoop.security.AuthenticationFilterInitializer

hadoop.http.authentication.type = kerberos

[#10]

Hi @P D,

The patch that was committed was slightly different from the one that was attached to the JIRA. I have now attached the one that was committed.

https://issues.apache.org/jira/secure/attachment/12832196/AMBARI-18187-2.patch

Can you try with that?