Assuming you bounced all of Ambari Metrics and possibly the services, and the metrics still aren't appearing, that could be a separate question. Also, it looks like you're trying to use BigInsights 4.2, that's been deprecated by IBM in favor of HDP. You may want to look at upgrading instead of working through these issues on BI. ... View more

I just tried this with HDP 2.5.3 + Ambari 2.4.2, for the SASL_SSL example, I had to make a couple of changes to get it to work. For producer, When I tried the example able, I got "new-producer is not a recognized option" I had to create a ssl properties file (I called mine /tmp/kakfka.ssl.properties) then run producer as: /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list kafka-node1.domain:6668,kafka-node2.domain:6668 -topic test --producer.config /tmp/kakfka.ssl.properties --security-protocol SASL_SSL Similarly for consumer, I had to add a reference to the ssl.properties file: I used this: /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --consumer.config /tmp/kakfka.ssl.properties -bootstrap-server kafka-node1.domain:6668,kafka-node2.domain:6668 --topic test --from-beginning --security-protocol SASL_SSL --new-consumer ... View more

What's the status of your /var/run/ambari-server/bootstrap and /var/run/ambari-server/stack-recommendations directories? Do they exist? I've seen this problem when those directories are missing for some reason. Manually creating these as empty directories, owned by the user ambari fixed the problem. ... View more

If they instead look like 'SYSTEM.CATALOG' then they are still the original way. If instead your Phoenix System tables look like 'SYSTEM:CATALOG' they were migrated to the new namespace mechanism already. I think you'll likely have to add 'phoenix.schema.isNamespaceMappingEnabled' to your hbase-site.xml, I dont see a way to go back. ... View more

Can we see what the Phoenix System tables look like in HBase Shell? Run 'hbase shell' then 'list' and post the output. Example: [root@sandbox ~]# hbase shell HBase Shell; enter 'help<RETURN>' for list of supported commands. Type "exit<RETURN>" to leave the HBase Shell Version 1.1.2.2.5.0.0-1245, r53538b8ab6749cbb6fdc0fe448b89aa82495fb3f, Fri Aug 26 01:32:27 UTC 2016 hbase(main):001:0> list TABLE SYSTEM:CATALOG SYSTEM:FUNCTION SYSTEM:SEQUENCE SYSTEM:STATS 3 row(s) in 0.2090 seconds => ["SYSTEM:CATALOG", "SYSTEM:FUNCTION", "SYSTEM:SEQUENCE", "SYSTEM:STATS"] ... View more

Also, this is a good related article: https://community.hortonworks.com/content/kbentry/18088/ambari-shows-hdp-services-to-be-down-whereas-they.html ... View more

I've only ever seen this happen when the ambari agent process terminates on a host. In 3) you mention stopping and starting the agent. What's that status of the agent when ambari is showing the wrong state? What's does the ambari agent log file (/var/log/ambari-agent/ambari-agent.log) say ? [root@sandbox ~]# ambari-agent status Found ambari-agent PID: 2877 ambari-agent running. Agent PID at: /var/run/ambari-agent/ambari-agent.pid Agent out at: /var/log/ambari-agent/ambari-agent.out Agent log at: /var/log/ambari-agent/ambari-agent.log ... View more