How can I change the AMS hbase.rootdir in embedded mode?

[#1]

Changing it broke Metrics Collector...

I need to move the AMS hbase.rootdir to another partition, so I created a directory, did a chown -R ams:hadoop MYDIR, changed the configuration value and restarted AMS. The Metrics Collector will not start. It's throwing a connection refused exception when trying to connect to zookeeper on localhost:61181. Unfortunately, I don't have access to the exact exception at the moment).

This is on HDP 2.4

Nothing is listening on port 61181, which I believe should be the embedded ZK port.

hbase.zookeeper.property.clientPort={{zookeeper_clientPort}}.

I killed all ams processes to be sure something was not in a bad state. I also tried copying the old hbase.rootdir to my new directory with the same permissions but it still fails. When I switch back to the old location it works fine. This seems very similar to changing to distributed mode, so I don't understand what's going wrong.

[#2 · Accepted solution]

@james.jones

If this is a production system, please reach out to Hortonworks Technical Support team.

Otherwise, I hope you have copied the existing data from the old hbase.rootdir to the new hbase.rootdir. Post this if it doesn't come up good, backup hbase.tmp.dir/zookeeper and then remove hbase.tmp.dir/zookeeper/* files and retry.

If the existing data is not required and you would like to completely clean up, please follow Cleaning Up AMS data.

Could you please share the values for hbase.rootdir and hbase.tmp.dir from /etc/ams-hbase/hbase-site.xml file?

[#3]

Thanks for the info @vpoornalingam. We resolved the issue. It turns out that when we changed the path, Ambari suggested making other changes, which we mindlessly accepted. Obviously we should have paid more attention. One of the changes was to drop the embedded hbase master heap size to a much lower value. I realized this after looking at the log hbase-ams-master-<FQDN>.out rather than ambari-metrics-collector.log that I was looking at.

[#4]

Ah! James Jones --- I have a Recommendation - Never believe recommendations from Ambari 😉