question Re: alert on sandbox.hortonworks.com HDP 2.3.2 in Archives of Support Questions (Read Only)

alert on sandbox.hortonworks.com HDP 2.3.2

tech — Mon, 04 Jan 2016 04:38:22 GMT

Hi all!

I just download the latest sandbox.hortonworks.com HDP 2.3.2 but when started and running for an hour I still have some alerts on the ambari dashboard! Anybody has the same problem?

Here are what I can copy from the ambari dashboard:

HiveServer2 Process

Connection failed on host sandbox.hortonworks.com:10000 (Execution of 'ambari-sudo.sh su ambari-qa -l -s /bin/bash -c 'export PATH='"'"'/usr/sbin:/sbi

Hive Metastore Process

Metastore on sandbox.hortonworks.com failed (Execution of 'ambari-sudo.sh su ambari-qa -l -s /bin/bash -c 'export PATH='"'"'/usr/sbin:/sbin:/usr/lib/a

Ambari

There are 21 stale alerts from 1 host(s): History Server RPC Latency, HiveServer2 Process, Ranger Usersync Process, WebHCat Server Status, Ambari Agent Disk Usage, History Server Web UI, App Timeline Web UI, Hive Metastore Process, Atlas Metadata Server Process, History Server CPU Utilization, History Server Process, ResourceManager Web UI, NodeManager Health Summary, ZooKeeper Server Process, ResourceManager CPU Utilization, NodeManager Health, Flume Agent Status, Ranger Admin Process, NodeManager Web UI, ResourceManager RPC Latency, Metadata Server Web UI CRIT

for about an hour

There are 21 stale alerts from 1 host(s): History Server RPC Latency,

I don't understand why the vm has multiples alerts out of the box!

Is there a way to fix this or is there a vm without any errors?

Re: alert on sandbox.hortonworks.com HDP 2.3.2

andrewg — Mon, 04 Jan 2016 22:11:50 GMT

Peter, the intent is not to start every single service in the sandbox VM - it will get very memory-heavy quickly. I'm not even sure 8GB would be a comfortable setting. In addition, because it's a VM which will be suspended and resumed often, some services will complain and raise alerts due to a clock drift (e.g. HBase is notorious for that).

All-in-all, that shouldn't be an issue. I would, however, check with core services states first. Is Ambari Metrics service running?

Re: alert on sandbox.hortonworks.com HDP 2.3.2

aervits — Mon, 04 Jan 2016 22:12:10 GMT

you can ignore some of the alerts, Sandbox 2.3.2 comes with Secondary NameNode turned off by default, which will trigger an alert. That's by design, so if you're working with something that is not directly affecting your work, ignore it.

Re: alert on sandbox.hortonworks.com HDP 2.3.2

andrewg — Mon, 04 Jan 2016 22:18:25 GMT

As a tip - one can put a service into a Maintenance mode (check out the service actions menu when you go to a service screen). This will suppress alerts and also make this service ignore bulk action requests (like restart all).

Re: alert on sandbox.hortonworks.com HDP 2.3.2

rahulpathak109 — Mon, 19 Aug 2019 12:21:21 GMT

This is how out of box Sandbox worked for me.

Not that I reduced RAM and CPU count during import. Also I have to manually start MapReduce2

Re: alert on sandbox.hortonworks.com HDP 2.3.2

tech — Mon, 19 Aug 2019 12:21:13 GMT

Mine doesn't look at all like that!!! Just to be on the safe side! I just re-import the

HDP_2.3.2_virtualbox.ova with wirtualbox 4.3.26 I put 4 cpu and 16 gig of ram.

I still have problems... Here is the ambari after start:

I mean is it me or there shouldn't be any alerts??? That mean when horton saved the VM the alert were there! So I did a yum update on the vm but it's worst ! Here is the ambari after restart:

The thing is all these alert make me questioned the accuracy and solidity of the distro. I mean I'm not even shure it's working out of the box! What if I installed something on it and it doesn't work! Is it what I've installed or the Hortonworks environement?

I mean this is not right! Am I the only one with these problems?

...

Re: alert on sandbox.hortonworks.com HDP 2.3.2

nsabharwal — Wed, 13 Jan 2016 05:17:25 GMT

@Peter Young Alerts will/should go away eventually. You can restart services and check the behavior.

Re: alert on sandbox.hortonworks.com HDP 2.3.2

tech — Mon, 19 Aug 2019 12:21:00 GMT

18 hours later I'm down to 6 alerts: Here is the ambari screen shot:

That is after I started the Ambari Metrics. It looks like It's some connection errore. For example, here is one ranger alert:

Connection failed: [Errno 111] Connection refused to sandbox.hortonworks.com:5151

But if I connect to sandbox.hortonworks.com:5151 with a browser. It works!!!

Looks like a dns resolution or a firewall connectivity problem...

Is there a way to just reset the alerts?

Re: alert on sandbox.hortonworks.com HDP 2.3.2

amcbarnett — Fri, 15 Jan 2016 01:08:16 GMT

@Peter Young Is the email on your profile accurate? We want to reach out to you.

Re: alert on sandbox.hortonworks.com HDP 2.3.2

tech — Fri, 15 Jan 2016 08:25:04 GMT

yes you can reach me there!

Re: alert on sandbox.hortonworks.com HDP 2.3.2

vance_wei — Thu, 21 Jan 2016 02:04:54 GMT

I have found out how to resolve the "Connection failed on host sandbox.hortonworks.com" failures for my sandbox.

The fix is to add sandbox.hortonworks.com to your no_proxy variable in /etc/profile.

It turns out I receive this error because I am using sandbox behind corporate proxy, and have http_proxy and https_proxy setup. Even though sandbox.hortonworks.com is resolved to a local ip in the /etc/hosts file, that local ip is not always the localhost, but an IP in range 10.0.X.X if you are using NAT. Therefore, the request is hitting the proxy first, which resulted in a failure.

Re: alert on sandbox.hortonworks.com HDP 2.3.2

nsabharwal — Thu, 21 Jan 2016 02:10:29 GMT

@Peter Young

You can check sandbox setup here https://community.hortonworks.com/articles/10788/hortonworks-sandbox-setup.html

Re: alert on sandbox.hortonworks.com HDP 2.3.2

amcbarnett — Thu, 21 Jan 2016 03:47:32 GMT

@Peter Young After contacting support someone posted the answer to stale alerts here:

https://community.hortonworks.com/questions/9762/how-to-get-rid-of-stale-alerts-in-ambari.html

Re: alert on sandbox.hortonworks.com HDP 2.3.2

tech — Fri, 22 Jan 2016 04:44:30 GMT

Today I spoke with Robert Molina from Hortonworks and possibly found what is creating all those alerts!

The sandbox is intended to be run on a desktop with a NAT networked interface. I set it up on a dedicaded headless server with a bridge adaptor. Looks like sandbox have a problem with that and that cause some of the services configs to not function properly! As a result some services works but reports network connections alerts! After some config change. The related alerts weren't there anymore. So always use a vm for and how it was intended to be used.

Thanks to the hortonworks team and Robert who wanted to go to the bottom of this.

Conclusion: If you want, like me, to test drive hortonworks on a headless server. Start from scratch and build it! What every sysadmin should do anyways... That's what I'll do this week end...