Support Questions
Find answers, ask questions, and share your expertise

alert on sandbox.hortonworks.com HDP 2.3.2

Explorer

Hi all!

I just download the latest sandbox.hortonworks.com HDP 2.3.2 but when started and running for an hour I still have some alerts on the ambari dashboard! Anybody has the same problem?

Here are what I can copy from the ambari dashboard:

HiveServer2 Process

Connection failed on host sandbox.hortonworks.com:10000 (Execution of 'ambari-sudo.sh su ambari-qa -l -s /bin/bash -c 'export PATH='"'"'/usr/sbin:/sbi

Hive Metastore Process

Metastore on sandbox.hortonworks.com failed (Execution of 'ambari-sudo.sh su ambari-qa -l -s /bin/bash -c 'export PATH='"'"'/usr/sbin:/sbin:/usr/lib/a

Ambari

There are 21 stale alerts from 1 host(s): History Server RPC Latency, HiveServer2 Process, Ranger Usersync Process, WebHCat Server Status, Ambari Agent Disk Usage, History Server Web UI, App Timeline Web UI, Hive Metastore Process, Atlas Metadata Server Process, History Server CPU Utilization, History Server Process, ResourceManager Web UI, NodeManager Health Summary, ZooKeeper Server Process, ResourceManager CPU Utilization, NodeManager Health, Flume Agent Status, Ranger Admin Process, NodeManager Web UI, ResourceManager RPC Latency, Metadata Server Web UI CRIT

for about an hour

There are 21 stale alerts from 1 host(s): History Server RPC Latency,

I don't understand why the vm has multiples alerts out of the box!

Is there a way to fix this or is there a vm without any errors?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: alert on sandbox.hortonworks.com HDP 2.3.2

Explorer

Today I spoke with Robert Molina from Hortonworks and possibly found what is creating all those alerts!

The sandbox is intended to be run on a desktop with a NAT networked interface. I set it up on a dedicaded headless server with a bridge adaptor. Looks like sandbox have a problem with that and that cause some of the services configs to not function properly! As a result some services works but reports network connections alerts! After some config change. The related alerts weren't there anymore. So always use a vm for and how it was intended to be used.

Thanks to the hortonworks team and Robert who wanted to go to the bottom of this.

Conclusion: If you want, like me, to test drive hortonworks on a headless server. Start from scratch and build it! What every sysadmin should do anyways... That's what I'll do this week end...

P

View solution in original post

13 REPLIES 13

Re: alert on sandbox.hortonworks.com HDP 2.3.2

Peter, the intent is not to start every single service in the sandbox VM - it will get very memory-heavy quickly. I'm not even sure 8GB would be a comfortable setting. In addition, because it's a VM which will be suspended and resumed often, some services will complain and raise alerts due to a clock drift (e.g. HBase is notorious for that).

All-in-all, that shouldn't be an issue. I would, however, check with core services states first. Is Ambari Metrics service running?

Re: alert on sandbox.hortonworks.com HDP 2.3.2

Mentor

you can ignore some of the alerts, Sandbox 2.3.2 comes with Secondary NameNode turned off by default, which will trigger an alert. That's by design, so if you're working with something that is not directly affecting your work, ignore it.

Re: alert on sandbox.hortonworks.com HDP 2.3.2

As a tip - one can put a service into a Maintenance mode (check out the service actions menu when you go to a service screen). This will suppress alerts and also make this service ignore bulk action requests (like restart all).

Re: alert on sandbox.hortonworks.com HDP 2.3.2

Expert Contributor

This is how out of box Sandbox worked for me.

1149-capture.png

Not that I reduced RAM and CPU count during import. Also I have to manually start MapReduce2

Re: alert on sandbox.hortonworks.com HDP 2.3.2

Explorer

Mine doesn't look at all like that!!! Just to be on the safe side! I just re-import the

HDP_2.3.2_virtualbox.ova with wirtualbox 4.3.26 I put 4 cpu and 16 gig of ram.

I still have problems... Here is the ambari after start:

1289-20160112-valilla-sandbox.jpg

I mean is it me or there shouldn't be any alerts??? That mean when horton saved the VM the alert were there! So I did a yum update on the vm but it's worst ! Here is the ambari after restart:

1301-20160112-valilla-sandbox-update.jpg

The thing is all these alert make me questioned the accuracy and solidity of the distro. I mean I'm not even shure it's working out of the box! What if I installed something on it and it doesn't work! Is it what I've installed or the Hortonworks environement?

I mean this is not right! Am I the only one with these problems?

...

P

Re: alert on sandbox.hortonworks.com HDP 2.3.2

@Peter Young Alerts will/should go away eventually. You can restart services and check the behavior.

Re: alert on sandbox.hortonworks.com HDP 2.3.2

Explorer

18 hours later I'm down to 6 alerts: Here is the ambari screen shot:

1306-20160113-valilla-sandbox.jpg

That is after I started the Ambari Metrics. It looks like It's some connection errore. For example, here is one ranger alert:

Connection failed: [Errno 111] Connection refused to sandbox.hortonworks.com:5151

But if I connect to sandbox.hortonworks.com:5151 with a browser. It works!!!

Looks like a dns resolution or a firewall connectivity problem...

Is there a way to just reset the alerts?

Re: alert on sandbox.hortonworks.com HDP 2.3.2

@Peter Young Is the email on your profile accurate? We want to reach out to you.

Re: alert on sandbox.hortonworks.com HDP 2.3.2

Explorer

yes you can reach me there!