question Re: Metron vagrant start hadoop services faiing in Archives of Support Questions (Read Only)

Metron vagrant start hadoop services faiing

sunile_manjee — Sun, 24 Apr 2016 10:18:18 GMT

Trying to get a instance of metron up and running. Ambari is up so not sure if things are ok. I performed vagrant up and things went smoothly until task to start hadoop services. here is my error log:

TASK [ambari_config : Start All Hadoop Services node1] *************************
failed: [node1] (item=HDFS) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "HDFS", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=tbclunfsajxy1bnyfpvk7twl;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/HDFS", "user": "admin"}
failed: [node1] (item=YARN) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "YARN", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=1k3byxj6l24791q0zzxpojzmpe;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/YARN", "user": "admin"}
failed: [node1] (item=MAPREDUCE2) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "MAPREDUCE2", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=onwvdexdk4x513bjb4tzj0wfz;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/MAPREDUCE2", "user": "admin"}
failed: [node1] (item=ZOOKEEPER) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "ZOOKEEPER", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=1n3omygdur23a1tsoo4n49e8xt;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/ZOOKEEPER", "user": "admin"}
failed: [node1] (item=HBASE) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "HBASE", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=18uowb8b50qx21vqpumqqasroq;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/HBASE", "user": "admin"}
failed: [node1] (item=STORM) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "STORM", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=10s5zpez54rti1ol96n7f2orhh;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/STORM", "user": "admin"}
failed: [node1] (item=KAFKA) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "KAFKA", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=g7746coyq9kindh256rofz7v;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/KAFKA", "user": "admin"}
to retry, use: --limit @../../playbooks/metron_full_install.retry
PLAY RECAP *********************************************************************
node1                      : ok=38changed=28   unreachable=0    failed=1

Ansible failed to complete successfully. Any error output should be

visible above. Please fix these errors and try again.

Re: Metron vagrant start hadoop services faiing

nallen — Wed, 27 Apr 2016 04:09:56 GMT

Have you logged into Ambari to check it out? Try logging into http://node1:8080 as admin/admin. What does that look like? Do you see much red?

Re: Metron vagrant start hadoop services faiing

sunile_manjee — Wed, 27 Apr 2016 04:11:00 GMT

@nallen I have logged into ambari and no red. All looks good. No storm topologies running.

Re: Metron vagrant start hadoop services faiing

nallen — Wed, 27 Apr 2016 04:21:07 GMT

A log should exist at 'deployment/vagrant/singlenode-vagrant/ansible.log'. Can you attach that here in HCC?

Re: Metron vagrant start hadoop services faiing

sunile_manjee — Wed, 27 Apr 2016 04:27:03 GMT

ansiblelog.txt @nallen log attached.

Re: Metron vagrant start hadoop services faiing

bala1 — Wed, 27 Apr 2016 05:12:14 GMT

Getting the same exact error and the ansible.log looks the same.

Re: Metron vagrant start hadoop services faiing

lgeorge — Wed, 27 Apr 2016 06:17:29 GMT

The ansible playbook failed at the end of ambari_install.yml with Ambari & HDP running, so I re-provisioned the single-node cluster with the second half of the playbook: the metron_install.yml script. That deployed all four topologies and the Metron UI. I saw an error in the HDFS section of hadoop_setup so I reran/re-provisioned that part, and now I'm seeing data in the Metron UI. My guess is that the playbook timed out before the HDP components had time to come up. Next I'm going to suspend the VMs before my Mac melts down... the fan's pegged! 🙂

Re: Metron vagrant start hadoop services faiing

jsirota — Wed, 27 Apr 2016 07:51:14 GMT

What laptop are you running this on? I've never gotten this issue and I build this at least once a day. Maybe the older hardware has issues with our VM?

Re: Metron vagrant start hadoop services faiing

ravi1 — Wed, 27 Apr 2016 10:41:42 GMT

Same exact error here too. From Ambari server logs,

Below is the log from ambari-server log which has an error.

metronambariserver.txt

Re: Metron vagrant start hadoop services faiing

sunile_manjee — Wed, 27 Apr 2016 11:05:22 GMT

@lgeorge could you please provide details of how you fixed the issue so it could help others as well. I am using a macbook pro so i believe I have enough resources to bring up a vm.

Re: Metron vagrant start hadoop services faiing

lgeorge — Wed, 27 Apr 2016 12:12:32 GMT

@Sunile Manjee, sure. My approach is more of a workaround than a solution, but I'll do my best to describe it here.

I followed these steps while the VM was still running with Ambari and HDP deployed (the point at which the error occurred).

Make a backup copy of the Vagrantfile in incubator-metron-Metron_0.1BETA_rc7/deployment/vagrant/singlenode-vagrant/
Edit Vagrantfile so that ansible uses the metron_install.yml file instead of the metron_full_install.yml file:
Near the end of the file, change the line with ansible.playbook = "../../playbooks/metron_full_install.yml"
to
ansible.playbook = "../../playbooks/metron_install.yml"
Then save your changes.
From the deployment/vagrant/singlenode-vagrant directory, issue the following command:
vagrant provision
(If the VM is stopped/halted you'd probably want to use vagrant up instead. It shouldn't hurt to run vagrant up; if it isn't needed it will let you know.)
At this point the Metron installation playbook stepped through nine blocks of instructions; looks they're called "plays". You can see the list in the metron_install.yml file, and you can look in the roles directory for a better understanding of the tasks associated with the role in each play (hadoop_setup, mysql-server, etc.).

When the provisioning finished I could see all four topologies in the Storm UI but I didn't see any data. I'd gotten an error during the hadoop_setup role (under hosts: hadoop_client) so I re-ran that part by making a backup copy of metron_install.yml, removing everything except the hadoop_client section from the original file, and running vagrant provision again. A little while later I started seeing data in the Metron UI -- great to see it!

I hope I've remembered everything; let me know if any of it isn't clear.

Re: Metron vagrant start hadoop services faiing

lgeorge — Wed, 27 Apr 2016 12:14:13 GMT

@jsirota, maybe so! MacBook Pro, 2.5 GHz i7 with 16 GB of memory, OS X 10.9.5, vanilla VirtualBox config.

Re: Metron vagrant start hadoop services faiing

bala1 — Wed, 27 Apr 2016 21:14:27 GMT

I went through the same approach and still running into a few more failures. 1) HDFS 2) HBase table creation and 3) install exited after "Build snort" task. WIll try again this mrng and update.

Re: Metron vagrant start hadoop services faiing

cstella — Wed, 27 Apr 2016 21:30:12 GMT

Not an answer, but I noticed the exception in these logs is the same as https://community.hortonworks.com/questions/19734/hortonworks-2-ambari-unable-to-log-on-to-ambari-ui.html

Re: Metron vagrant start hadoop services faiing

cstella — Fri, 29 Apr 2016 00:16:04 GMT

Ok, I just saw with one with a member of the team here on Metron who has a new machine. I think the issue is our ansible scripts interacting negatively with ansible version 2.0.2. All of the members of the team here have 2.0.0.2. While this is corrected, I would suggest reverting to 2.0.0.2 and letting me know if it fixes the issue.

In case you might not know how to do that, what *I* do is remove ansible from homebrew (brew uninstall ansible) and then install directly via pip (pip install ansible==2.0.0.2).

Let me know and sorry for the bug!

Re: Metron vagrant start hadoop services faiing

gvetticaden1 — Fri, 29 Apr 2016 01:29:18 GMT

on my new mac book pro, i saw the same issue this morning. I followed casey's directions and it fixed it for me. Until we have a better fix, I have updated the documentation to instruction ansible 2.0.0.2 be installed): https://community.hortonworks.com/articles/24818/metron-tech-preview-1-install-instructions-on-sing.html

Re: Metron vagrant start hadoop services faiing

ravi1 — Fri, 29 Apr 2016 02:01:19 GMT

Yes. This is the issue. Once i switched to 2.0.0.2 of ansible, I was able to go all the way. Earlier version was 2.0.2.0 and not 2.2 though. Thanks!

Re: Metron vagrant start hadoop services faiing

cstella — Fri, 29 Apr 2016 02:01:20 GMT

Yes! 2.0.2, not 2.2 I misstyped. I will correct the original answer.