Metron vagrant start hadoop services faiing

[#1]

Trying to get a instance of metron up and running. Ambari is up so not sure if things are ok. I performed vagrant up and things went smoothly until task to start hadoop services. here is my error log:

TASK [ambari_config : Start All Hadoop Services node1] *************************
failed: [node1] (item=HDFS) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "HDFS", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=tbclunfsajxy1bnyfpvk7twl;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/HDFS", "user": "admin"}
failed: [node1] (item=YARN) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "YARN", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=1k3byxj6l24791q0zzxpojzmpe;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/YARN", "user": "admin"}
failed: [node1] (item=MAPREDUCE2) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "MAPREDUCE2", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=onwvdexdk4x513bjb4tzj0wfz;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/MAPREDUCE2", "user": "admin"}
failed: [node1] (item=ZOOKEEPER) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "ZOOKEEPER", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=1n3omygdur23a1tsoo4n49e8xt;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/ZOOKEEPER", "user": "admin"}
failed: [node1] (item=HBASE) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "HBASE", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=18uowb8b50qx21vqpumqqasroq;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/HBASE", "user": "admin"}
failed: [node1] (item=STORM) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "STORM", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=10s5zpez54rti1ol96n7f2orhh;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/STORM", "user": "admin"}
failed: [node1] (item=KAFKA) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "KAFKA", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=g7746coyq9kindh256rofz7v;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/KAFKA", "user": "admin"}
to retry, use: --limit @../../playbooks/metron_full_install.retry
PLAY RECAP *********************************************************************
node1                      : ok=38changed=28   unreachable=0    failed=1

Ansible failed to complete successfully. Any error output should be

visible above. Please fix these errors and try again.

[#2]

Not an answer, but I noticed the exception in these logs is the same as https://community.hortonworks.com/questions/19734/hortonworks-2-ambari-unable-to-log-on-to-ambari-ui...

[#3]

@lgeorge could you please provide details of how you fixed the issue so it could help others as well. I am using a macbook pro so i believe I have enough resources to bring up a vm.

[#4]

@Sunile Manjee, sure. My approach is more of a workaround than a solution, but I'll do my best to describe it here.

I followed these steps while the VM was still running with Ambari and HDP deployed (the point at which the error occurred).

1. Make a backup copy of the Vagrantfile in incubator-metron-Metron_0.1BETA_rc7/deployment/vagrant/singlenode-vagrant/
2. Edit Vagrantfile so that ansible uses the metron_install.yml file instead of the metron_full_install.yml file:

   Near the end of the file, change the line with ansible.playbook = "../../playbooks/metron_full_install.yml"

   to

   ansible.playbook = "../../playbooks/metron_install.yml"

   Then save your changes.

3. From the deployment/vagrant/singlenode-vagrant directory, issue the following command:

   vagrant provision

   (If the VM is stopped/halted you'd probably want to use vagrant up instead. It shouldn't hurt to run vagrant up; if it isn't needed it will let you know.)

   At this point the Metron installation playbook stepped through nine blocks of instructions; looks they're called "plays". You can see the list in the metron_install.yml file, and you can look in the roles directory for a better understanding of the tasks associated with the role in each play (hadoop_setup, mysql-server, etc.).

When the provisioning finished I could see all four topologies in the Storm UI but I didn't see any data. I'd gotten an error during the hadoop_setup role (under hosts: hadoop_client) so I re-ran that part by making a backup copy of metron_install.yml, removing everything except the hadoop_client section from the original file, and running vagrant provision again. A little while later I started seeing data in the Metron UI -- great to see it!

I hope I've remembered everything; let me know if any of it isn't clear.

[#5]

I went through the same approach and still running into a few more failures. 1) HDFS 2) HBase table creation and 3) install exited after "Build snort" task. WIll try again this mrng and update.

[#6 · Accepted solution]

Ok, I just saw with one with a member of the team here on Metron who has a new machine. I think the issue is our ansible scripts interacting negatively with ansible version 2.0.2. All of the members of the team here have 2.0.0.2. While this is corrected, I would suggest reverting to 2.0.0.2 and letting me know if it fixes the issue.

In case you might not know how to do that, what *I* do is remove ansible from homebrew (brew uninstall ansible) and then install directly via pip (pip install ansible==2.0.0.2).

Let me know and sorry for the bug!

[#7]

on my new mac book pro, i saw the same issue this morning. I followed casey's directions and it fixed it for me. Until we have a better fix, I have updated the documentation to instruction ansible 2.0.0.2 be installed): https://community.hortonworks.com/articles/24818/metron-tech-preview-1-install-instructions-on-sing....

[#8]

Yes. This is the issue. Once i switched to 2.0.0.2 of ansible, I was able to go all the way. Earlier version was 2.0.2.0 and not 2.2 though. Thanks!

[#9]

Yes! 2.0.2, not 2.2 I misstyped. I will correct the original answer.