Metron vagrant start hadoop services faiing

[#1]

Trying to get a instance of metron up and running. Ambari is up so not sure if things are ok. I performed vagrant up and things went smoothly until task to start hadoop services. here is my error log:

TASK [ambari_config : Start All Hadoop Services node1] *************************
failed: [node1] (item=HDFS) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "HDFS", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=tbclunfsajxy1bnyfpvk7twl;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/HDFS", "user": "admin"}
failed: [node1] (item=YARN) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "YARN", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=1k3byxj6l24791q0zzxpojzmpe;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/YARN", "user": "admin"}
failed: [node1] (item=MAPREDUCE2) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "MAPREDUCE2", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=onwvdexdk4x513bjb4tzj0wfz;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/MAPREDUCE2", "user": "admin"}
failed: [node1] (item=ZOOKEEPER) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "ZOOKEEPER", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=1n3omygdur23a1tsoo4n49e8xt;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/ZOOKEEPER", "user": "admin"}
failed: [node1] (item=HBASE) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "HBASE", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=18uowb8b50qx21vqpumqqasroq;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/HBASE", "user": "admin"}
failed: [node1] (item=STORM) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "STORM", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=10s5zpez54rti1ol96n7f2orhh;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/STORM", "user": "admin"}
failed: [node1] (item=KAFKA) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "KAFKA", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=g7746coyq9kindh256rofz7v;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/KAFKA", "user": "admin"}
to retry, use: --limit @../../playbooks/metron_full_install.retry
PLAY RECAP *********************************************************************
node1                      : ok=38changed=28   unreachable=0    failed=1

Ansible failed to complete successfully. Any error output should be

visible above. Please fix these errors and try again.

[#2]

Have you logged into Ambari to check it out? Try logging into http://node1:8080 as admin/admin. What does that look like? Do you see much red?

[#3]

@nallen I have logged into ambari and no red. All looks good. No storm topologies running.

[#4]

A log should exist at 'deployment/vagrant/singlenode-vagrant/ansible.log'. Can you attach that here in HCC?

[#5]

ansiblelog.txt @nallen log attached.

[#6]

Getting the same exact error and the ansible.log looks the same.

[#7]

The ansible playbook failed at the end of ambari_install.yml with Ambari & HDP running, so I re-provisioned the single-node cluster with the second half of the playbook: the metron_install.yml script. That deployed all four topologies and the Metron UI. I saw an error in the HDFS section of hadoop_setup so I reran/re-provisioned that part, and now I'm seeing data in the Metron UI. My guess is that the playbook timed out before the HDP components had time to come up. Next I'm going to suspend the VMs before my Mac melts down... the fan's pegged! 🙂

[#8]

What laptop are you running this on? I've never gotten this issue and I build this at least once a day. Maybe the older hardware has issues with our VM?

[#9]

@jsirota, maybe so! MacBook Pro, 2.5 GHz i7 with 16 GB of memory, OS X 10.9.5, vanilla VirtualBox config.

[#10]

Same exact error here too. From Ambari server logs,

Below is the log from ambari-server log which has an error.

metronambariserver.txt