[Apache Metron] Ansible Failed To Complete Successfully

[#1]

Hi,

my platform-info.sh gives me this:

Metron 0.6.0
--
* master
--
commit 62310743a0c1d4a27db47a89cf79496d594c8bcf
Author: justinleet <justinjleet@gmail.com>
Date:   Tue Sep 4 19:53:37 2018 -0400
    METRON-1764 Update version to 0.6.0 (justinleet) closes apache/metron#1183
--
--
ansible 2.5.0
  config file = None
  configured module search path = [u'/home/akaticentos6/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.6/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.6.6 (r266:84292, Aug 18 2016, 15:13:37) [GCC 4.4.7 20120313 (Red Hat 4.4.7-17)]
--
Vagrant 2.1.4
--
vagrant-hostmanager (1.8.9, global)
--
Python 2.7.15
--
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)
Maven home: /opt/maven
Java version: 1.8.0_181, vendor: Oracle Corporation
Java home: /usr/java/jdk1.8.0_181-amd64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-754.el6.x86_64", arch: "amd64", family: "unix"
--
Docker version 1.7.1, build 786b29d/1.7.1
--
node
v0.10.48
--
npm
1.3.6
--
g++ (GCC) 4.8.2 20140120 (Red Hat 4.8.2-15)
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
--
Compiler is C++11 compliant
--
Linux localhost.localdomain 2.6.32-754.el6.x86_64 #1 SMP Tue Jun 19 21:26:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
--
Total System Memory = 9075.27 MB
Processor Model: Intel(R) Core(TM) i5-3570 CPU @ 3.40GHz
Processor Speed: 3392.420 MHz
Total Physical Processors: 2
Total cores: 2
Disk information:
/dev/mapper/vg_akaticentos6-lv_root
/dev/sda1             477M   35M  417M   8% /boot
/dev/sr0               71M   71M     0 100% /media/VMware Tools
This CPU appears to support virtualization

It seems all pre-requisites are fulfilled.

But when I run vagrant provision for metron-deployment/development/centos6, it shows this:

TASK [metron-builder : Build Metron RPM Packages] ******************************
failed: [node1 -> localhost] (item=mvn package -DskipTests -Pbuild-rpms -T 2C) => {"changed": true, "cmd": "mvn package -DskipTests -Pbuild-rpms -T 2C", "delta": "0:00:22.816706", "end": "2018-09-06 16:36:28.706172", "item": "mvn package -DskipTests -Pbuild-rpms -T 2C", "msg": "non-zero return code", "rc": 1, "start": "2018-09-06 16:36:05.889466", "stderr": "Post http:///var/run/docker.sock/v1.19/build?cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=... dial unix /var/run/docker.sock: permission denied. Are you trying to connect to a TLS-enabled daemon without TLS?", "stderr_lines": ["Post http:///var/run/docker.sock/v1.19/build?cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=... dial unix /var/run/docker.sock: permission denied. Are you trying to connect to a TLS-enabled daemon without TLS?"], "stdout": "[INFO] Scanning for projects...\n[INFO] ------------------------------------------------------------------------\n[INFO] Reactor Build Order:\n[INFO] \n[INFO] metron-deployment\n[INFO] metron-rpm\n[INFO] \n[INFO] Using the MultiThreadedBuilder implementation with a thread count of 4\n[INFO]                                                                         \n[INFO] ------------------------------------------------------------------------\n[INFO] Building metron-deployment 0.6.0\n[INFO] ------------------------------------------------------------------------\n[INFO] \n[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-versions) @ metron-deployment ---\n[INFO] \n[INFO] --- jacoco-maven-plugin:0.7.9:prepare-agent (default) @ metron-deployment ---\n[INFO] argLine set to -javaagent:/home/.../.m2/repository/org/jacoco/org.jacoco.agent/0.7.9/org.jacoco.agent-0.7.9-runtime.jar=destfile=/home/.../metron/metron-deployment/target/jacoco.exec\n[INFO] \n[INFO] --- jacoco-maven-plugin:0.7.9:report (report) @ metron-deployment ---\n[INFO] Skipping JaCoCo execution due to missing execution data file.\n[INFO]                                                                         \n[INFO] ------------------------------------------------------------------------\n[INFO] Building metron-rpm 0.6.0\n[INFO] ------------------------------------------------------------------------\n[INFO] \n[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-versions) @ metron-rpm ---\n[INFO] \n[INFO] --- maven-antrun-plugin:1.1:run (default) @ metron-rpm ---\n[INFO] Executing tasks\n     [echo] ******** Displaying value of property ********\n     [echo] ../../../..\n[INFO] Executed tasks\n[INFO] \n[INFO] --- jacoco-maven-plugin:0.7.9:prepare-agent (default) @ metron-rpm ---\n[INFO] argLine set to -javaagent:/home/.../.m2/repository/org/jacoco/org.jacoco.agent/0.7.9/org.jacoco.agent-0.7.9-runtime.jar=destfile=/home/.../metron/metron-deployment/packaging/docker/rpm-docker/target/jacoco.exec\n[INFO] \n[INFO] --- jacoco-maven-plugin:0.7.9:report (report) @ metron-rpm ---\n[INFO] Skipping JaCoCo execution due to missing execution data file.\n[INFO] \n[INFO] --- maven-resources-plugin:3.0.1:copy-resources (copy-rpm-sources) @ metron-rpm ---\n[INFO] Using 'UTF-8' encoding to copy filtered resources.\n[INFO] Copying 1 resource\n[INFO] Copying 1 resource\n[INFO] Copying 1 resource\n[INFO] Copying 1 resource\n[INFO] Copying 1 resource\n[INFO] Copying 1 resource\n[INFO] Copying 1 resource\n[INFO] Copying 1 resource\n[INFO] Copying 1 resource\n[INFO] Copying 1 resource\n[INFO] Copying 1 resource\n[INFO] Copying 1 resource\n[INFO] Copying 1 resource\n[INFO] Copying 1 resource\n[INFO] Copying 1 resource\n[INFO] \n[INFO] --- exec-maven-plugin:1.5.0:exec (docker-build) @ metron-rpm ---\nSending build context to Docker daemon \r\n[ERROR] Command execution failed.\norg.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)\n\tat org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)\n\tat org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:166)\n\tat org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:764)\n\tat org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:711)\n\tat org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:289)\n\tat org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)\n\tat org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)\n\tat org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)\n\tat org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)\n\tat org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)\n\tat org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:185)\n\tat org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:181)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\n[INFO] ------------------------------------------------------------------------\n[INFO] Reactor Summary:\n[INFO] \n[INFO] metron-deployment .................................. SUCCESS [  1.499 s]\n[INFO] metron-rpm ......................................... FAILURE [  8.590 s]\n[INFO] ------------------------------------------------------------------------\n[INFO] BUILD FAILURE\n[INFO] ------------------------------------------------------------------------\n[INFO] Total time: 10.252 s (Wall Clock)\n[INFO] Finished at: 2018-09-06T16:36:17+08:00\n[INFO] Final Memory: 15M/244M\n[INFO] ------------------------------------------------------------------------\n[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.5.0:exec (docker-build) on project metron-rpm: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]\n[ERROR] \n[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.\n[ERROR] Re-run Maven using the -X switch to enable full debug logging.\n[ERROR] \n[ERROR] For more information about the errors and possible solutions, please read the following articles:\n[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException\n[ERROR] \n[ERROR] After correcting the problems, you can resume the build with the command\n[ERROR]   mvn <goals> -rf :metron-rpm", "stdout_lines": ["[INFO] Scanning for projects...", "[INFO] ------------------------------------------------------------------------", "[INFO] Reactor Build Order:", "[INFO] ", "[INFO] metron-deployment", "[INFO] metron-rpm", "[INFO] ", "[INFO] Using the MultiThreadedBuilder implementation with a thread count of 4", "[INFO]                                                                         ", "[INFO] ------------------------------------------------------------------------", "[INFO] Building metron-deployment 0.6.0", "[INFO] ------------------------------------------------------------------------", "[INFO] ", "[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-versions) @ metron-deployment ---", "[INFO] ", "[INFO] --- jacoco-maven-plugin:0.7.9:prepare-agent (default) @ metron-deployment ---", "[INFO] argLine set to -javaagent:/home/.../.m2/repository/org/jacoco/org.jacoco.agent/0.7.9/org.jacoco.agent-0.7.9-runtime.jar=destfile=/home/.../metron/metron-deployment/target/jacoco.exec", "[INFO] ", "[INFO] --- jacoco-maven-plugin:0.7.9:report (report) @ metron-deployment ---", "[INFO] Skipping JaCoCo execution due to missing execution data file.", "[INFO]                                                                         ", "[INFO] ------------------------------------------------------------------------", "[INFO] Building metron-rpm 0.6.0", "[INFO] ------------------------------------------------------------------------", "[INFO] ", "[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-versions) @ metron-rpm ---", "[INFO] ", "[INFO] --- maven-antrun-plugin:1.1:run (default) @ metron-rpm ---", "[INFO] Executing tasks", "     [echo] ******** Displaying value of property ********", "     [echo] ../../../..", "[INFO] Executed tasks", "[INFO] ", "[INFO] --- jacoco-maven-plugin:0.7.9:prepare-agent (default) @ metron-rpm ---", "[INFO] argLine set to -javaagent:/home/.../.m2/repository/org/jacoco/org.jacoco.agent/0.7.9/org.jacoco.agent-0.7.9-runtime.jar=destfile=/home/.../metron/metron-deployment/packaging/docker/rpm-docker/target/jacoco.exec", "[INFO] ", "[INFO] --- jacoco-maven-plugin:0.7.9:report (report) @ metron-rpm ---", "[INFO] Skipping JaCoCo execution due to missing execution data file.", "[INFO] ", "[INFO] --- maven-resources-plugin:3.0.1:copy-resources (copy-rpm-sources) @ metron-rpm ---", "[INFO] Using 'UTF-8' encoding to copy filtered resources.", "[INFO] Copying 1 resource", "[INFO] Copying 1 resource", "[INFO] Copying 1 resource", "[INFO] Copying 1 resource", "[INFO] Copying 1 resource", "[INFO] Copying 1 resource", "[INFO] Copying 1 resource", "[INFO] Copying 1 resource", "[INFO] Copying 1 resource", "[INFO] Copying 1 resource", "[INFO] Copying 1 resource", "[INFO] Copying 1 resource", "[INFO] Copying 1 resource", "[INFO] Copying 1 resource", "[INFO] Copying 1 resource", "[INFO] ", "[INFO] --- exec-maven-plugin:1.5.0:exec (docker-build) @ metron-rpm ---", "Sending build context to Docker daemon ", "[ERROR] Command execution failed.", "org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)", "\tat org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)", "\tat org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:166)", "\tat org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:764)", "\tat org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:711)", "\tat org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:289)", "\tat org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)", "\tat org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)", "\tat org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)", "\tat org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)", "\tat org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)", "\tat org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:185)", "\tat org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:181)", "\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)", "\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)", "\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)", "\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)", "\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)", "\tat java.lang.Thread.run(Thread.java:748)", "[INFO] ------------------------------------------------------------------------", "[INFO] Reactor Summary:", "[INFO] ", "[INFO] metron-deployment .................................. SUCCESS [  1.499 s]", "[INFO] metron-rpm ......................................... FAILURE [  8.590 s]", "[INFO] ------------------------------------------------------------------------", "[INFO] BUILD FAILURE", "[INFO] ------------------------------------------------------------------------", "[INFO] Total time: 10.252 s (Wall Clock)", "[INFO] Finished at: 2018-09-06T16:36:17+08:00", "[INFO] Final Memory: 15M/244M", "[INFO] ------------------------------------------------------------------------", "[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.5.0:exec (docker-build) on project metron-rpm: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]", "[ERROR] ", "[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.", "[ERROR] Re-run Maven using the -X switch to enable full debug logging.", "[ERROR] ", "[ERROR] For more information about the errors and possible solutions, please read the following articles:", "[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException", "[ERROR] ", "[ERROR] After correcting the problems, you can resume the build with the command", "[ERROR]   mvn <goals> -rf :metron-rpm"]}
NO MORE HOSTS LEFT *************************************************************
    to retry, use: --limit @/home/.../metron/metron-deployment/development/centos6/ansible/playbook.retry
PLAY RECAP *********************************************************************
node1                      : ok=9    changed=3    unreachable=0    failed=1   
Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

What should I check?

[#2]

hi @asubramanian,

I've installed node.js. :

Metron 0.6.0
--
* Metron_0.6.0
--
commit 62310743a0c1d4a27db47a89cf79496d594c8bcf
Author: justinleet <justinjleet@gmail.com>
Date:   Tue Sep 4 19:53:37 2018 -0400
    METRON-1764 Update version to 0.6.0 (justinleet) closes apache/metron#1183
--
 metron-deployment/development/centos6/ansible.cfg          |  3 +++
 metron-deployment/development/centos6/ansible/playbook.yml | 12 ++++++++++--
 2 files changed, 13 insertions(+), 2 deletions(-)
--
ansible 2.5.0
  config file = None
  configured module search path = [u'/home/<user>/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Jul 13 2018, 13:06:57) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]
--
Vagrant 2.1.5
--
vagrant-hostmanager (1.8.9, global)
--
Python 2.7.5
--
Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-18T02:33:14+08:00)
Maven home: /usr/local/src/apache-maven
Java version: 1.8.0_181, vendor: Oracle Corporation, runtime: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-862.14.4.el7.x86_64", arch: "amd64", family: "unix"
--
Docker version 18.09.0-beta3, build 7718f80
--
node
v6.14.3
--
npm
3.10.10
--
g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
--
Compiler is C++11 compliant
--
Linux localhost.localdomain 3.10.0-862.14.4.el7.x86_64 #1 SMP Wed Sep 26 15:12:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
--
Total System Memory = 9670.98 MB
Processor Model: Intel(R) Core(TM) i5-3570 CPU @ 3.40GHz
Processor Speed: 3392.409 MHz
Total Physical Processors: 2
Total cores: 2
Disk information:
/dev/mapper/centos-root   45G   18G   27G  41% /
/dev/sda1               1014M  221M  794M  22% /boot
This CPU appears to support virtualization
<br>

This is the error I got, (6000 char exceeded, so I put it on pastebin):Metron Fourth Error Issue

[#3]

Hi @asubramanian,

I managed to troubleshoot the problem above by changing folder permission for the .npm and manually remove the _locks, then re-run vagrant up.

The thing is, when running vagrant provision, I noticed one of the task were building cluster Ambari. That shouldn't happen for single-VM, right? But it did.

The process ended due to several times CPU locked out/ overcommitment error, and then aborted. I have a feeling that this is because it is creating clusters instead of running on single-VM.

[#4]

Hi @Nor Liana Kamaruzzaman, the single node uses Ambari to deploy the cluster. So what you are seeing is correct.

I am suspecting if there is resource starvation because of which you are seeing the CPU locked out issues. Can you try to find a server with higher resources? Or upgrade the present server?

[#5]

Hey @Nor Liana Kamaruzzaman, this is great news! I am glad you were able to resolve the issues.

Would you mind accepting the answer that helped you the most?

Cheers,

Anand

[#6]

Hey @Nor Liana Kamaruzzaman ,

I am hoping you have got a chance to set up a different system with CentOS 7 and able to proceed further. In the context of this thread, would you mind marking the best answer if you feel the original question was answered? Thanks!

[#7]

Hi @asubramanian,

I've increased the host VM's RAM. It reduces the cpu lock out, and can finally run. 🙂 Thanks.