Trying to deploy Apache Metron on single node VM, but Ansible and maven dependency failures, please help I am stuck

[#1]

Hi,

I am trying to deploy Apache Metron on a single node VM, but after vagrant up, when I run vagrant provision, it gives me errors on maven dependencies and ansible failed to setup successfully.

I have installed all packages defined for Metron deployment, Ansible 2.0.0.2, Vagrant 1.9.6, Maven 3.3.9, Virtual Box 5.1.30 JDK 1.8 and Python 2.7.5

Also, when I run the command "mvn clean package", metron-config error pops up and it doesnt complete its installation, followed by metron-rest, metron-rest client

[#2]

Hi @Gaurav Bapat, can you post the output of running the command metron-deployment/scripts/platform-info.sh ?

I am guessing this could be due to an older version of node and npm modules. Here is the output from my system, for which I am able to get everything working.

Metron 0.4.2
--
* master
--
--
--
ansible 2.0.0.2
  config file =
  configured module search path = Default w/o overrides
--
Vagrant 1.9.4
--
Python 2.7.12
--
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T22:11:47+05:30)
Maven home: /usr/local/Cellar/maven/3.3.9/libexec
Java version: 1.8.0_91, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.12.6", arch: "x86_64", family: "mac"
--
Docker version 1.12.1, build 23cf638, experimental
--
node
v8.9.1
--
npm
5.5.1
--
Darwin HW12724.local 16.7.0 Darwin Kernel Version 16.7.0: Wed Oct  4 00:17:00 PDT 2017; root:xnu-3789.71.6~1/RELEASE_X86_64 x86_64
--
Total System Memory = 16384 MB
Processor Model: Intel(R) Core(TM) i7-4870HQ CPU
Processor Speed: 2.50GHz
Total Physical Processors: 4
Total cores: 4
Disk information:
/dev/disk1     465Gi  402Gi   63Gi    87% 2338554 4292628725    0%   /
/dev/disk3s1    96Mi   94Mi  1.7Mi    99%      12 4294967267    0%   /Volumes/VirtualBox
/dev/disk4s1    99Mi   97Mi  1.6Mi    99%      12 4294967267    0%   /Volumes/VirtualBox 1
This CPU appears to support virtualization

[#3]

When I run vagrant provision, it gives me metron-config error, I ran mvn clean package command in metron-config folder but it fails to build metron-config and others successfully, please help or is the files in the tar corrupt?

@asubramanian

[#4]

Can you please run these steps and paste the output?

1. cd to the directory Metron top level directory (E.g. cd /home/user1/metron)

2. Run ./metron-deployment/scripts/platform-info.sh

[#5]

Metron 0.4.1
fatal: Not a git repository (or any parent up to mount point /home)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
--
ansible 2.0.0.2
  config file = 
  configured module search path = Default w/o overrides
--
Vagrant 1.9.6
--
Python 2.7.5
--
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T11:41:47-05:00)
Maven home: /usr/local/maven/apache-maven-3.3.9
Java version: 1.8.0_151, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.151-1.b12.el7_4.x86_64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-693.el7.x86_64", arch: "amd64", family: "unix"
--
Docker version 1.12.6, build 85d7426/1.12.6
--
node
v6.12.0
--
npm
3.10.10
--
Linux metron.sequretek.com 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
--
Total System Memory = 9672.71 MB
Processor Model: Intel(R) Core(TM) i3-2120 CPU @ 3.30GHz
Processor Speed: 3291.748 MHz
Total Physical Processors: 2
Total cores: 2
Disk information:
/dev/mapper/centos-root   50G   19G   31G  38% /
/dev/sda1               1014M  179M  836M  18% /boot
/dev/mapper/centos-home   65G  8.6G   56G  14% /home
/dev/sr0                 4.3G  4.3G     0 100% /run/media/gbapat/CentOS 7 x86_64
This CPU appears to support virtualization

@asubramanian Thanks

[#6]

Sure.. you can follow these commands as per your OS - https://www.tecmint.com/install-nodejs-npm-in-centos-ubuntu/.

Please retry compiling Metron once you have upgraded the node. npm and let me know if it fixes the issue.

[#7]

It gives Skipping Ansible version error still

[#8]

after i run vagrant provision, this happensThanks @asubramanian

[#9]

@asubramanian Is it going good and what time will it take to complete?

[#10]

@asubramanian Why this no hosts matched? and I am so thankful that you helped me do this, Thanks alot!!

[#11]

Hey @Gaurav Bapat, you are most welcome! This is looking good. Depending on your the available system resources, the time to deploy will be anywhere between 45 mins to few hours.

Btw, would you mind marking the answer as accepted since the original issue is resolved?

[#12]

@asubramanian

When I restarted my VM, same errors are popping up, like - mvn dependencies, do I need to enter mvn clean packages -Dskiptests everytime and which is the best directory for the same?

Also I am attaching step 3, please let me know how to setup SSH private key?

,

When I restarted the VM, it is again giving me same errors of mvn dependencies, do I need to again enter "mvn clean packages"? , also which is the best directory to enter this command?

I am attached a picture of Step 3 here, please let me know how to setup private SSH key.

@asubramanian Thanks

[#13 · Accepted solution]

Hey @Gaurav Bapat, you need not run the mvn command every time. Just once should suffice. Also, if you have used the full-dev vagrant deployment, you need not have to do anything with Ambari UI (or the Step 3 screenshot) you have pasted.

So, let me paraphrase the steps for you so that you can give this a try once more, afresh. Let me know how it goes.

Assumption: Say your metron folder is located at: /user/gaurav/metron

Step 0 - Cleanup all existing deployments:

* Run the following commands:

cd /user/gaurav/metron/metron-deployment/vagrant/full-dev-platform
vagrant destroy -f

* The above destroy command would terminate and delete the existing running VM. You can also run 'VirtualBox' command to launch the application console, and can delete any other stray VMs that are not required.

Step 1 - Compile metron

* Run the following commands:

cd /user/gaurav/metron
mvn clean compile -DskipTests

* The above command will take about 10 mins or so to complete, and should end with a "SUCCESS" message

Step 2 - Bring up metron VM

* Run the following commands:

cd /user/gaurav/metron/metron-deployment/vagrant/full-dev-platform
vagrant up

* The above command will take about 45 mins to over an hour to complete. I would recommend that you disable auto-sleep of the system in case it is enabled.

At the end of the step 2 above, you should be able to see a message saying that the deployment completed successfully.

[#14]

When I re-run vagrant provision, it is giving me those errors back again 😞 @asubramanian

[#15]

Please dont re-run. Follow the 3 steps what I have mentioned above with the cleanup.

[#16]

Ahh.. wait @Gaurav Bapat, you also need to have Docker installed and running. In the above error message you can see that it says "Cannot connect to the Docker daemon".

Please install Docker (https://docs.docker.com/engine/installation) and ensure that it is running before you proceed with the deployment.

[#17]

@asubramanian

Thank you so much, I am following your steps, but why am I getting npm dependency warnings or angular-cli ?? do i have to type any other command for it?

[#18]

Yeah, I have seen that too. I guess you can ignore them and move on, for now.

[#19]

@asubramanian How do I see the metron UI, node1:8888 or something else?

[#20]

Metron UI comprises the Kibana dashboard (node1:5000), service management (node1:2812), the Management UI (node1:4200) and the Alerts UI (node1:4201). Each of these cater to different things you would like to perform. The Metron site-book is an excellent place to know more about these - https://metron.apache.org/current-book/index.html. Give it a try.

[#21]

You are too good, and can I use Apache NiFi, Flume, Kafka and Spark??