Member since
11-26-2017
88
Posts
0
Kudos Received
0
Solutions
04-06-2018
06:12 AM
Hello Community, I need help, and I hope you someone can resolve my issue. I have installed Apache Metron 0.4.1 on server and I want to get syslogs to NiFi and then to Kibana dashboard, has anyone tried doing that? I want to know which processors do I use in NiFi, I am using ListenSyslogs and PutKafka processor in NiFi, but it is giving me failed acks with Kafka error. Also I have created a Kafka topic but I want to know which processor do I use here? And how to get syslogs to Metron UI, also is it possible to get them or do I drop this plan.?
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache Metron
-
Apache NiFi
04-03-2018
11:29 AM
Metron 0.4.1 fatal: Not a git repository (or any parent up to mount point /home) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). -- ansible 2.0.0.2 config file = configured module search path = Default w/o overrides -- Vagrant 1.9.6 -- Python 2.7.5 -- Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T22:11:47+05:30) Maven home: /opt/maven/3.3.9 Java version: 1.8.0_161, vendor: Oracle Corporation Java home: /opt/jdk1.8.0_161/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-327.el7.x86_64", arch: "amd64", family: "unix" -- ./platform-info.sh: line 64: docker: command not found -- node v6.12.3 -- npm 3.10.10 -- Linux localhost.localdomain 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux -- Total System Memory = 48112 MB Processor Model: Intel(R) Xeon(R) CPU E5-2420 v2 @ 2.20GHz Processor Speed: 2163.820 MHz Processor Speed: 2054.593 MHz Processor Speed: 2200.000 MHz Processor Speed: 2199.312 MHz Processor Speed: 2045.656 MHz Processor Speed: 2198.882 MHz Processor Speed: 2199.312 MHz Processor Speed: 2200.343 MHz Processor Speed: 2197.593 MHz Processor Speed: 2177.656 MHz Processor Speed: 2199.914 MHz Total Physical Processors: 12 Total cores: 72 Disk information: /dev/mapper/centos-root 4.0T 13G 3.9T 1% / /dev/sda2 238G 199M 238G 1% /boot /dev/sda1 100G 11M 100G 1% /boot/efi /dev/mapper/centos-home 3.0T 9.7G 3.0T 1% /home /dev/mapper/centos-var 1.8T 1.2G 1.8T 1% /var This CPU appears to support virtualization
... View more
04-03-2018
06:57 AM
I run mvn clean compile -DskipTests -PbuildRmps in metron-deployment folder
... View more
04-03-2018
05:17 AM
@asubramanian Hey Anand, I have downloaded incubator-metron, apache-metron-0.4.1-rc4 & apache-metron-0.4.2-rc2 but none of them are able to build properly, it is giving me mvn failed to build error. I have correctly installed maven 3.3.9 and also put correct path in bash_profile. Can you help me out with proper installation for the same. Regards, Gaurav
... View more
Labels:
- Labels:
-
Apache Metron
01-16-2018
03:48 AM
@Jasper My Kafka processor is giving me this error - "Failed while waiting for acks from Kafka error" Do I need to configure Kafka & ZK on vagrant ssh?
... View more
01-15-2018
12:20 PM
Hello All, I have imported logs from HP logger to NiFi using
ListenSyslog processor but when I am connecting it to PutKafka
processor, it is giving me failed while waiting for acks from Kafka
error. Can you give me Kafka and Zookeeper configs? I have set
ZOOKEEPER to node1:2181 & BROKERLIST to node1:6667 but still why I
am getting this error?
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache Metron
01-15-2018
12:15 PM
Hello All, I have imported logs from HP logger to NiFi using ListenSyslog processor but when I am connecting it to PutKafka processor, it is giving me failed while waiting for acks from Kafka error. Can you give me Kafka and Zookeeper configs? I have set ZOOKEEPER to node1:2181 & BROKERLIST to node1:6667 but still why I am getting this error?
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache Metron
01-04-2018
09:28 AM
I have deployed Metron on single node using version 0.4.3, I got my logs through NiFi into Kafka topic but I cant find a way out to configure it on Elastic Search and Kibana. I was told that since this is a new version I have to configure indexing and parsing topology on my own but I dont know how do I do that I have attached some screenshots with this post Please Help !!
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Metron
01-04-2018
09:18 AM
@Jasper None of the links are working 😞
... View more
01-04-2018
06:01 AM
@asubramanian I am using Metron 0.4.2, my logs are hitting NiFi and my Kafka topic but I cant find them in Kibana, I was told that since this is an incubator version I will have to configure Indexing & Parsing topology, I dont know how do I do that so can you give me any workaround. Or do I need to change the Metron version to 0.3 or 0.4.0/4.1??
... View more
01-04-2018
04:22 AM
@Jasper Which version of Metron would you recommend me to use so that I can get the logs directly in ES & KIBANA?
... View more
01-04-2018
04:22 AM
@Jasper Which version of Metron would you recommend me to use so that I can get the logs directly in ES & KIBANA?
... View more
01-04-2018
03:30 AM
@Jasper I am not understanding after getting logs into KAFKA why it is not showing up on KIBANA, I cant find any indexing topology or parsing topology in Ambari, nor do I find any tutorial for the same. Do you have any solution for this, and then I have to apply Machine Learning on the logs and do analytics Cant find any way through, need help & Thank You!!
... View more
01-03-2018
11:13 AM
I have my logs coming into Metron Alerts UI but I cant see them in Kibana, I dont have any tutorial on how do I integrate my logs into the dashboard. Do you have any tutotrial on how do I get them in the dashboard, I cant find any option in ambari and my logs mappings are also empty. I can see my topic in Storm UI and it is emitting but it has yellow status on port 9300. Also I dont have my logs as source and I can only see Bro & Snort and not my log as a source. Thanks for your reply!
... View more
01-03-2018
03:49 AM
@Jasper Thank you for your reply, how do I configure Kafka and Zookeeper ?? I have followed the above tutorial and set up KAFKA & ZOOKEEPER to node1:6667 and node1:2181 But I am still not able to get the data to KAFKA and then to KIBANA Please help & Thanks!!
... View more
01-02-2018
09:31 AM
Hey Jasper, Thank you for your reply, I have ingested logs through NiFi and it is showing in metron management UI but I cant see them on Kibana dashboard how do I configure them through ES, what is the ideal configuration to setup Metron on single node? @Jasper
... View more
12-28-2017
09:05 AM
Dear Metron Community, I have setup metron full-dev on single node, all components went up normally but when I added some services all components failed and when I checked the alerts it shows Connection failed - Error No 111 on node1 Please help me here I have attached screenshots
... View more
Labels:
- Labels:
-
Apache Metron
12-18-2017
03:06 AM
Even I am facing the same issue of connection, please help me resolve the issue Do I need to create any certificate keys??
... View more
12-15-2017
12:07 PM
@asubramanian SELinux is preventing /usr/bin/install from read access Also there is TLS handshake error and connection refused error on my Hbase&Metron component
... View more
12-12-2017
11:14 AM
@asubramanian All the components get failed when I run mvn clean install from the top directory, what can be the error??
... View more
12-08-2017
10:23 AM
When I do vagrant up , I get this error, from were do I download the Apache-metron folder ?? @asubramanian
... View more
12-07-2017
08:53 AM
I have attached few screenshots with this post, I am not able to setup Metron on a single node VM using ansible and Vagrant, I have used 13 GB RAM and i5 Core processor, I have installed all the required packages but still facing this issue of components heartbeat getting lost Is this because of system crunch or I have not installed some of the packages still?? I see metron up for some time and it fails back and doesnt get up 😞 When I drill down deep it states connection failed ERROR No 111 connection refused on node1:50070 I updated python 2.7.5 to 2.7.12 and the problem still persists Please help!!!
... View more
Labels:
- Labels:
-
Apache Metron
12-07-2017
08:25 AM
After increasing the timeout, the error still persists and do you think this might be because of system coonfiguration??
... View more
12-07-2017
04:36 AM
@Akhil S Naik I cant get metron components up after upgrading to python 2.7.12 and verify = disable
... View more
12-06-2017
12:31 PM
@asubramanian Metron Indexing fails to install, what might be the reason?? fatal: [node1]: FAILED! => {"changed": false, "failed": true, "invocation": {"module_args": {"blueprint_name": "metron_blueprint", "blueprint_var": {"groups": [{"cardinality": 1, "components": [{"name": "NAMENODE"}, {"name": "SECONDARY_NAMENODE"}, {"name": "RESOURCEMANAGER"}, {"name": "HISTORYSERVER"}, {"name": "ZOOKEEPER_SERVER"}, {"name": "NIMBUS"}, {"name": "STORM_UI_SERVER"}, {"name": "DRPC_SERVER"}, {"name": "HBASE_MASTER"}, {"name": "HBASE_CLIENT"}, {"name": "APP_TIMELINE_SERVER"}, {"name": "DATANODE"}, {"name": "HDFS_CLIENT"}, {"name": "NODEMANAGER"}, {"name": "YARN_CLIENT"}, {"name": "MAPREDUCE2_CLIENT"}, {"name": "ZOOKEEPER_CLIENT"}, {"name": "SUPERVISOR"}, {"name": "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name": "KIBANA_MASTER"}, {"name": "METRON_INDEXING"}, {"name": "METRON_PROFILER"}, {"name": "METRON_ENRICHMENT_MASTER"}, {"name": "METRON_PARSERS"}, {"name": "METRON_REST"}, {"name": "METRON_MANAGEMENT_UI"}, {"name": "METRON_ALERTS_UI"}, {"name": "ES_MASTER"}], "configurations": [], "name": "host_group_1"}], "required_configurations": [{"metron-env": {"es_hosts": "node1", "storm_rest_addr": "http://node1:8744", "zeppelin_server_url": "node1:9995"}}, {"metron-rest-env": {"metron_jdbc_driver": "org.h2.Driver", "metron_jdbc_password": "root", "metron_jdbc_platform": "h2", "metron_jdbc_url": "jdbc:h2:file:~/metrondb", "metron_jdbc_username": "root"}}, {"kibana-env": {"kibana_default_application": "dashboard/Metron-Dashboard", "kibana_es_url": "http://node1:9200", "kibana_log_dir": "/var/log/kibana", "kibana_pid_dir": "/var/run/kibana", "kibana_server_port": 5000}}], "stack_name": "HDP", "stack_version": "2.5"}, "cluster_name": "metron_cluster", "cluster_state": "present", "configurations": [{"zoo.cfg": {"dataDir": "/data1/hadoop/zookeeper"}}, {"hadoop-env": {"dtnode_heapsize": 512, "hadoop_heapsize": 1024, "namenode_heapsize": 2048, "namenode_opt_permsize": "128m"}}, {"hbase-env": {"hbase_master_heapsize": 512, "hbase_regionserver_heapsize": 512, "hbase_regionserver_xmn_max": 512}}, {"hdfs-site": {"dfs.datanode.data.dir": "/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data", "dfs.journalnode.edits.dir": "/data1/hadoop/hdfs/journalnode", "dfs.namenode.checkpoint.dir": "/data1/hadoop/hdfs/namesecondary", "dfs.namenode.name.dir": "/data1/hadoop/hdfs/namenode", "dfs.replication": 1}}, {"yarn-env": {"apptimelineserver_heapsize": 512, "nodemanager_heapsize": 512, "resourcemanager_heapsize": 1024, "yarn_heapsize": 512}}, {"mapred-env": {"jobhistory_heapsize": 256}}, {"mapred-site": {"mapreduce.jobhistory.recovery.store.leveldb.path": "/data1/hadoop/mapreduce/jhs", "mapreduce.map.java.opts": "-Xmx1024m", "mapreduce.map.memory.mb": 1229, "mapreduce.reduce.java.opts": "-Xmx1024m", "mapreduce.reduce.memory.mb": 1229}}, {"yarn-site": {"yarn.nodemanager.local-dirs": "/data1/hadoop/yarn/local", "yarn.nodemanager.log-dirs": "/data1/hadoop/yarn/log", "yarn.nodemanager.resource.memory-mb": 4096, "yarn.timeline-service.leveldb-state-store.path": "/data1/hadoop/yarn/timeline", "yarn.timeline-service.leveldb-timeline-store.path": "/data1/hadoop/yarn/timeline"}}, {"storm-site": {"storm.local.dir": "/data1/hadoop/storm", "supervisor.slots.ports": "[6700, 6701, 6702, 6703, 6704]", "topology.classpath": "/etc/hbase/conf:/etc/hadoop/conf"}}, {"kafka-env": {"content": "\n#!/bin/bash\n\n# Set KAFKA specific environment variables here.\n\n# The java implementation to use.\nexport KAFKA_HEAP_OPTS=\"-Xms256M -Xmx256M\"\nexport KAFKA_JVM_PERFORMANCE_OPTS=\"-server -XX:+UseG1GC -XX:+DisableExplicitGC -Djava.awt.headless=true\"\nexport JAVA_HOME={{java64_home}}\nexport PATH=$PATH:$JAVA_HOME/bin\nexport PID_DIR={{kafka_pid_dir}}\nexport LOG_DIR={{kafka_log_dir}}\nexport KAFKA_KERBEROS_PARAMS={{kafka_kerberos_params}}\n# Add kafka sink to classpath and related depenencies\nif [ -e \"/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\" ]; then\n export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\n export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/lib/*\nfi\nif [ -f /etc/kafka/conf/kafka-ranger-env.sh ]; then\n . /etc/kafka/conf/kafka-ranger-env.sh\nfi"}}, {"kafka-broker": {"delete.topic.enable": "true", "log.dirs": "/data1/kafka-log"}}, {"metron-rest-env": {"metron_spring_profiles_active": "dev"}}, {"metron-parsers-env": {"parsers": "bro,snort"}}, {"elastic-site": {"gateway_recover_after_data_nodes": 1, "index_number_of_replicas": 0, "index_number_of_shards": 1, "masters_also_are_datanodes": "1", "network_host": "[ _local_, _eth1_ ]", "zen_discovery_ping_unicast_hosts": "[ node1 ]"}}], "host": "node1", "password": "admin", "port": 8080, "username": "admin", "wait_for_complete": true}, "module_name": "ambari_cluster_state"}, "msg": "Request failed with status FAILED"}
PLAY RECAP *********************************************************************
node1 : ok=55 changed=32 unreachable=0 failed=1
Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.
... View more
12-06-2017
08:58 AM
What are the maven commands that need to be run before vagrant up, I use mvn clean compile -DskipTests in the top metron directory
... View more
12-06-2017
08:55 AM
I want to know what is the ideal configuration for RAM and CPU to install Metron and also add extra services in the cluster, I used 13 GB RAM but my components lose heartbeat or face connection failed (ERR NO 111) errors Also what is the ideal Python version as I installed Python 2.7.5 Need urgent help
... View more
Labels:
- Labels:
-
Apache Metron
12-06-2017
08:43 AM
@nallen Hey, even I am not able to setup Metron on a single node VM, it is giving me heartbeat lost error and when I check ambari-agent logs it gives me SSL certificate verify failed Need Help!!
... View more