Support Questions
Find answers, ask questions, and share your expertise

Debugging Metron Deployment with Monit

Debugging Metron Deployment with Monit

Explorer

Hi,

I'm running Metron on a single node, full-dev-platform. I'm not getting any data on Kibana and am trying to debug what happened with Monit, based on the suggestions here: https://community.hortonworks.com/content/kbentry/59801/troubleshooting-missing-events-in-metron-qui...

Unfortunately, I'm not able to bring up Monit UI on http://node1:2812 . Is this the right port as suggested in the above article?

Calling "sudo monit summary" on the console gives me the following:

Process 'yaf'                       Running
Process 'snort'                     Running
Process 'snort-logs'                Execution failed
Process 'pcap-service'              Running
Process 'pcap-replay'               Running
Program 'pcap-parser'               Execution failed
Program 'yaf-parser'                Execution failed
Program 'bro-parser'                Execution failed
Program 'snort-parser'              Execution failed
Process 'mysql'                     Running
Process 'kibana'                    Running
Program 'indexing'                  Execution failed
Program 'enrichment'                Execution failed
Process 'elasticsearch'             Running
Process 'bro'                       Running
System 'node1'                      Running

1) How reliable is Monit's output for debugging purposes? Do you recommend using it or is there a better way?

2) It seems that the reason I'm not getting any data on Kibana is the fact that snort-parser, yaf-parser, pcap-parser (and possibly the others whose execution failed) are not running. I tried using Monit to restart them, but that didn't work as the processes fail to execute.

I should add that I do not see any alerts on Ambari, indicating an error. Mysql is running, Storm has "enrichment", "pcap", "snort", and "yaf" topologies, and the following topics have been created on Kafka:

bro
enrichments
indexing
indexing_error
parser_error
parser_invalid
pcap
snort
yaf 

Furthermore, It seems that some data is coming through the topics, as the following command in "/usr/hdp/current/kafka-broker":

$ ./bin/kafka-console-consumer.sh --topic bro --zookeeper localhost:2181

results in something like

{metadata.broker.list=node1:6667, request.timeout.ms=30000, client.id=console-consumer-80191, security.protocol=PLAINTEXT}
{"http": {"ts":1477100588.011956,"uid":"CK38Oa4To9c4C0Hi0k","id.orig_h":"192.168.138.158","id.orig_p":49205,"id.resp_h":"95.163.121.204","id.resp_p":80,"trans_depth":1,"method":"GET","host":"7oqnsnzwwnm6zb7y.gigapaysun.com","uri":"/11iQmfg","user_agent":"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)","request_body_len":0,"response_body_len":3289,"status_code":200,"status_msg":"OK","tags":[],"resp_fuids":["F9oD3k2wgqQ6WzQot1"],"resp_mime_types":["text/html"]}}
{"http": {"ts":1477100590.698489,"uid":"Cpax4X1rrfzVYDVAT4","id.orig_h":"192.168.138.158","id.orig_p":49206,"id.resp_h":"95.163.121.204","id.resp_p":80,"trans_depth":1,"method":"GET","host":"7oqnsnzwwnm6zb7y.gigapaysun.com","uri":"/img/style.css","referrer":"http://7oqnsnzwwnm6zb7y.gigapaysun.com/11iQmfg","user_agent":"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)","request_body_len":0,"response_body_len":4492,"status_code":200,"status_msg":"OK","tags":[],"resp_fuids":["FVYVnk4BazYbfR3sOg"],"resp_mime_types":["text/plain"]}}
{"http": {"ts":1477100592.541302,"uid":"Cpax4X1rrfzVYDVAT4","id.orig_h":"192.168.138.158","id.orig_p":49206,"id.resp_h":"95.163.121.204","id.resp_p":80,"trans_depth":2,"method":"GET","host":"7oqnsnzwwnm6zb7y.gigapaysun.com","uri":"/img/flags/it.png","referrer":"http://7oqnsnzwwnm6zb7y.gigapaysun.com/11iQmfg","user_agent":"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)","request_body_len":0,"response_body_len":552,"status_code":200,"status_msg":"OK","tags":[],"resp_fuids":["Ff0bri1CjR5gXsAuF4"],"resp_mime_types":["image/png"]}}

But

$ ./bin/kafka-console-consumer.sh --topic snort --zookeeper localhost:2181

only results in

{metadata.broker.list=node1:6667, request.timeout.ms=30000, client.id=console-consumer-49141, security.protocol=PLAINTEXT}

which I guess is do to snort-logs process not getting executed (is that correct?).

I appreciate any help you could provide to debug this deployment.

1 REPLY 1
Highlighted

Re: Debugging Metron Deployment with Monit

Guru

When running the full dev platform, you will usually run into problems with lack of capacity if all services are turned on. Please make sure that you increase the RAM available to your virtual machine (8GB is really an absolute minimum for the full platform).

The two most common scenarios here are that you do not have sufficient supervisor slots to run all the topologies, so some will not start. You can increase the slots by adding some ports to supervisor.slots.ports in the Storm config. I usually put 6 or so in on the dev platforms to run everything. The other thing that tends to happen is memory constraints killing the mysql service used for the geo-enrichment. You can try restarting mysqld on the box to help with this.

Other things you should look at include the Storm UI (Ambari -> Storm -> Quick Links -> Storm UI, http://node1:8744/) which will show you more details on the topologies.

Often I find that I have to turn off monit to preserve RAM on the dev platforms. This can make controlling of the Storm topologies a little complex, but as we move towards the Ambari Mpack based install, this will move into the Ambari interface in any case, and the need for monit will be much reduced.

Don't have an account?