Hi Anil - One problem here is that a failed assignment expression in the REPL does not provide a helpful error message. I submitted a fix for this here https://github.com/apache/metron/pull/966. To work around that in the REPL, you can just do something like the following to test your Profiler definition; basically don't use assignment. [Stellar]>>> conf := SHELL_EDIT(conf) { "profiles":[ { "profile":"demo_iplogon_failed", "foreach":"ip_address", "onlyif":"source.type == 'demo_windowsnxlog' and event_id == 4625", "init":{ "count":"0" }, "update":{ "count":"count + 1" }, "result":{ "profile":"count", "triage":{ "logon_failed_count":"count" } } } ] } [Stellar]>>> [Stellar]>>> PROFILER_INIT(conf) The issue with the profile definition, is that you don't have a 'result/profile' expression. The 'result/profile' expression which persists the data in HBase is required. Just add one like so below. [Stellar]>>> conf { "profiles":[ { "profile":"demo_iplogon_failed", "foreach":"ip_address", "onlyif":"source.type == 'demo_windowsnxlog' and event_id == 4625", "init":{ "count":"0" }, "update":{ "count":"count + 1" }, "result":{ "profile":"count", "triage":{ "logon_failed_count":"count" } } } ] } [Stellar]>>> PROFILER_INIT(conf) Profiler{1 profile(s), 0 messages(s), 0 route(s)} ... View more

Did you change the profile duration of the Profiler? By default, it is 15 minutes, but in the link that you sent it tells you to change the duration to 1 minute. I would guess that your Profiler configuration probably looks like this. profiler.period.duration=1 profiler.period.duration.units=MINUTES If so, you also need to change the profiler duration on the client side to match. You can do this in a couple different ways. For example, change the client-side equivalent settings in global settings first, then fetch the data. %define profiler.client.period.duration := "1" %define profiler.client.period.duration.units := "MINUTES" PROFILE_GET("url-length", "127.0.0.1", PROFILE_FIXED(5,"HOURS")) Or simply override those values when fetching the data. PROFILE_GET("url-length", "127.0.0.1", PROFILE_FIXED(5,"HOURS", {'profiler.client.period.duration' : '1', 'profiler.client.period.duration.units' : 'MINUTES'}) ... View more

> 1. How to get the pcap data collected/stored in HDFS indexed into ElasticSearch? In Apache Metron there is not a mechanism to ingest raw pcap data into Elasticsearch. I have found a search index like Elasticsearch more useful for higher level meta information like flows. There is a tool called Pcap Query to search and retrieve slices of the raw pcap stored in HDFS. This queries against the data stored in HDFS and returns a libpcap-compliant file containing the raw pcap data that you can then load into 3rd party tools like Wireshark. > 2. How to get the pcap panel on Metron dashboard like the old version of Metron? The Pcap Panel from the original OpenSOC project was not carried forward due to technical limitations. ... View more

Yes, we do that a lot for testing. First, use a tool like 'tcpreplay' to replay a pcap file to a network interface. There is even a simple tool in Metron (https://github.com/apache/metron/tree/master/metron-deployment/roles/pcap_replay) that effectively wraps 'tcpreplay' to make it easy to replay packet captures to a virtual network interface. Then use 'pycapa' in producer mode to sniff the packets from that network interface and land them in Kafka. ... View more