question Re: Exception in indexingBolt of indexing topology in Archives of Support Questions (Read Only)

Exception in indexingBolt of indexing topology

308741831 — Fri, 24 Feb 2017 13:30:23 GMT

After runing Metron a little while,I received this exception:

index [bro_index_2017.02.23.16], type [bro_doc], id [AVpp_hu_luwdJ-LP4qUA], message [MapperParsingException[failed to parse [ip_dst_addr]]; nested: IllegalArgumentException[failed to parse ip [ff02::0001:0003], not a valid ipv4 address (4 dots)];]

How do i resolve it ? I'll appreciate it for any help!

Re: Exception in indexingBolt of indexing topology

cstella — Fri, 24 Feb 2017 14:01:05 GMT

Well, the problem is actually in the elasticsearch indexing templates. Normally, I'd say that you could use a message filter to filter out the IPv6 data in the parser, but I know that they don't work in HCS 1.0. As a workaround, you could transform the IPv6 addresses to 0.0.0.0 and they'll index. You can also save off the old address in a new field. This would be how you would do it with Stellar field transformations.

Edit $METRON_HOME/config/zookeeper/parsers/bro.json to add the "fieldTransformations" section, like so:

{
  "parserClassName":"org.apache.metron.parsers.bro.BasicBroParser",
  "sensorTopic":"bro",
  "parserConfig": {},
  "fieldTransformations" : [
    {
      "transformation" : "STELLAR"
    ,"output" : [ "raw_dst_ip"
                , "ip_dst_addr"
                ]
    ,"config" : {
 "raw_dst_ip" : "ip_dst_addr"
,"ip_dst_addr" : "if IS_IP(ip_dst_addr, 'IPV4') then ip_dst_addr else '0.0.0.0'"
                }
    }
                           ]
}

If things work out like they should, you'll have a raw_dst_ip field and ip_dst_addr will either be IPv4 or '0.0.0.0', which will index just fine.

In the next release, you'll have a message filter that works so you could drop them easier.

Hope this helps! Report back if you get into trouble.

Re: Exception in indexingBolt of indexing topology

308741831 — Fri, 24 Feb 2017 14:01:06 GMT

@cstella is there any approches to do this?

Re: Exception in indexingBolt of indexing topology

cstella — Fri, 24 Feb 2017 14:01:06 GMT

We do not currently support IPv6 addresses in Metron. You have unfortunately hit https://issues.apache.org/jira/browse/METRON-293

Re: Exception in indexingBolt of indexing topology

308741831 — Fri, 24 Feb 2017 14:01:07 GMT

so i wanna disable ipv6 in bro ,do you know how to do that?

Re: Exception in indexingBolt of indexing topology

308741831 — Fri, 24 Feb 2017 14:01:08 GMT

i just do not want ipv6 show up in bro logs

Re: Exception in indexingBolt of indexing topology

cstella — Fri, 24 Feb 2017 14:03:22 GMT

I should point out that you will need to, after you make that change, push the configs to zookeeper via $METRON_HOME/bin/zk_load_configs.sh --mode PUSH -i $METRON_HOME/config/zookeeper -z $ZK_QUORUM

where ZK_QUORUM is something like hostname:2181

Re: Exception in indexingBolt of indexing topology

308741831 — Fri, 24 Feb 2017 14:31:12 GMT

actually ,i hope bro logs capture ipv4 info only ,is there any configuration to set

Re: Exception in indexingBolt of indexing topology

308741831 — Fri, 24 Feb 2017 15:03:53 GMT

or why dont you translate ipv6 to ipv4 with stellar script?

Re: Exception in indexingBolt of indexing topology

zeolla — Sat, 25 Feb 2017 02:44:35 GMT

What cstella is suggesting should work, but you can also filter upstream in bro using a predicate. I can give more help later if necessary but I first suggest you read and understand the below post and look at my bro script. My script filters IPv6 traffic for Conn, HTTP, and dns, and also filters all non-internet traffic (you can simply remove that part of the logic for your situation) if you are using the Kafka plugin.

http://blog.bro.org/2012/02/filtering-logs-with-bro.html

https://github.com/JonZeolla/Development/blob/master/bro/logs-to-kafka.bro

Hope that helps.

Re: Exception in indexingBolt of indexing topology

308741831 — Tue, 28 Feb 2017 10:52:41 GMT

Thank you so much! I will feed back if it works.

Re: Exception in indexingBolt of indexing topology

308741831 — Tue, 28 Feb 2017 15:43:41 GMT

Thank you ! It works.