Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Exception in indexingBolt of indexing topology

avatar
Contributor

After runing Metron a little while,I received this exception:

index [bro_index_2017.02.23.16], type [bro_doc], id [AVpp_hu_luwdJ-LP4qUA], message [MapperParsingException[failed to parse [ip_dst_addr]]; nested: IllegalArgumentException[failed to parse ip [ff02::0001:0003], not a valid ipv4 address (4 dots)];]

How do i resolve it ? I'll appreciate it for any help!

1 ACCEPTED SOLUTION

avatar
Contributor

Well, the problem is actually in the elasticsearch indexing templates. Normally, I'd say that you could use a message filter to filter out the IPv6 data in the parser, but I know that they don't work in HCS 1.0. As a workaround, you could transform the IPv6 addresses to 0.0.0.0 and they'll index. You can also save off the old address in a new field. This would be how you would do it with Stellar field transformations.

Edit $METRON_HOME/config/zookeeper/parsers/bro.json to add the "fieldTransformations" section, like so:

{
  "parserClassName":"org.apache.metron.parsers.bro.BasicBroParser",
  "sensorTopic":"bro",
  "parserConfig": {},
  "fieldTransformations" : [
    {
      "transformation" : "STELLAR"
    ,"output" : [ "raw_dst_ip"
                , "ip_dst_addr"
                ]
    ,"config" : {
 "raw_dst_ip" : "ip_dst_addr"
,"ip_dst_addr" : "if IS_IP(ip_dst_addr, 'IPV4') then ip_dst_addr else '0.0.0.0'"
                }
    }
                           ]
}

If things work out like they should, you'll have a raw_dst_ip field and ip_dst_addr will either be IPv4 or '0.0.0.0', which will index just fine.

In the next release, you'll have a message filter that works so you could drop them easier.

Hope this helps! Report back if you get into trouble.

View solution in original post

11 REPLIES 11

avatar
Contributor

Well, the problem is actually in the elasticsearch indexing templates. Normally, I'd say that you could use a message filter to filter out the IPv6 data in the parser, but I know that they don't work in HCS 1.0. As a workaround, you could transform the IPv6 addresses to 0.0.0.0 and they'll index. You can also save off the old address in a new field. This would be how you would do it with Stellar field transformations.

Edit $METRON_HOME/config/zookeeper/parsers/bro.json to add the "fieldTransformations" section, like so:

{
  "parserClassName":"org.apache.metron.parsers.bro.BasicBroParser",
  "sensorTopic":"bro",
  "parserConfig": {},
  "fieldTransformations" : [
    {
      "transformation" : "STELLAR"
    ,"output" : [ "raw_dst_ip"
                , "ip_dst_addr"
                ]
    ,"config" : {
 "raw_dst_ip" : "ip_dst_addr"
,"ip_dst_addr" : "if IS_IP(ip_dst_addr, 'IPV4') then ip_dst_addr else '0.0.0.0'"
                }
    }
                           ]
}

If things work out like they should, you'll have a raw_dst_ip field and ip_dst_addr will either be IPv4 or '0.0.0.0', which will index just fine.

In the next release, you'll have a message filter that works so you could drop them easier.

Hope this helps! Report back if you get into trouble.

avatar
Contributor

@cstella is there any approches to do this?

avatar
Contributor

We do not currently support IPv6 addresses in Metron. You have unfortunately hit https://issues.apache.org/jira/browse/METRON-293

avatar
Contributor

so i wanna disable ipv6 in bro ,do you know how to do that?

avatar
Contributor

i just do not want ipv6 show up in bro logs

avatar
Contributor

I should point out that you will need to, after you make that change, push the configs to zookeeper via $METRON_HOME/bin/zk_load_configs.sh --mode PUSH -i $METRON_HOME/config/zookeeper -z $ZK_QUORUM

where ZK_QUORUM is something like hostname:2181

avatar
Contributor

actually ,i hope bro logs capture ipv4 info only ,is there any configuration to set

avatar
Contributor

or why dont you translate ipv6 to ipv4 with stellar script?

avatar
Explorer

What cstella is suggesting should work, but you can also filter upstream in bro using a predicate. I can give more help later if necessary but I first suggest you read and understand the below post and look at my bro script. My script filters IPv6 traffic for Conn, HTTP, and dns, and also filters all non-internet traffic (you can simply remove that part of the logic for your situation) if you are using the Kafka plugin.

http://blog.bro.org/2012/02/filtering-logs-with-bro.html

https://github.com/JonZeolla/Development/blob/master/bro/logs-to-kafka.bro

Hope that helps.