Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Exception in indexingBolt of indexing topology

Solved Go to solution

Exception in indexingBolt of indexing topology

New Contributor

After runing Metron a little while,I received this exception:

index [bro_index_2017.02.23.16], type [bro_doc], id [AVpp_hu_luwdJ-LP4qUA], message [MapperParsingException[failed to parse [ip_dst_addr]]; nested: IllegalArgumentException[failed to parse ip [ff02::0001:0003], not a valid ipv4 address (4 dots)];]

How do i resolve it ? I'll appreciate it for any help!

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Exception in indexingBolt of indexing topology

New Contributor

Well, the problem is actually in the elasticsearch indexing templates. Normally, I'd say that you could use a message filter to filter out the IPv6 data in the parser, but I know that they don't work in HCS 1.0. As a workaround, you could transform the IPv6 addresses to 0.0.0.0 and they'll index. You can also save off the old address in a new field. This would be how you would do it with Stellar field transformations.

Edit $METRON_HOME/config/zookeeper/parsers/bro.json to add the "fieldTransformations" section, like so:

{
  "parserClassName":"org.apache.metron.parsers.bro.BasicBroParser",
  "sensorTopic":"bro",
  "parserConfig": {},
  "fieldTransformations" : [
    {
      "transformation" : "STELLAR"
    ,"output" : [ "raw_dst_ip"
                , "ip_dst_addr"
                ]
    ,"config" : {
 "raw_dst_ip" : "ip_dst_addr"
,"ip_dst_addr" : "if IS_IP(ip_dst_addr, 'IPV4') then ip_dst_addr else '0.0.0.0'"
                }
    }
                           ]
}

If things work out like they should, you'll have a raw_dst_ip field and ip_dst_addr will either be IPv4 or '0.0.0.0', which will index just fine.

In the next release, you'll have a message filter that works so you could drop them easier.

Hope this helps! Report back if you get into trouble.

11 REPLIES 11

Re: Exception in indexingBolt of indexing topology

New Contributor

Well, the problem is actually in the elasticsearch indexing templates. Normally, I'd say that you could use a message filter to filter out the IPv6 data in the parser, but I know that they don't work in HCS 1.0. As a workaround, you could transform the IPv6 addresses to 0.0.0.0 and they'll index. You can also save off the old address in a new field. This would be how you would do it with Stellar field transformations.

Edit $METRON_HOME/config/zookeeper/parsers/bro.json to add the "fieldTransformations" section, like so:

{
  "parserClassName":"org.apache.metron.parsers.bro.BasicBroParser",
  "sensorTopic":"bro",
  "parserConfig": {},
  "fieldTransformations" : [
    {
      "transformation" : "STELLAR"
    ,"output" : [ "raw_dst_ip"
                , "ip_dst_addr"
                ]
    ,"config" : {
 "raw_dst_ip" : "ip_dst_addr"
,"ip_dst_addr" : "if IS_IP(ip_dst_addr, 'IPV4') then ip_dst_addr else '0.0.0.0'"
                }
    }
                           ]
}

If things work out like they should, you'll have a raw_dst_ip field and ip_dst_addr will either be IPv4 or '0.0.0.0', which will index just fine.

In the next release, you'll have a message filter that works so you could drop them easier.

Hope this helps! Report back if you get into trouble.

Re: Exception in indexingBolt of indexing topology

New Contributor

@cstella is there any approches to do this?

Re: Exception in indexingBolt of indexing topology

New Contributor

We do not currently support IPv6 addresses in Metron. You have unfortunately hit https://issues.apache.org/jira/browse/METRON-293

Re: Exception in indexingBolt of indexing topology

New Contributor

so i wanna disable ipv6 in bro ,do you know how to do that?

Re: Exception in indexingBolt of indexing topology

New Contributor

i just do not want ipv6 show up in bro logs

Re: Exception in indexingBolt of indexing topology

New Contributor

I should point out that you will need to, after you make that change, push the configs to zookeeper via $METRON_HOME/bin/zk_load_configs.sh --mode PUSH -i $METRON_HOME/config/zookeeper -z $ZK_QUORUM

where ZK_QUORUM is something like hostname:2181

Re: Exception in indexingBolt of indexing topology

New Contributor

actually ,i hope bro logs capture ipv4 info only ,is there any configuration to set

Re: Exception in indexingBolt of indexing topology

New Contributor

or why dont you translate ipv6 to ipv4 with stellar script?

Highlighted

Re: Exception in indexingBolt of indexing topology

New Contributor

What cstella is suggesting should work, but you can also filter upstream in bro using a predicate. I can give more help later if necessary but I first suggest you read and understand the below post and look at my bro script. My script filters IPv6 traffic for Conn, HTTP, and dns, and also filters all non-internet traffic (you can simply remove that part of the logic for your situation) if you are using the Kafka plugin.

http://blog.bro.org/2012/02/filtering-logs-with-bro.html

https://github.com/JonZeolla/Development/blob/master/bro/logs-to-kafka.bro

Hope that helps.