I'm trying to run pcap_query for only 1.5 hours I constantly getting OOME when it's about to write to a pcap file as the result. I tried to modify the memory setting for mapreduce but nothing does the trick so far.
I also tried to reduce a smaller window of exporting but I get no results.
I attached the stack trace below. Any suggestion is appreciated.
I found a work around for this issue by breaking it down to 30min per run, into 3 runs and it works fine. I was also trying to increase the number of reducer. It looks like the default number of mapper to be 15 and reducer to be 10. I tried to overwrite the mapreduce.job.reduces to a higher number but it doesn't take it. I hope it's not hardcoded.