question Re: How to create custom hashmap accumulator in spark using java? in Support Questions

question Re: How to create custom hashmap accumulator in spark using java? in Support Questions https://community.cloudera.com/t5/Support-Questions/How-to-create-custom-hashmap-accumulator-in-spark-using-java/m-p/308607#M223633 Here's a sample problem and a custom Accumulator solution in java, you could use this as a sample to your own use case.Input:HashMap<String, String> Output:HashMap<String, Int> that will contain the count for each key in the input HashMaps, Example:Input HashMaps:1. {"key1", "Value1"}, {"key2", "Value2"}2. {"key1", "Value2"} Output:{"key1", 2}, {"key2", 1} //Since key1 is repeated 2 times. Code:<LI-CODE lang="java">import org.apache.spark.util.AccumulatorV2; import java.util.HashMap; public class CustomAccumulator extends AccumulatorV2<HashMap<String, String>, HashMap<String, Integer>> { private HashMap<String, Integer> outputHashMap; public CustomAccumulator(){ this.outputHashMap = new HashMap<>(); } @Override public boolean isZero() { return outputHashMap.size() == 0; } @Override public AccumulatorV2<HashMap<String, String>, HashMap<String, Integer>> copy() { CustomAccumulator customAccumulatorCopy = new CustomAccumulator(); customAccumulatorCopy.merge(this); return customAccumulatorCopy; } @Override public void reset() { this.outputHashMap = new HashMap<>(); } @Override public void add(HashMap<String, String> v) { v.forEach((key, value) -> { this.outputHashMap.merge(key, 1, (oldValue, newValue) -> oldValue + newValue); }); } @Override public void merge(AccumulatorV2<HashMap<String, String>, HashMap<String, Integer>> other) { other.value().forEach((key, value) -> { this.outputHashMap.merge(key, value, (oldValue, newValue) -> oldValue + newValue); }); } @Override public HashMap<String, Integer> value() { return this.outputHashMap; } }</LI-CODE> Sun, 03 Jan 2021 15:36:40 GMT code_girl 2021-01-03T15:36:40Z