This statement for bigram generation always gives the right result, no matter how much paralleliziation provided. Why does it always give the correct result? Due to sliding itself? Parallel processing for adding +1 to a number is easy to understand, but this here has to do with data over partitions. Reading a big file or mapPartitions approach will result in minor loss of accuracy (verified practically although obvious), but why not here? It must be simple, but I cannot see it. val rdd = sc.parallelize(Array("A", "B", "C", "D", "E", "F"),5)
rdd.sliding(2).collect()
... View more
