When running this small piece of Scala code I get a "org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://xxx.eu-west-1.compute.internal:8020/user/cloudbreak/data/testdfsio-write". Below the piece of code where the `saveAsTextFile` is executed. The directory does not exist before running this script. Why is this FileAlreadyExistsException being raised? // Create a Range and parallelize it, on nFiles partitions
// The idea is to have a small RDD partitioned on a given number of workers
// then each worker will generate data to write
val a = sc.parallelize(1 until config.nFiles + 1, config.nFiles)
val b = a.map(i => {
// generate an array of Byte (8 bit), with dimension fSize
// fill it up with "0" chars, and make it a string for it to be saved as text
// TODO: this approach can still cause memory problems in the executor if the array is too big.
val x = Array.ofDim[Byte](fSizeBV.value).map(x => "0").mkString("")
x
})
// Force computation on the RDD
sc.runJob(b, (iter: Iterator[_]) => {})
// Write output file
val (junk, timeW) = profile {
b.saveAsTextFile(config.file)
}
... View more
