What exception/error happens in code 2? Just curious. foreachRDD is the prescriptive method to write to external systems. So you should be using foreachRDD. The outer loop executes on the driver and inner loop on the executors. Executors run on remote machines in a cluster. However in the code above its not clear how dynamoConnection is available to executors since such network connections are usually not serializable. Or is the following line inadvertently missing from snippet 1. val dynamoConnection = setupDynamoClientConnection() If yes, then the slowness could stem from repeatedly creating a dynamoClientConnection for each record. The recommended pattern is to use foreachPartition() to create the connection once per partition and then rdd.foreach() to write the records using that connection. For more info please search for foreachPartition see http://spark.apache.org/docs/latest/streaming-programming-guide.html
... View more
