The easiest way I know to get Spark working with Ipython and the Jupyter Notebook is by setting the following two environment variables as described in the book "Learning Spark":   IPYTHON=1 IPYTHON_OPTS="notebook"   Afterwards running ./bin/pyspark   NB: it's possible to pass more Jupyter options using IPYTHON_OPTS; by googling a bit you'll find them. ... View more

(1) I would start by loading the SparkR package into RStudio so you can make use of it. See the following link under heading "Using SparkR from RStudio" https://github.com/apache/spark/tree/master/R.   (2) Now you are ready to run through the following tutorial. However instead of reading the data from "hdfs" load it from your local file system. http://www.r-bloggers.com/a-first-look-at-spark/    (3) Study the SparkR Guide to gain more indepth knowledge. http://spark.apache.org/docs/latest/sparkr.html   (4) Study Spark (Dataframes, RDDs, etc) for example with the Oreilly book "Learning Spark". I find that it always helps to understand how something works under the hood. The same holds for SparkR you can easily find some videos about Youtube to understand how it works under the hood, especially the distributed character of SparkR + Spark. ... View more

If I may take a different approach on your problem I would use Spark to do the job. Load the data of each file into a separate Spark Data Frame add a new column with the desired value write everything back to HDFS preferably in a format such as Parquet and compressed with snappy. ... View more