@Banana Joe It may be related to the resources in VM, please see the doc for resource requirement. http://hortonworks.com/wp-content/uploads/2016/02/Import_on_Vbox_3_1_2016.pdf RAM - "At least 8 GB of RAM (The more, the better) If you wish to enable services such as Ambari, HBase, Storm, Kafka, or Spark please ensure you have at least 10 Gb of physical RAM in order to run the VM using 8 GB". ... View more

@Benjamin Leonhardi how --files is differ from SparkContext.addFile() apart from the way we use them? ... View more

Took 65944ms to send a batch of 1 edits (205 bytes) to remote journal 192.168.1.47:8485 2016-06-02 You either have a serious network problem b/w nodes or may be underlining disk write is very slow. ... View more

@clukasik I don't see any performance issue if running it on yarn-client mode however as per initial info they needs to use distributed cache kind of thing in spark, which they can achieve through SparkContext.addFile() ... View more

@Eric Periard Technically two NN can't be at same status if its happening then either you have configuration issues or hitting some bug. ... View more

@akeezhadath Kindly use below API to cache the file on all the nodes. SparkContext.addFile() Add a file to be downloaded with this Spark job on every node. The path passed can be either a local file, a file in HDFS (or other Hadoop-supported filesystems), or an HTTP, HTTPS or FTP URI. To access the file in Spark jobs, use SparkFiles.get(fileName) to find its download location. A directory can be given if the recursive option is set to true. Currently directories are only supported for Hadoop-supported filesystems. ... View more

@akeezhadath You can place the file on HDFS and access the file through "hdfs:///path/file". ... View more