NiFi will do that very easily, then you can trigger some Spark jobs to do final processing. ... View more

The https://stomp.github.io/implementations.html Stomp protocol is not supported,but someone could right a processor to read and write it fairly easily as STOMP has some solid Java clients. ... View more

Any updates for 2016 ... View more

KryoSerializer is pretty awesome. It is a faster Java serializer. This will speed up Spark, not related to your issue, but I like to add that to all my Spark projects. When RDDs are in memory they are serialized objects. So a faster, smaller serialization will help with speed and memory. ... View more

Great find Ryan. Let's see if we can get that added to the tutorial. ... View more

You really need more cores. But 2 may work. spark-submit --class "com.stuff.Class" \ --master yarn --deploy-mode client --driver-memory 1024m --executor-memory 1024m --conf spark.ui.port=4244 MyJar.jar remove this from your code .set("spark.master","yarn-client"); add this sparkConf.set("spark.cores.max", "1") sparkConf.set("spark.serializer", classOf[KryoSerializer].getName) sparkConf.set("spark.sql.tungsten.enabled", "true") sparkConf.set("spark.eventLog.enabled", "true") sparkConf.set("spark.app.id", "YourId") ... View more

can you post the spark-submit you listed? ... View more

what does it saw on the main page for YARN resources? ... View more