The best resource is probably the web site at http://oryx.io as well as the source code. http://github.com/OryxProject/oryx ... View more

is it possible with Spark to handle big data cleansing ? ... View more

I got the solution. In my Spark Streaming application I had set SparkConf.setMaster("local[*]") and in spark-submit I was providing --master yarn-cluster. So there was conflict in both the masters and it was remaining in ACCEPTED state and exiting. ... View more

It's possible to just use a static Executor in your code and use it to run multi-threaded operations within each function call. This may not be efficient though. If your goal is simply full utilization of cores, then make sure you have enough executors with enough cores running to use all of your cluster. Then make sure your number of partitions is at least this large. Then each operation can be single-threaded. ... View more

With the --files option you put the file in your working directory on the executor. You are trying to point to the file using an absolute path which is not what files option does for you. Can you use just the name "rule2.xml" and not a path. When you read the documentation for the files. See the important note at the bottom of the page running on yarn.   Also do not use the Resources.getResource() but just use a open of a java construct like: new FileInputStream("rule2.xml") or something like it.   Wilfred ... View more

Done! I reinstall a new version of Hive: hive 1.2.1. And the job is run well! ... View more

In your Spark UI do you see it working with a large number of partitions (large number of tasks)?  It could be that you are loading all 70G into memory at once if you have a small number of partitions.   Also it could be that you have one huge partition with 99% of the data and lots of small ones.  Then when Spark processes your huge partition it will load it all into memory.  This can happen if you are mapping to a tuple e.g. (x, y) and the key (x) is the same for 99% of the data.   Have a look at your Spark UI to see the size of the tasks you are running.  It's likely that you will see a small number of tasks, or one huge task and a lot of small ones. ... View more