Meaning, multi-layer neural nets? No, no plans for that. The idea is to support common business use cases, and there is not a lot of broad use for this technique as a classifier over, say, RDF. ... View more

(By the way, there is a separate forum for Spark: http://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/bd-p/Spark )   If the error is definitely in the shell / REPL, then I believe you just want to set SPARK_REPL_OPTS:   SPARK_REPL_OPTS="-XX:MaxPermSize=256m" spark-shell   I find this other setting can help with permgen usage in Scala, which is an issue when running tests too:   SPARK_REPL_OPTS="-XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256m" spark-shell     ... View more

I would bet that this means that the amount of memory you have requested for your executors exceeds the amount of memory that any single worker of yours has. What are these sizes? ... View more

No, sbt and scala are not installed. SBT is a build tool and used at development time, rather than at runtime -- it's a lot like Maven in this respect. You would use SBT (or Maven) on your development machine to create a .jar file, and run that on your cluster. ... View more

Are you on CDH5 beta 2? It already includes Spark. I wonder if its setup of Spark is interfering with whatever you have installed separately, or vice versa. Can you simply use the built-in deployment? It would be easier. ... View more

At the moment, CDH5b2 deploys Spark in "standalone" mode: https://spark.apache.org/docs/0.9.0/spark-standalone.html   This simply means Spark tries to manage resources itself, rather than participating in a cluster manager like YARN or Mesos. As an end user, it shouldn't make much difference to you at all. Just fire up the shell and go.   Once a few kinks are worked out, Spark's YARN integration will be used in the future, as I understand. ... View more

No they do not. The serving layer must be able to access HDFS. It takes its config by default from Hadoop config on the local machine at /etc/hadoop/conf. If that is hard I can say more about how to directly specify the HDFS URL. For this reason you can run these on many different machines. ... View more