@Robert Levas Would you recommend setting up the MIT KDC on it's own dedicated VM or on one of the masters? What kind of resources does it require? ... View more

@cduby All of the examples I see are for doing a spark-submit. I'm trying to run the code within the IntelliJ IDE by right clicking on the class and selecting run. I was doing this by setting the master in the code to local and it worked fine up until the point where I wanted to use hive and then I started getting errors, so I tried to set the master to my remote cluster instead. Is there a better way to do that, so you can actually run your code before you package it and submit it and check the logs to see if it worked? ... View more

I have been able to fix the issue for the invalid calss exception on the spark.rdd class, but now I'm getting errors for other classes and anonymous functions in my class. I created a new project with maven dependency instead of sbt and followed the instructions for pointing to the hortonworks repo as @cduby suggested. I used the full spark version in my depencies (1.6.2.2.5.3.0-37). I then removed the Spark I had downloaded locally and instead downloaded the 1.6.2 source code and built it with scala 2.10.5. Now when I run my kafka streaming spark program from the IDE using sparkConf().setMaster("spark://myRemotecluster.domain.com:7077"), I get a class not found exception for org.apache.spark.streaming.kafka.KafkaReceiver instead. I tried running the SparkPi example class from the spark examples the same way as I ran my streaming project, so I could test to see if the core spark libraries would run and I got a class not found exception for com.mycompany.scala.SparkPi$$anonfun$1, so I'm thinking this is a problem with my classes needing to be available on the remote cluster. Is there something I'm missing, or am I just going about this the wrong way? If I could run in local mode and still interact with Hive, HBase, and Hadoop I wouldn't care so much about running my program from the IDE before submitting it to the server. ... View more

@Timothy Spann I'm using java 1.8 JDK, compiling on Windows. I'm pretty sure I'm building and pushing to 1.6.2. I have a couple of versions of Spark on my Windows machine, but my SPARK_HOME is set to the 1.6.2 version. I tried submitting it from the command line on my local machine with the --master option set to my remote cluster and received a class not found error, but this time it was for KafkaUtils and not the org.apache.spark.rdd.RDD class that I get when I try to run it feom IntelliJ ... View more

Thank you @Timothy Spann. I've had problems testing in Zeppelin before when I need to import in kafka streaming, but I'll give it another go. I've been able to test using the local master for the most part, but when I'm trying to test out some interaction with Hive and possibly HBase and that's why I have been trying to run the code against my cluster where I have the Hive and HBase services running. ... View more

@cduby I created a maven project instead and didn't get the dependency resolution errors, but I'm getting class not found errors when I try to run the program. Is the only way to run your code on a cluster to do a spark-submit? I was really wanting to just debug my code as I add to it to make sure it behaves as expected before and then move it to the cluster and submit it as a job ... View more

To clarify this. I'm wanting to run the code from the IDE, not through spark-submit. I want to test my code and debug before I do a spark-submit ... View more

@cduby I haven't tried to do a spark-submit command line yet. I'm just trying to get the program to compile without error right now. ... View more

@cduby Tried it, but it doesn't change anything ... View more

@cduby here's a screenshot buildsbt.png ... View more