What is the best IDE to do spark development. I am using scala api, so i am using Scala IDE. It takes a lot of time to build the code and making life frustrating. any thoughts would be great.
What will be the right environment for developing spark apps?
One more question is if i want to add a custom jar to a spark-shell session, how to do it?
IntelliJ IDEA is a great IDE, and there are some good resources out there showing how use it for Spark projects:
IntelliJ IDEA worked for me also, where others like NetBeans do not seem to have the support plugins, that is not to say they could not be made to work, but IntelliJ seems to be the most straightforard.
In addition, you can use one of the data science notebooking tools like Zeppelin, Jupyter, or IPython. They are a cross between developer IDE and sharing platform where you can document and comment on code. Zeppelin is built-in to Hortonworks HDP 2.4 distribution so you can give it a try by following instructions here:
You are saying that it takes a lot of time to build. How big is the resulted jar file? Are you packaging all dependencies in your jar? If that is the case, that is not a problem of the IDE you use, but a problem with your build approach.