Support Questions
Find answers, ask questions, and share your expertise

spark development

Highlighted

spark development

What is the best IDE to do spark development. I am using scala api, so i am using Scala IDE. It takes a lot of time to build the code and making life frustrating. any thoughts would be great.

What will be the right environment for developing spark apps?

One more question is if i want to add a custom jar to a spark-shell session, how to do it?

8 REPLIES 8
Highlighted

Re: spark development

Expert Contributor
Highlighted

Re: spark development

+1 for IntelliJ IDEA

Highlighted

Re: spark development

New Contributor

IntelliJ IDEA worked for me also, where others like NetBeans do not seem to have the support plugins, that is not to say they could not be made to work, but IntelliJ seems to be the most straightforard.

Highlighted

Re: spark development

Expert Contributor

To add custom jars to a spark-shell session, start up the spark-shell session using this:

spark-shell --jars /path/to/jar1,/path/to/jar2
Highlighted

Re: spark development

@ARUNKUMAR RAMASAMY

In addition, you can use one of the data science notebooking tools like Zeppelin, Jupyter, or IPython. They are a cross between developer IDE and sharing platform where you can document and comment on code. Zeppelin is built-in to Hortonworks HDP 2.4 distribution so you can give it a try by following instructions here:

http://hortonworks.com/hadoop-tutorial/apache-zeppelin-hdp-2-4/

Highlighted

Re: spark development

Expert Contributor

+1 for Zeppelin, which integrates well with the HDP platform.

For custom jars, see also

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_spark-guide/content/ch_spark-add-librarie....

Highlighted

Re: spark development

@ARUN

You are saying that it takes a lot of time to build. How big is the resulted jar file? Are you packaging all dependencies in your jar? If that is the case, that is not a problem of the IDE you use, but a problem with your build approach.

Highlighted

Re: spark development

New Contributor

There are a lot of different idioms for developing spark apps. One possibility is to use Ipython and Pyspark

More info here