Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

[Newbie] Is it possible to write/run Spark code in JAVA on HDP2.4 (Centos 6.7)

avatar
New Contributor

I am a newbie to ambari and HDP, currently playing on HDP2.4, I have gone through a couple of sample exercises mentioned in hortonworks pages, given my interest/focus is on Spark, I understand that I can write spark code in Zepplin-notebook, the exercises that I went through were already in Scala.

Given that I hail from java background,

1) Is it possible to write/run spark code in java on zepplin or any other workaround?

2) Could anyone link me to good starter-intermediate spark examples/questions (not necessarily Java)?

Many thanks in advance,

Naveen

1 ACCEPTED SOLUTION

avatar

Zeppelin comes with a long list of interpreters (including Spark/Scala, Python/Pyspark, Hive, Cassandra, SparkSQL, Phoenix, Markdown and Shell), which basically provide language binding to run code that you type into a notebook cell. Currently, the list of interpreters does not include Java, so you will need to compile your code first and build a jar file, which can be submitted to Spark via spark-submit, as described here:

http://spark.apache.org/docs/latest/submitting-applications.html

View solution in original post

4 REPLIES 4

avatar
Master Guru

For Spark on HDP, please check HDP Spark Guide. For general Spark programming check Spark Programming Guide with all examples given in Scala, Java, and Python. If you want to program in Java you need a development environment like Eclipse or IntelliJ.

avatar
New Contributor

Thanks for the reply, gives me a good start, on that node, is it possible to add GUI to the HDP VM and install intelliJ there?

avatar

Zeppelin comes with a long list of interpreters (including Spark/Scala, Python/Pyspark, Hive, Cassandra, SparkSQL, Phoenix, Markdown and Shell), which basically provide language binding to run code that you type into a notebook cell. Currently, the list of interpreters does not include Java, so you will need to compile your code first and build a jar file, which can be submitted to Spark via spark-submit, as described here:

http://spark.apache.org/docs/latest/submitting-applications.html

avatar
New Contributor

good to know, thanks for the sharing that information.