Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

IDE for creating and running CDH jobs using Scala, Python, Jave etc.

avatar
Contributor

Hi, I recently started learning Hadoop/Cloudera.

 

I have successfully installed single node Cloudera Manager on my test Linux workstation

 

I am looking for the best or suggested IDEs for creating projects to run jobs on my Hadoop fs using Scala, Java, Python etc.

 

Any how-to docs will be greately appreciated

 

Regards All

1 ACCEPTED SOLUTION

avatar
Mentor
MapReduce jobs can be submitted with ease, as all they mostly require is the correct config on the classpath (such as under src/main/resources for Maven projects).

Spark/PySpark greatly relies on its script tooling to submit to a remote cluster so it is a little more involved to achieve this. IntelliJ IDEA has a remote execution option in its run targets that can be configured to copy over the build jar and invoke any arbitrary command on an edge host. This can be combined with remote debugging perhaps to get equal experience as MR.

Another option is to use a web interface based editor such as CDSW.

View solution in original post

1 REPLY 1

avatar
Mentor
MapReduce jobs can be submitted with ease, as all they mostly require is the correct config on the classpath (such as under src/main/resources for Maven projects).

Spark/PySpark greatly relies on its script tooling to submit to a remote cluster so it is a little more involved to achieve this. IntelliJ IDEA has a remote execution option in its run targets that can be configured to copy over the build jar and invoke any arbitrary command on an edge host. This can be combined with remote debugging perhaps to get equal experience as MR.

Another option is to use a web interface based editor such as CDSW.