IDE for creating and running CDH jobs using Scala, Python, Jave etc.

[#1]

Hi, I recently started learning Hadoop/Cloudera.

I have successfully installed single node Cloudera Manager on my test Linux workstation

I am looking for the best or suggested IDEs for creating projects to run jobs on my Hadoop fs using Scala, Java, Python etc.

Any how-to docs will be greately appreciated

Regards All

[#2 · Accepted solution]

MapReduce jobs can be submitted with ease, as all they mostly require is the correct config on the classpath (such as under src/main/resources for Maven projects).

Spark/PySpark greatly relies on its script tooling to submit to a remote cluster so it is a little more involved to achieve this. IntelliJ IDEA has a remote execution option in its run targets that can be configured to copy over the build jar and invoke any arbitrary command on an edge host. This can be combined with remote debugging perhaps to get equal experience as MR.

Another option is to use a web interface based editor such as CDSW.