Skip to main content
ChineduLB
ChineduLB
Silver ⭐️⭐️⭐️
February 14, 2019
Solved

IDE for creating and running CDH jobs using Scala, Python, Jave etc.

Hi, I recently started learning Hadoop/Cloudera.

 

I have successfully installed single node Cloudera Manager on my test Linux workstation

 

I am looking for the best or suggested IDEs for creating projects to run jobs on my Hadoop fs using Scala, Java, Python etc.

 

Any how-to docs will be greately appreciated

 

Regards All

Harsh J
Harsh JAnswer
Gold ⭐️⭐️
March 7, 2019
MapReduce jobs can be submitted with ease, as all they mostly require is the correct config on the classpath (such as under src/main/resources for Maven projects).

Spark/PySpark greatly relies on its script tooling to submit to a remote cluster so it is a little more involved to achieve this. IntelliJ IDEA has a remote execution option in its run targets that can be configured to copy over the build jar and invoke any arbitrary command on an edge host. This can be combined with remote debugging perhaps to get equal experience as MR.

Another option is to use a web interface based editor such as CDSW.
View in thread

1 reply

Harsh J
Gold ⭐️⭐️
Harsh JAnswer
Gold ⭐️⭐️
March 7, 2019
MapReduce jobs can be submitted with ease, as all they mostly require is the correct config on the classpath (such as under src/main/resources for Maven projects).

Spark/PySpark greatly relies on its script tooling to submit to a remote cluster so it is a little more involved to achieve this. IntelliJ IDEA has a remote execution option in its run targets that can be configured to copy over the build jar and invoke any arbitrary command on an edge host. This can be combined with remote debugging perhaps to get equal experience as MR.

Another option is to use a web interface based editor such as CDSW.
Translate
    Terms & ConditionsAccessibility statement