Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

IDE for creating and running CDH jobs using Scala, Python, Jave etc.

SOLVED Go to solution
Highlighted

IDE for creating and running CDH jobs using Scala, Python, Jave etc.

Explorer

Hi, I recently started learning Hadoop/Cloudera.

 

I have successfully installed single node Cloudera Manager on my test Linux workstation

 

I am looking for the best or suggested IDEs for creating projects to run jobs on my Hadoop fs using Scala, Java, Python etc.

 

Any how-to docs will be greately appreciated

 

Regards All

1 ACCEPTED SOLUTION

Accepted Solutions

Re: IDE for creating and running CDH jobs using Scala, Python, Jave etc.

Master Guru
MapReduce jobs can be submitted with ease, as all they mostly require is the correct config on the classpath (such as under src/main/resources for Maven projects).

Spark/PySpark greatly relies on its script tooling to submit to a remote cluster so it is a little more involved to achieve this. IntelliJ IDEA has a remote execution option in its run targets that can be configured to copy over the build jar and invoke any arbitrary command on an edge host. This can be combined with remote debugging perhaps to get equal experience as MR.

Another option is to use a web interface based editor such as CDSW.
1 REPLY 1

Re: IDE for creating and running CDH jobs using Scala, Python, Jave etc.

Master Guru
MapReduce jobs can be submitted with ease, as all they mostly require is the correct config on the classpath (such as under src/main/resources for Maven projects).

Spark/PySpark greatly relies on its script tooling to submit to a remote cluster so it is a little more involved to achieve this. IntelliJ IDEA has a remote execution option in its run targets that can be configured to copy over the build jar and invoke any arbitrary command on an edge host. This can be combined with remote debugging perhaps to get equal experience as MR.

Another option is to use a web interface based editor such as CDSW.