Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Deploy Spark app to Yarn

Highlighted

Deploy Spark app to Yarn

Contributor

I'm trying to make applications in Spark in IntelliJ and push the same into YARN. By now I was reading about how to push jars through spark-submit but I really want to avoid working in terminal.

Can you suggest me what to set, how to set or even better, where to work with Spark aplications and be able to debug them like I can in IntelliJ

I'm using Ambari, and I setted there YARN and Spark cluster. Lets say that this is IP of mine ResourceManager is bigdata5.local:8050.

How to set setMaster method, spark.setMaster("spark://bigdata5.local:8050") maybe? Because like that I'm getting error about

INFO Utils: Successfully started service 'SparkUI' on port 4040. 17/06/07 14:06:50 <br>

INFO SparkUI: Started SparkUI at http://localhost:4040 17/06/07 14:06:50<br>

INFO AppClient$ClientEndpoint: Connecting to master spark://bigdata5.local:8050... 17/06/07 14:06:50<br>

ERROR TransportResponseHandler: Still have 1 requests outstanding when connection from bigdata5.local:8050 is closed

Please lead me with best practices :)

UPDATE 1:

Also after a while, I'm getting as add-on this error:

java.io.IOException: An established connection was aborted by the software in your host machine