Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Not able to run docker container on yarn even after configuring all the recommendations for running Docker containers on YARN. On GCP VM Instance

Highlighted

Not able to run docker container on yarn even after configuring all the recommendations for running Docker containers on YARN. On GCP VM Instance

New Contributor

Hi,

I followed all the recommendatiions and pre-requisite to execute docker containers on Yarn but failed to launch container. I have used single node VM Instance on Google Cloud Platform and I installed Apache Hadoop 3.1.1 and Apache Spark 2.3.0 on it.

Link used for recommendations:

https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/data-operating-system/managing-data-operati...

I was able to launch spark-shell on VM Instance(Spark 2.3.0) and by pulling docker image from Docker Hub and able to launch spark-shell on docker container(by running "docker container exec -it <containerID> bash" command) and I ran some sample examples.

When I am trying to launch spark-shell with docker on yarn it is failing to connect with docker container.

Following is the spark-shell command for launching Spark shell with docker container on YARN:

spark-2.3.0-bin-hadoop2.7/bin/spark-shell --master yarn --deploy-mode client --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=uhopper/hadoop-spark --conf spark.yarn.AppMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=uhopper/hadoop-spark --conf spark.yarn.AppMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_CONTAINER_NETWORK=host --conf spark.yarn.AppMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS=/var/hadoop/yarn/local-dir -verbose

Below is the error:

INFO retry.RetryInvocationHandler: java.net.ConnectException: Call From dockerubuntu18.us-central1-c.c.fine-entry-219708.internal/10.128.0.4 to dockerubuntu18.us-central1-c.c.fine-entry-219708.internal:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused, while invoking ApplicationClientProtocolPBClientImpl.getResourceTypeInfo over null after 1 failover attempts. Trying to failover after sleeping for 25389ms.

Please advice on the above issue.

Don't have an account?
Coming from Hortonworks? Activate your account here