Spark-shell - Cannot run program "/etc/hadoop/conf.cloudera.yarn/topology.py"

cberez — Fri, 16 Sep 2022 09:49:13 GMT

I deployed CDH 5.4.8 on a 6 machine cluster (4 datanode, 1 namenode, 1 edgenode) with cloudera manager and parcels.

When launching `spark-shell` I get the following exception :

java.io.IOException: Cannot run program "/etc/hadoop/conf.cloudera.yarn/topology.py" (in directory "/home/sherloqq"): error=2, No such file or directory

When searching on this error I found the following two informations :

- It happens with Impala & Llama, deploying Yarn Gateways on the client nodes should fix it (here, third issue)

- It's happened before and copying the `/etc/hadoop/conf.cloudera.yarn` fixes it but it's supposed to be fixed (here)

Adding a Yarn Gateway on the node with the Spark gateway didn't do anythig

Copying the directory to the node did fix it, however it seems to be a temporary fix

Is there a better fix to this issue ?

Thanks

Re: Spark-shell - Cannot run program "/etc/hadoop/conf.cloudera.yarn/topology.py"

yuzjang — Thu, 23 Mar 2017 07:50:55 GMT

you can deploy a yarn gateway on the machine which run commond "spark-shell" .

Re: Spark-shell - Cannot run program "/etc/hadoop/conf.cloudera.yarn/topology.py"

amankumar42 — Wed, 29 Nov 2017 10:02:58 GMT

Can you please mention the steps?

question Re: Spark-shell - Cannot run program "/etc/hadoop/conf.cloudera.yarn/topology.py" in Archives of Support Questions (Read Only)

Spark-shell - Cannot run program "/etc/hadoop/conf.cloudera.yarn/topology.py"

Re: Spark-shell - Cannot run program "/etc/hadoop/conf.cloudera.yarn/topology.py"

Re: Spark-shell - Cannot run program "/etc/hadoop/conf.cloudera.yarn/topology.py"