Archives of Support Questions (Read Only)

This is an archived board for historical reference. Information and links may no longer be available or relevant
Announcements
This board is archived and read-only for historical reference. To ask a new question, please post a new topic on the appropriate active board.

Who agreed with this topic

Spark-shell - Cannot run program "/etc/hadoop/conf.cloudera.yarn/topology.py"

avatar
Reader

Hi

 

I deployed CDH 5.4.8 on a 6 machine cluster (4 datanode, 1 namenode, 1 edgenode) with cloudera manager and parcels.

 

When launching `spark-shell` I get the following exception :

 

java.io.IOException: Cannot run program "/etc/hadoop/conf.cloudera.yarn/topology.py" (in directory "/home/sherloqq"): error=2, No such file or directory 

 

When searching on this error I found the following two informations :

 

- It happens with Impala & Llama, deploying Yarn Gateways on the client nodes should fix it (here, third issue)

- It's happened before and copying the `/etc/hadoop/conf.cloudera.yarn` fixes it but it's supposed to be fixed (here)

 

Adding a Yarn Gateway on the node with the Spark gateway didn't do anythig

Copying the directory to the node did fix it, however it seems to be a temporary fix

 

Is there a better fix to this issue ?

 

Thanks

 

C.

Who agreed with this topic