Member since
07-29-2015
5
Posts
0
Kudos Received
0
Solutions
09-01-2015
07:43 AM
Thank you for the quick response, I really appreciate helping me clear my questions. The answer was exactly what I was looking for. It is automated and users cannot control the elements of classpath.txt file. Pardon my naive question, but can it pose a problem having different versions of same dependencies on classpath? Example: 09:39:34 $ cat /etc/spark/conf/classpath.txt | grep jersey-server /opt/cloudera/parcels/CDH-5.4.4-1.cdh5.4.4.p886.563/jars/jersey-server-1.9.jar /opt/cloudera/parcels/CDH-5.4.4-1.cdh5.4.4.p886.563/jars/jersey-server-1.14.jar
... View more
09-01-2015
06:44 AM
Thank you for your response Wilfred. It sure helps me. However, my question was more towards understanding how classpath.txt file mentioned below is created? Does CM create this file on all nodes, is it something we can configure through CM? 08:42:43 $ ll /etc/spark/conf/ total 60 drwxr-xr-x 3 root root 4096 Aug 25 12:28 ./ drwxr-xr-x 3 root root 4096 Aug 25 12:28 ../ -rw-r--r-- 1 root root 29228 Aug 25 12:28 classpath.txt -rw-r--r-- 1 root root 21 Aug 25 12:28 __cloudera_generation__ -rw-r--r-- 1 root root 550 Aug 25 12:28 log4j.properties -rw-r--r-- 1 root root 800 Aug 25 12:28 spark-defaults.conf -rw-r--r-- 1 root root 1122 Aug 25 12:28 spark-env.sh drwxr-xr-x 2 root root 4096 Aug 25 12:28 yarn-conf/
... View more
08-27-2015
11:31 AM
We have Spark installed via Cloudera Manager on a YARN cluster. It appears there is a classpath.txt file in /etc/spark/conf that include list of jars that should be available on spark's distributed classpath. And spark-env.sh seems to be the on that's exporting this configuration. It is my understanding that cloudera manager creates the classpath.txt file. I would like how does cloudera manger evaluate the list of jars that go into this file, and is it something that can be controlled through cloudera manager. Thank you!
... View more
Labels:
- Labels:
-
Apache Spark
-
Apache YARN
-
Cloudera Manager
08-17-2015
11:14 AM
I'm having a similar issue but with a different stack trace. Were you able to resolve this problem?
... View more