Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

[CDH 5.8.3] Spark log4j custom log class

Highlighted

[CDH 5.8.3] Spark log4j custom log class

Rising Star

Hi everybody,

 

I'm trying to use a custom log class for log4j application logs. I tried to include my class both in a fat-jar and in a separate jar added via the --jars option. But none of them works, I always get a ClassNotFound exception. I found this JIRA (https://issues.apache.org/jira/browse/SPARK-9826) on Spark but it is fixed in Spark 1.5 and I'm using Spark 1.6.

 

Moreover I found that I'm not the only one experiencing this issue, since I found this question on stackoverflow about the same issue: http://stackoverflow.com/questions/36790395/custom-log4j-layout-in-spark-on-yarn-classnotfoundexcept....

 

Do you have any idea?

 

Thank you,

Marco

1 REPLY 1

Re: [CDH 5.8.3] Spark log4j custom log class

Champion
I have always used --files to send my properties to each executor.