Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

[CDH 5.8.3] Spark log4j custom log class

Highlighted

[CDH 5.8.3] Spark log4j custom log class

Rising Star

Hi everybody,

 

I'm trying to use a custom log class for log4j application logs. I tried to include my class both in a fat-jar and in a separate jar added via the --jars option. But none of them works, I always get a ClassNotFound exception. I found this JIRA (https://issues.apache.org/jira/browse/SPARK-9826) on Spark but it is fixed in Spark 1.5 and I'm using Spark 1.6.

 

Moreover I found that I'm not the only one experiencing this issue, since I found this question on stackoverflow about the same issue: http://stackoverflow.com/questions/36790395/custom-log4j-layout-in-spark-on-yarn-classnotfoundexcept....

 

Do you have any idea?

 

Thank you,

Marco

1 REPLY 1

Re: [CDH 5.8.3] Spark log4j custom log class

Champion
I have always used --files to send my properties to each executor.
Don't have an account?
Coming from Hortonworks? Activate your account here