Reply
New Contributor
Posts: 2
Registered: ‎01-09-2017

[CDH 5.8.3] Spark log4j custom log class

Hi everybody,

 

I'm trying to use a custom log class for log4j application logs. I tried to include my class both in a fat-jar and in a separate jar added via the --jars option. But none of them works, I always get a ClassNotFound exception. I found this JIRA (https://issues.apache.org/jira/browse/SPARK-9826) on Spark but it is fixed in Spark 1.5 and I'm using Spark 1.6.

 

Moreover I found that I'm not the only one experiencing this issue, since I found this question on stackoverflow about the same issue: http://stackoverflow.com/questions/36790395/custom-log4j-layout-in-spark-on-yarn-classnotfoundexcept....

 

Do you have any idea?

 

Thank you,

Marco

Highlighted
Posts: 642
Topics: 3
Kudos: 121
Solutions: 67
Registered: ‎08-16-2016

Re: [CDH 5.8.3] Spark log4j custom log class

I have always used --files to send my properties to each executor.