I'm trying to use a custom log class for log4j application logs. I tried to include my class both in a fat-jar and in a separate jar added via the --jars option. But none of them works, I always get a ClassNotFound exception. I found this JIRA (https://issues.apache.org/jira/browse/SPARK-9826) on Spark but it is fixed in Spark 1.5 and I'm using Spark 1.6.
Moreover I found that I'm not the only one experiencing this issue, since I found this question on stackoverflow about the same issue: http://stackoverflow.com/questions/36790395/custom-log4j-layout-in-spark-on-yarn-classnotfoundexcept....
Do you have any idea?