Did you manage to get any resolution for this? I am able to run spark job as a --proxy-user under yarn cluster mode. However I can successfully run using yarn-client mode.
This is when using CDH 6.2.1 version of Spark.
There is no problem when using opensource version of Spark with --proxy-user either on client or cluster mode.