We didn't do a custom build - our ops guys used Cloudera Manager to deploy Spark Standalone.
When running spark-submit on the master with --deploy-mode cluster, we always get this error.
If it's not a bug - i.e. it's something we did wrong - do you have any ideas what we could possibly have done to cause this problem for ourselves?
As a test we deployed Spark on YARN also, again using Cloudera Manager. On those nodes spark-submit is working as expected.
Above, are you not comparing to a build you created yourself? Yes, you're not saying you deployed that build. I'm speaking to the difference you observe in the assembly. That is, you should be getting Jackson for Hadoop/Spark from the cluster itself, and the scripts should be taking care of all this for you. If you made any changes to the local environment, those might be worth investigating.
I haven't heard of problems like this, but would only hear of this indirectly. I have rarely used standalone mode myself, and not sure I ever used cluster mode. I honestly would have to check whether we support this combo, but think we would in general. Still, Spark on YARN is certainly recommended. Are you able to use that?