Support Questions
Find answers, ask questions, and share your expertise

PySpark: Broadcast Java objects

PySpark: Broadcast Java objects

Rising Star

I'm experimenting with broadcast variables in PySpark at the moment, and I've noticed that whenever I create an explicit Java object using sc._jvm, I get errors when I try to broadcast these variables. Looking at the stack trace, the problem seems to be related to pickling. Does anybody know how I can broadcast such variables?