https://community.cloudera.com/t5/Support-Questions/spark-join-with-udf-fails/m-p/122873#M85626 <P><A rel="user" href="https://community.cloudera.com/users/10875/xrcsblue.html" nodeid="10875">@xrcs blue</A> </P><P>Looks like you are using Spark python API. The pyspark documentation says:</P><P><STRONG>join</STRONG> : </P><UL> <LI><STRONG>on</STRONG> – a string for join column name, a list of column names, , a join expression (Column) or a list of Columns. If <CITE>on</CITE> is a string or a list of string indicating the name of the join column(s), the column(s) must exist on both sides, and this performs an equi-join.</LI></UL><P>Therefore, do the columns exist on both sides of join tables? Also, wondering if you can encode the "condition" separately, then pass it to the join() method, like this:</P><PRE>&gt;&gt;&gt; cond = [df.name == df3.name, df.age == df3.age] &gt;&gt;&gt; df.join(df3, cond, 'outer')</PRE> Tue, 12 Jul 2016 01:35:19 GMT phargis 2016-07-12T01:35:19Z