Hi @Reza77, as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post. ... View more

I found the correct answer like make dataframe in pyspark left join with itself and filter nulls and not nulls, put everything into a loop until df.take(1)=[] null. --------------------------------------------------------------------------- df = df_tr.select(col("old_id"),col("new__id")).distinct() df2 = df df_tr = spark.createDataFrame([('null', 'null')], while (df.take(1) != []): df = df.alias("df1").join(df2.alias("df2"), col('df1.new__id') == col('df2.old_id'), 'left_outer') df_null = df.filter(col('df2.new_id').isNull()).select(col('df1.old_id'), col('df1.new_id')) df = df.filter(col('df2.new_id').isNotNull()).select(col('df1.old_id'), col('df2.new_id')) df_tr = df_null.union(df_tracking) ... View more

Thank you Emily for your reply. ... View more