I found the correct answer like make dataframe in pyspark left join with itself and filter nulls and not nulls, put everything into a loop until df.take(1)=[] null. --------------------------------------------------------------------------- df = df_tr.select(col("old_id"),col("new__id")).distinct() df2 = df df_tr = spark.createDataFrame([('null', 'null')], while (df.take(1) != []): df = df.alias("df1").join(df2.alias("df2"), col('df1.new__id') == col('df2.old_id'), 'left_outer') df_null = df.filter(col('df2.new_id').isNull()).select(col('df1.old_id'), col('df1.new_id')) df = df.filter(col('df2.new_id').isNotNull()).select(col('df1.old_id'), col('df2.new_id')) df_tr = df_null.union(df_tracking) ... View more

Thank you Emily for your reply. ... View more

@mqureshi I still failed to export blanks as blanks. ... View more

data: a||c |b|c a|b| a|\N\c a|N|c a|\\N|c a|NULL|c Table: CREATE SET TABLE panda.test_tera2 ,NO FALLBACK , NO BEFORE JOURNAL, NO AFTER JOURNAL, CHECKSUM = DEFAULT, DEFAULT MERGEBLOCKRATIO ( col VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC, col2 CHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC, col3 VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC) PRIMARY INDEX ( col ); --------------------------- Command: sqoop export --connect jdbc:teradata://xxx/database=xx --connection-manager org.apache.sqoop.teradata.TeradataConnManager --username --password xx --export-dir /test/xxtemp/teradata2 --table test_tera2 --input-fields-terminated-by '|' -m 2 ------------------------- Result: col col2 col3 1 a ? c 2 ? b c 3 a \N\c ? 4 a N c 5 a b ? 6 a NULL c 7 a \\N c ------------------------------------------------ Command: sqoop export --connect jdbc:teradata://xx/database=xx --connection-manager org.apache.sqoop.teradata.TeradataConnManager --username xx --password xx --export-dir /test/xx/teradata2 --table test_tera2 --input-fields-terminated-by '|' -m 2 --input-null-non-string "\\N" RESULT: col col2 col3 1 a ? c 2 ? b c 3 a \N\c ? 4 a N c 5 a b ? 6 a NULL c 7 a \\N c ------------------------------------------------ Command: sqoop export --connect jdbc:teradata://xx/database=xx --connection-manager org.apache.sqoop.teradata.TeradataConnManager --username xx --password xx --export-dir /test/xx/teradata2 --table test_tera2 --input-fields-terminated-by '|' -m 2 --input-null-non-string "\\N" --input-null-string "\\N" RESULT: col col2 col3 1 a ? c 2 ? b c 3 a \N\c ? 4 a N c 5 a b ? 6 a NULL c 7 a \\N c -------------------------------------------------- I am still failed to export blanks || field between pipes as blanks in teradata with sqoop export. Could someone explain me the reason and way to solve it? ... View more

Thank you, I already tried it. But It converts blanks into nulls. I want blanks as blanks for pig output stored in hdfs like a,,b,, ... View more