I am working on the filter the duplicates and creating the autogenerated Id. Here is the code
df3=df.distinct df3.createOrReplaceTempView('df3') x = spark.sql('select row_number() over (order by ZipCode, District, Division, Region) As GeographyID, District, Division, Region, RegionName, ZipCode, City, State from df3')
after filtering the duplicates I have a problem with city data with the same name, I am not able to get the distinct city values.
Here is the example for City Data
how Can I deal with this kind of string value to get the distinct values?