I have a requirement to filter out specific string values like AB01, BC01,CD05 stored in a variable which can change later and I don't want to hard code this values in my conditional split comparison. Also I have a limitation not to filter these values in the source so I want to control it in Conditional split.
My condition is COMP_Name != (AB01, BC01,CD05)
Any help is highly appreciated.
@Saanvi Sharma, you're doing this in Spark?
You can declare the variable as a list, for exampe - comp_list=['AB01','BC01']
Then you can filter it with rdd.filter(lambda x:not(x in comp_list)), if you're filtering on this string directly,
Otherwise you can put your field as an argument to lambda expression.
Similar concept may be applied if you're doing this any other way.