Support Questions
Find answers, ask questions, and share your expertise

Conditional Split on String Variable condition


I have a requirement to filter out specific string values like AB01, BC01,CD05 stored in a variable which can change later and I don't want to hard code this values in my conditional split comparison. Also I have a limitation not to filter these values in the source so I want to control it in Conditional split.

My condition is COMP_Name != (AB01, BC01,CD05)

Any help is highly appreciated.

Thank you

Saanvi S


Re: Conditional Split on String Variable condition

Expert Contributor

@Saanvi Sharma, you're doing this in Spark?

You can declare the variable as a list, for exampe - comp_list=['AB01','BC01']

Then you can filter it with rdd.filter(lambda x:not(x in comp_list)), if you're filtering on this string directly,

Otherwise you can put your field as an argument to lambda expression.

Similar concept may be applied if you're doing this any other way.