Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Conditional Split on String Variable condition

Conditional Split on String Variable condition


I have a requirement to filter out specific string values like AB01, BC01,CD05 stored in a variable which can change later and I don't want to hard code this values in my conditional split comparison. Also I have a limitation not to filter these values in the source so I want to control it in Conditional split.

My condition is COMP_Name != (AB01, BC01,CD05)

Any help is highly appreciated.

Thank you

Saanvi S


Re: Conditional Split on String Variable condition

Expert Contributor

@Saanvi Sharma, you're doing this in Spark?

You can declare the variable as a list, for exampe - comp_list=['AB01','BC01']

Then you can filter it with rdd.filter(lambda x:not(x in comp_list)), if you're filtering on this string directly,

Otherwise you can put your field as an argument to lambda expression.

Similar concept may be applied if you're doing this any other way.

Don't have an account?
Coming from Hortonworks? Activate your account here