Support Questions

Find answers, ask questions, and share your expertise
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

Conditional Split on String Variable condition


I have a requirement to filter out specific string values like AB01, BC01,CD05 stored in a variable which can change later and I don't want to hard code this values in my conditional split comparison. Also I have a limitation not to filter these values in the source so I want to control it in Conditional split.

My condition is COMP_Name != (AB01, BC01,CD05)

Any help is highly appreciated.

Thank you

Saanvi S


Expert Contributor

@Saanvi Sharma, you're doing this in Spark?

You can declare the variable as a list, for exampe - comp_list=['AB01','BC01']

Then you can filter it with rdd.filter(lambda x:not(x in comp_list)), if you're filtering on this string directly,

Otherwise you can put your field as an argument to lambda expression.

Similar concept may be applied if you're doing this any other way.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.