Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Filter OUT Null from all columns

avatar
New Contributor

Hi Guys,

My sample file has 27 columns with INT and Chararray data types.

My requirement is to filter out all the null values which are present in these columns

What I am trying to do is this,

Sample_fil = FILTER Adzone by (Div is not null) and (Zone_yak is not null) and (ProdGroup is not null) and (Zonename is not null) and (Store_fruit is not null) and (Comp_Zone is not null) and (Department is not null);

This script is only for the first 7 columns only. I can write the same script even for the rest of the columns but I am looking for a way to optimize the script. My sample file has 12 Integer datatype and the rest are Chararray.

Please give your suggestions.

Regards,

Pradeep.

1 ACCEPTED SOLUTION

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
3 REPLIES 3

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Contributor

Its better to make use of UDFs in this condition. Check the below link, it has a UDF for the same,

http://stackoverflow.com/questions/12959001/how-to-filter-records-with-a-null-value-in-pig

Hope this helps.

Regards,

Arun

avatar
New Contributor

Ya I have seen this already, I was just wondering if there was a better way to do it. Thanks anyway.