question Re: Apache PIG - Guarantee that all the value in a column are Numeric Values in Archives of Support Questions (Read Only)

Apache PIG - Guarantee that all the value in a column are Numeric Values

m2014227 — Wed, 27 Jul 2016 23:35:37 GMT

Hi experts, I've a dataset with 4 columns and want to know if the column B only have numbers, if the job detect some non numeric value I want to put that value into null. Could I do this in PIG or must be Python embed pig? Many thanks!

Re: Apache PIG - Guarantee that all the value in a column are Numeric Values

arunak — Thu, 28 Jul 2016 02:13:35 GMT

You can use org.apache.pig.piggybank.evaluation.IsNumeric

Some thing like

X = foreach Y generate ((org.apache.pig.piggybank.evaluation.IsNumeric($1)==true)?(int)$1:null)

I have applied generate for one column, you can add rest of columns

Re: Apache PIG - Guarantee that all the value in a column are Numeric Values

m2014227 — Thu, 28 Jul 2016 05:13:53 GMT

Brilliant 🙂 Only one more question: How can I add a Case Statement (or a If) to my X var;

Re: Apache PIG - Guarantee that all the value in a column are Numeric Values

arunak — Thu, 28 Jul 2016 20:26:03 GMT

You can apply a filter or bincond operator on any column(s) of your relation X. You can get more details on available operators here