Reply
Explorer
Posts: 40
Registered: ‎08-29-2017

Pig Error - Encountered Warning ACCESSING_NON_EXISTENT_FIELD 2 time(s)

[ Edited ]

--PIG script to parse out metrics from nmon log collection for one day
--Get collection file name from command line argument : pig -f nmon2file.pig -param input=collected_file save=file_delimiter

REGISTER /usr/lib/pig/piggybank.jar;

-- streaming routines for additional processing
DEFINE CMD `awk 'BEGIN{FS=",";} { for(N=5; N < NF/2; N+=1) { print \$1","\$2","\$4","\$3","\$N","\$(N+(NF/2))} }'` INPUT(stdin USING PigStreaming(',')) OUTPUT (stdout USING PigStreaming(','));

DEFINE AAA_CMD `awk '{for (i=0;i<4;i++) sub(",", "|", \$0); print}' | tr ',' '#' | tr '|' ','` INPUT(stdin USING PigStreaming(',')) OUTPUT (stdout USING PigStreaming(','));


--set pig.tmpfilecompression true;
--set pig.tmpfilecompression.codec lzo;
SET DEFAULT_PARALLEL 5;

raw_nmon_DS = LOAD '$input' using PigStorage(',');
TimeStamp_raw = FILTER raw_nmon_DS BY $2=='ZZZZ' and $6 is null;
TimeStamp = FOREACH TimeStamp_raw generate (chararray)$0 as fileName:chararray,$1 as ymd:chararray,$3 as Tvalue, ToUnixTime(ToDate(CONCAT(CONCAT($4,' '),$5),'H:mm:ss dd-MMM-yyyy','PST8PDT')) as Tunix:long, CONCAT(CONCAT($4,' '),$5) as Tdate:chararray;

CPU_ALL,CPU Total a01gicomapp1a,User%,Sys%,Wait%,Idle%,Busy,PhysicalCPUs
--CPU_ALL
cpu_all_data_raw = filter raw_nmon_DS by $2=='CPU_ALL' and ($3 matches '^T[0-9].*');
cpu_all_data = foreach cpu_all_data_raw generate (chararray)$0 as fileName:chararray,$1 as ymd:chararray, (chararray)$2 as metricName:chararray,$3 as TS, $4 as User:float,$5 as Sys:float, $6 as Wait:float, $7 as Idle:float,($8 is NULL ? 0.0 : $8) as Busy:float,$9 as CPUs:int;
cpu_all_data_TS = join cpu_all_data by (fileName,ymd,TS), TimeStamp by (fileName,ymd,Tvalue) USING 'replicated';
cpu_all_data_TS = foreach cpu_all_data_TS generate cpu_all_data::fileName,metricName,Tdate,Tunix,User,Sys,Wait,Idle,Busy,CPUs,cpu_all_data::ymd;
rmf nmon_cpu_all_$save
STORE cpu_all_data_TS INTO 'nmon_cpu_all_$save' using org.apache.pig.piggybank.storage.CSVExcelStorage(',', 'NO_MULTILINE', 'UNIX');

 

 

----------------------------------

 

 

exception

2018-02-20 10:09:56,797 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning ACCESSING_NON_EXISTENT_FIELD 2 time(s).
2018-02-22 12:09:20,447 [main] WARN org.apache.pig.PigServer - Encountered Warning IMPLICIT_CAST_TO_CHARARRAY 7 time(s).
2018-02-22 12:09:20,447 [main] WARN org.apache.pig.PigServer - Encountered Warning USING_OVERLOADED_FUNCTION 4 time(s).
2018-02-22 12:09:20,447 [main] WARN org.apache.pig.PigServer - Encountered Warning IMPLICIT_CAST_TO_DOUBLE 1 time(s).
2018-02-22 12:09:20,469 [main] WARN org.apache.pig.PigServer - Encountered Warning IMPLICIT_CAST_TO_CHARARRAY 7 time(s).
2018-02-22 12:09:20,469 [main] WARN org.apache.pig.PigServer - Encountered Warning USING_OVERLOADED_FUNCTION 4 time(s).
2018-02-22 12:09:20,469 [main] WARN org.apache.pig.PigServer - Encountered Warning IMPLICIT_CAST_TO_DOUBLE 1 time(s)

 

2018-02-20 10:09:56,797 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!

-----------------------------------------------------------------------

 

 

My job is success and a folder is created in Hadoop but the partm file is empty

 

hadoop version 2.6.0-cdh5.11.1 pig version- 0.12.0-cdh5.11.1        

 

 

I am unable to fix this error. Can anyone help pls. Thanks.

 

 

 

 

Explorer
Posts: 40
Registered: ‎08-29-2017

Re: Pig Error - Encountered Warning ACCESSING_NON_EXISTENT_FIELD 2 time(s)

2018-02-23 10:47:10,097 WARN [main] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as: (auth:SIMPLE) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
2018-02-23 10:47:10,098 WARN [main] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
2018-02-23 10:47:10,098 WARN [main] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:(auth:SIMPLE) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error

2018-02-23 10:47:10,260 INFO [main] org.apache.pig.data.SchemaTupleBackend: Key [pig.schematuple] was not set... will not generate code.

Announcements
New solutions