Script:
a = LOAD 'geolocation' using org.apache.hive.hcatalog.pig.HCatLoader();
b = filter a by event != 'normal';
c = foreach b generate driverid, event, (int) '1' as occurance;
d = group c by driverid;
e = foreach d generate group as driverid, SUM(c.occurance) as t_occ;
g = LOAD 'drivermileage' using org.apache.hive.hcatalog.pig.HCatLoader();
h = join e by driverid, g by driverid;
final_data = foreach h generate $0 as driverid, $1 as events, $3 as totmiles, (float) $3/$1 as riskfactor;
store final_data into 'riskfactor' using org.apache.hive.hcatalog.pig.HCatStorer();
Error:
ls: cannot access /hadoop/yarn/local/usercache/maria_dev/appcache/application_1475181427748_0005/container_1475181427748_0005_01_000002/hive.tar.gz/hive/lib/slf4j-api-*.jar: No such file or directory
ls: cannot access /hadoop/yarn/local/usercache/maria_dev/appcache/application_1475181427748_0005/container_1475181427748_0005_01_000002/hive.tar.gz/hive/hcatalog/lib/*hbase-storage-handler-*.jar: No such file or directory
WARNING: Use "yarn jar" to launch YARN applications.
16/09/29 21:30:53 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
16/09/29 21:30:53 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE
16/09/29 21:30:53 INFO pig.ExecTypeProvider: Trying ExecType : TEZ_LOCAL
16/09/29 21:30:53 INFO pig.ExecTypeProvider: Trying ExecType : TEZ
16/09/29 21:30:53 INFO pig.ExecTypeProvider: Picked TEZ as the ExecType
2016-09-29 21:30:53,748 [main] INFO org.apache.pig.Main - Apache Pig version 0.16.0.2.5.0.0-1245 (rexported) compiled Aug 26 2016, 02:07:35
2016-09-29 21:30:53,748 [main] INFO org.apache.pig.Main - Logging error messages to: /hadoop/yarn/local/usercache/maria_dev/appcache/application_1475181427748_0005/container_1475181427748_0005_01_000002/pig_1475184653746.log
2016-09-29 21:30:54,859 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/yarn/.pigbootup not found
2016-09-29 21:30:55,011 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://sandbox.hortonworks.com:8020
2016-09-29 21:30:56,136 [main] INFO org.apache.pig.PigServer - Pig Script ID for the session: PIG-script.pig-720ef8a5-85e0-4929-abc5-514885b28262
2016-09-29 21:30:56,680 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://sandbox.hortonworks.com:8188/ws/v1/timeline/
2016-09-29 21:30:56,852 [main] INFO org.apache.pig.backend.hadoop.PigATSClient - Created ATS Hook
2016-09-29 21:30:57,964 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist
2016-09-29 21:30:58,026 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://sandbox.hortonworks.com:9083
2016-09-29 21:30:58,101 [main] INFO hive.metastore - Connected to metastore.
2016-09-29 21:30:58,731 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist
2016-09-29 21:30:58,733 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://sandbox.hortonworks.com:9083
2016-09-29 21:30:58,734 [main] INFO hive.metastore - Connected to metastore.
2016-09-29 21:30:58,983 [main] WARN org.apache.pig.newplan.BaseOperatorPlan - Encountered Warning IMPLICIT_CAST_TO_FLOAT 1 time(s).
2016-09-29 21:30:59,098 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist
2016-09-29 21:30:59,143 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://sandbox.hortonworks.com:9083
2016-09-29 21:30:59,145 [main] INFO hive.metastore - Connected to metastore.
2016-09-29 21:30:59,569 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 0:
Output Location Validation Failed for: 'riskfactor More info to follow:
Pig 'double' type in column 2(0-based) cannot map to HCat 'BIGINT'type. Target filed must be of HCat type {DOUBLE}
Details at logfile: /hadoop/yarn/local/usercache/maria_dev/appcache/application_1475181427748_0005/container_1475181427748_0005_01_000002/pig_1475184653746.log
2016-09-29 21:30:59,589 [main] INFO org.apache.pig.Main - Pig script completed in 6 seconds and
