About psivala

psivala · ‎02-26-2016

hadoop-common-2.7.1-amzn-0 ... can you please guide me to an article that tells about the installation of hcatalogs and classpath setup ?

psivala · ‎02-26-2016

tried it got below warnings...i assume it means that the jar's need to be added in order to use Hcatalog ? let me know if i am wrong pig -x tez -useHCatalog ls: cannot access /usr/lib/hive/lib/slf4j-api-*.jar: No such file or directory ls: cannot access /usr/lib/hive-hcatalog/share/hcatalog/*hcatalog-core-*.jar: No such file or directory ls: cannot access /usr/lib/hive-hcatalog/share/hcatalog/hcatalog-*.jar: No such file or directory ls: cannot access /usr/lib/hive-hcatalog/lib/*hbase-storage-handler-*.jar: No such file or directory ls: cannot access /usr/lib/hive-hcatalog/share/hcatalog/*hcatalog-pig-adapter-*.jar: No such file or directory

psivala · ‎02-26-2016

Its not resolved admin team would be looking into it...you can close this issue

psivala · ‎02-26-2016

error: pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.hive.hcatalog.pig.HCatLoader' with arguments 'null' A = LOAD 'patch' USING org.apache.hive.hcatalog.pig.HCatLoader(); registered the below jars:piggybank-0.15.0.jar;hive-hcatalog-hbase-storage-handler-0.13.1.jarhive-hcatalog-pig-adapter-1.2.1.jarhive-hcatalog-core-1.2.1.jarhive-shims-2.0.0.jar;

psivala · ‎02-26-2016

Its an environment error

psivala · ‎02-02-2016

Yes the value can be null in some cases

psivala · ‎02-02-2016

sample data i am trying to read "stTime": "2014-05-31T22:00:00-07:00", "tzone": -7, "Time": "2015-12-04T05:01:47-07:00", "user": "4", "body": { "prev": null, -------> i think this i where its failing not sure how to read it ? "status": "pairing" }, "type": "Status", "time": "2014-05-31T22:00:00-07:00", "device": "1:1"

psivala · ‎02-01-2016

I tried both in my home driectory and it worked fine and also it worked fine when i had my parm file on s3. But its not working when i try to do it from hdfs.

psivala · ‎01-28-2016

Below is my sample code A = load 's3://xxxx-etl/event_store_new/share' USING parquet.pig.ParquetLoader(); B = FOREACH A GENERATE address,id,managingOrganization,name,resourceType,flatten(telecom) as telecom; (101, Center,[state#CA,line#kinson St.,postalCode#0003,country#US,city#San Diego],[reference##12:239,display#dbbbb],localtion,[value#1,use#home,system#phone]) Ex: [state#CA,line#kinson St.,postalCode#0003,country#US,city#San Diego] to separate column values state CA line kinson St. postalCode 0003 country US city San Diego

psivala · ‎01-26-2016

pig -x mapreduce /home/hadoop/test.pig -param_file=hdfs://ip-XXX-XX-XX-XXX.ec2.internal:8020/home/hadoop/adh_time I am trying to run the above command and its not working saying undefined parameter i tired multiple ways and nothing works.Any help would be appreciated. Value inside the adh_time is input = 2015-06-16T14:59:55.000Z Error: 16/01/26 20:56:38 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL 16/01/26 20:56:38 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE 16/01/26 20:56:38 INFO pig.ExecTypeProvider: Picked MAPREDUCE as the ExecType 46 [main] INFO org.apache.pig.Main - Apache Pig version 0.14.0-amzn-0 (r: unknown) compiled Nov 16 2015, 21:51:33 16/01/26 20:56:38 INFO pig.Main: Apache Pig version 0.14.0-amzn-0 (r: unknown) compiled Nov 16 2015, 21:51:33 47 [main] INFO org.apache.pig.Main - Logging error messages to: /mnt/var/log/pig/pig_1453841798872.log 16/01/26 20:56:38 INFO pig.Main: Logging error messages to: /mnt/var/log/pig/pig_1453841798872.log 789 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/hadoop/.pigbootup not found 16/01/26 20:56:39 INFO util.Utils: Default bootup file /home/hadoop/.pigbootup not found 810 [main] ERROR org.apache.pig.impl.PigContext - Undefined parameter : input 16/01/26 20:56:39 ERROR impl.PigContext: Undefined parameter : input 828 [main] ERROR org.apache.pig.Main - ERROR 2997: Encountered IOException. org.apache.pig.tools.parameters.ParameterSubstitutionException: Undefined parameter : input 16/01/26 20:56:39 ERROR pig.Main: ERROR 2997: Encountered IOException. org.apache.pig.tools.parameters.ParameterSubstitutionException: Undefined parameter : input Details at logfile: /mnt/var/log/pig/pig_1453841798872.log 848 [main] INFO org.apache.pig.Main - Pig script completed in 952 milliseconds (952 ms) 16/01/26 20:56:39 INFO pig.Main: Pig script completed in 952 milliseconds (952 ms)

psivala · ‎01-25-2016

psivala · ‎01-21-2016

tried to define DEFINE DATE_TIME org.apache.pig.piggybank.evaluation.datetime.DATE_TIME(); still getting same error

psivala · ‎01-21-2016

Artem, when i try to store the data to s3 it says file already exists i want to overwrite it if it exists. STORE D INTO 's3://xxx-analytics-etl/output/Count' USING PigStorage(',');

psivala · ‎01-21-2016

If so can you provide an example

psivala · ‎01-20-2016

A = load 's3://xxxx-analytics-etl/etl_datasource/Count_2015-06-20.par' USING parquet.pig.ParquetLoader();

psivala · ‎01-20-2016

If i try to run the same script again it works fine

psivala · ‎01-20-2016

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. I tried registering all the jar files mentioned but still getting the same error.

psivala · ‎01-20-2016

Thanks Artem the PigStorage(',') worked

psivala · ‎01-19-2016

psivala · ‎01-19-2016

Actually they gave me a corrupted file which was causing the issue...between i have another question can i store the output from parquet as csv ?

psivala · ‎01-18-2016

A = load 's3://xxs-analytics-etl/event_store/xxxx_2015-06-15' USING parquet.pig.ParquetLoader() as (user: chararray); I tried just reading one column but got the same error

psivala · ‎01-18-2016

tried it got the same error

psivala · ‎01-18-2016

I am ok casting the field to string---- Below is my script A = load 's3://XXXX-analytics-etl/event_store/Count_2015-06-15' USING parquet.pig.ParquetLoader() as (class: chararray,localTime: chararray,updated: chararray,tzone: double,rid: chararray,value: double,time: chararray,user: chararray,patchId: long);

psivala · ‎01-18-2016

i am using parquet-pig-bundle-1.5.0.jar.

psivala · ‎01-18-2016

Pig Stack Trace --------------- ERROR 1200: can't convert optional int96 uploadTime Failed to parse: can't convert optional int96 uploadTime at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:201) at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1707) at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1680) at org.apache.pig.PigServer.registerQuery(PigServer.java:623) at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:1061) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:501) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66) at org.apache.pig.Main.run(Main.java:560) at org.apache.pig.Main.main(Main.java:170) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: parquet.pig.SchemaConversionException: can't convert optional int96 uploadTime at parquet.pig.PigSchemaConverter.convertFields(PigSchemaConverter.java:108) at parquet.pig.PigSchemaConverter.convert(PigSchemaConverter.java:84) at parquet.pig.TupleReadSupport.getPigSchemaFromMultipleFiles(TupleReadSupport.java:70) at parquet.pig.ParquetLoader.initSchema(ParquetLoader.java:204) at parquet.pig.ParquetLoader.setInput(ParquetLoader.java:108) at parquet.pig.ParquetLoader.getSchema(ParquetLoader.java:188) at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:175) at org.apache.pig.newplan.logical.relational.LOLoad.<init>(LOLoad.java:89) at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:901) at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568) at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625) at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102) at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560) at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421) at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:191) ... 16 more Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 0: NYI at parquet.pig.PigSchemaConverter$1.convertINT96(PigSchemaConverter.java:148) at parquet.pig.PigSchemaConverter$1.convertINT96(PigSchemaConverter.java:120) at parquet.schema.PrimitiveType$PrimitiveTypeName$7.convert(PrimitiveType.java:219) at parquet.pig.PigSchemaConverter.getSimpleFieldSchema(PigSchemaConverter.java:119) at parquet.pig.PigSchemaConverter.getFieldSchema(PigSchemaConverter.java:222) at parquet.pig.PigSchemaConverter.convertFields(PigSchemaConverter.java:99) ... 30 more ================================================================================

Online	Offline
Last Visited	‎06-22-2016 06:49 PM

Date Registered	‎01-18-2016 07:25 PM
Last Visited	‎06-22-2016 06:49 PM
Total Messages Posted	25
Total Kudos Received	14

Re: Pig joib failing SLF4J: Failed to load class "...

Re: Error reading data from parquet file ( I am pr...

Re: Error while trying to read Hive table using Pi...

Re: Error while trying to read Hive table using Pi...

Re: Pig joib failing SLF4J: Failed to load class "...

Error while trying to read Hive table using Pig

Re: Pig joib failing SLF4J: Failed to load class "...

Re: ERROR grunt.Grunt: ERROR 1200: null while read...

ERROR grunt.Grunt: ERROR 1200: null while reading ...

Re: Passing parameters to pig script from file iss...

Trying to read a nested parquet file using PIG, no...

Passing parameters to pig script from file issue

How to substractduration of 1min from a ISO time ...

ERROR 1070: Could not resolve org.apache.pig.piggy...

Re: can i overwrite files in S3 using pig ?

can i overwrite files in S3 using pig ?

Re: Pig joib failing SLF4J: Failed to load class "...

Re: Pig joib failing SLF4J: Failed to load class "...

Pig joib failing SLF4J: Failed to load class "org....

Re: Can i store the output from the parquet file a...

Can i store the output from the parquet file as cs...

Re: Error reading data from parquet file ( I am pr...

Re: Error reading data from parquet file ( I am pr...

Re: Error reading data from parquet file ( I am pr...

Re: Error reading data from parquet file ( I am pr...

Re: Error reading data from parquet file ( I am pr...

Error reading data from parquet file ( I am pretty...