Archives of Support Questions (Read Only)

This is an archived board for historical reference. Information and links may no longer be available or relevant
Announcements
This board is archived and read-only for historical reference. To ask a new question, please post a new topic on the appropriate active board.

How can I enable pig -useHCatalog via Ambari. I seem to get stuck everytime I am try to execute scripts via Pig View.

avatar
Guru

This is what Ambari says:

WARNING: Use "yarn jar" to launch YARN applications. 16/06/02 21:12:43 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL 16/06/02 21:12:43 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE 16/06/02 21:12:43 INFO pig.ExecTypeProvider: Trying ExecType : TEZ_LOCAL 16/06/02 21:12:43 INFO pig.ExecTypeProvider: Trying ExecType : TEZ 16/06/02 21:12:43 INFO pig.ExecTypeProvider: Picked TEZ as the ExecType 2016-06-02 21:12:43,603 [main] INFO org.apache.pig.Main - Apache Pig version 0.15.0.2.4.0.0-169 (rexported) compiled Feb 10 2016, 07:50:04 2016-06-02 21:12:43,603 [main] INFO org.apache.pig.Main - Logging error messages to: /hadoop/yarn/local/usercache/admin/appcache/application_1464899556124_0006/container_e16_1464899556124_0006_01_000002/pig_1464901963602.log 2016-06-02 21:12:44,356 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/yarn/.pigbootup not found 2016-06-02 21:12:44,477 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://sandbox.hortonworks.com:8020 2016-06-02 21:12:45,021 [main] INFO org.apache.pig.PigServer - Pig Script ID for the session: PIG-script.pig-b1da2d85-2845-4a3f-b59b-b2e367f74c1c 2016-06-02 21:12:45,420 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://sandbox.hortonworks.com:8188/ws/v1/timeline/ 2016-06-02 21:12:45,646 [main] INFO org.apache.pig.backend.hadoop.ATSService - Created ATS Hook 2016-06-02 21:12:46,138 [main] ERROR org.apache.pig.PigServer - exception during parsing: Error during parsing. Pig script failed to parse: <file script.pig, line 4, column 4> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.hive.hcatalog.pig.HCatLoader' with arguments 'null' Failed to parse: Pig script failed to parse: <file script.pig, line 4, column 4> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.hive.hcatalog.pig.HCatLoader' with arguments 'null' at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:199) at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1776) at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1484) at org.apache.pig.PigServer.parseAndBuild(PigServer.java:428) at org.apache.pig.PigServer.executeBatch(PigServer.java:453) at org.apache.pig.PigServer.executeBatch(PigServer.java:439) at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) at org.apache.pig.Main.run(Main.java:502) at org.apache.pig.Main.main(Main.java:177) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: <file script.pig, line 4, column 4> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.hive.hcatalog.pig.HCatLoader' with arguments 'null' at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:897) at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568) at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625) at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102) at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560) at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421) at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:191) ... 17 more Caused by: java.lang.RuntimeException: could not instantiate 'org.apache.hive.hcatalog.pig.HCatLoader' with arguments 'null' at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:772) at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:885) ... 23 more Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/NoSuchObjectException at org.apache.hive.hcatalog.pig.HCatLoader.<init>(HCatLoader.java:71) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at java.lang.Class.newInstance(Class.java:383) at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:742) ... 24 more Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.api.NoSuchObjectException at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 31 more 2016-06-02 21:12:46,148 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: Pig script failed to parse: <file script.pig, line 4, column 4> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.hive.hcatalog.pig.HCatLoader' with arguments 'null' Details at logfile: /hadoop/yarn/local/usercache/admin/appcache/application_1464899556124_0006/container_e16_1464899556124_0006_01_000002/pig_1464901963602.log 2016-06-02 21:12:46,167 [main] INFO org.apache.pig.Main - Pig script completed in 2 seconds and 722 milliseconds (2722 ms)

1 ACCEPTED SOLUTION

avatar
Super Guru

Have you tried adding "-useHCatalog" as a pig argument? On the pig script, there is an option in the lower right to add an argument. This worked for me. Without this option, I was unable to run an pig scripts in Ambari that required HCat based jars.

4878-screen-shot-2016-06-08-at-34626-pm.png

View solution in original post

12 REPLIES 12

avatar

Also from ambari i tried but its failing with error as not Hcatalog file does not exists.

Should i reinstall hortonworks sandbox once again.

avatar
Guru

Can you share the error message and log file for ambari server "/var/log/ambari-server/"

avatar

I know this is the silly question but where can you find the location. I am not able to trace back @srai