Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How can I enable pig -useHCatalog via Ambari. I seem to get stuck everytime I am try to execute scripts via Pig View.

avatar
Guru

This is what Ambari says:

WARNING: Use "yarn jar" to launch YARN applications. 16/06/02 21:12:43 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL 16/06/02 21:12:43 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE 16/06/02 21:12:43 INFO pig.ExecTypeProvider: Trying ExecType : TEZ_LOCAL 16/06/02 21:12:43 INFO pig.ExecTypeProvider: Trying ExecType : TEZ 16/06/02 21:12:43 INFO pig.ExecTypeProvider: Picked TEZ as the ExecType 2016-06-02 21:12:43,603 [main] INFO org.apache.pig.Main - Apache Pig version 0.15.0.2.4.0.0-169 (rexported) compiled Feb 10 2016, 07:50:04 2016-06-02 21:12:43,603 [main] INFO org.apache.pig.Main - Logging error messages to: /hadoop/yarn/local/usercache/admin/appcache/application_1464899556124_0006/container_e16_1464899556124_0006_01_000002/pig_1464901963602.log 2016-06-02 21:12:44,356 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/yarn/.pigbootup not found 2016-06-02 21:12:44,477 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://sandbox.hortonworks.com:8020 2016-06-02 21:12:45,021 [main] INFO org.apache.pig.PigServer - Pig Script ID for the session: PIG-script.pig-b1da2d85-2845-4a3f-b59b-b2e367f74c1c 2016-06-02 21:12:45,420 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://sandbox.hortonworks.com:8188/ws/v1/timeline/ 2016-06-02 21:12:45,646 [main] INFO org.apache.pig.backend.hadoop.ATSService - Created ATS Hook 2016-06-02 21:12:46,138 [main] ERROR org.apache.pig.PigServer - exception during parsing: Error during parsing. Pig script failed to parse: <file script.pig, line 4, column 4> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.hive.hcatalog.pig.HCatLoader' with arguments 'null' Failed to parse: Pig script failed to parse: <file script.pig, line 4, column 4> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.hive.hcatalog.pig.HCatLoader' with arguments 'null' at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:199) at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1776) at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1484) at org.apache.pig.PigServer.parseAndBuild(PigServer.java:428) at org.apache.pig.PigServer.executeBatch(PigServer.java:453) at org.apache.pig.PigServer.executeBatch(PigServer.java:439) at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) at org.apache.pig.Main.run(Main.java:502) at org.apache.pig.Main.main(Main.java:177) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: <file script.pig, line 4, column 4> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.hive.hcatalog.pig.HCatLoader' with arguments 'null' at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:897) at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568) at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625) at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102) at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560) at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421) at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:191) ... 17 more Caused by: java.lang.RuntimeException: could not instantiate 'org.apache.hive.hcatalog.pig.HCatLoader' with arguments 'null' at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:772) at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:885) ... 23 more Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/NoSuchObjectException at org.apache.hive.hcatalog.pig.HCatLoader.<init>(HCatLoader.java:71) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at java.lang.Class.newInstance(Class.java:383) at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:742) ... 24 more Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.api.NoSuchObjectException at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 31 more 2016-06-02 21:12:46,148 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: Pig script failed to parse: <file script.pig, line 4, column 4> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.hive.hcatalog.pig.HCatLoader' with arguments 'null' Details at logfile: /hadoop/yarn/local/usercache/admin/appcache/application_1464899556124_0006/container_e16_1464899556124_0006_01_000002/pig_1464901963602.log 2016-06-02 21:12:46,167 [main] INFO org.apache.pig.Main - Pig script completed in 2 seconds and 722 milliseconds (2722 ms)

1 ACCEPTED SOLUTION

avatar
Super Guru

Have you tried adding "-useHCatalog" as a pig argument? On the pig script, there is an option in the lower right to add an argument. This worked for me. Without this option, I was unable to run an pig scripts in Ambari that required HCat based jars.

4878-screen-shot-2016-06-08-at-34626-pm.png

View solution in original post

12 REPLIES 12

avatar

Also from ambari i tried but its failing with error as not Hcatalog file does not exists.

Should i reinstall hortonworks sandbox once again.

avatar
Guru

Can you share the error message and log file for ambari server "/var/log/ambari-server/"

avatar

I know this is the silly question but where can you find the location. I am not able to trace back @srai